Job Summary
Sr Developer role for an MNC focusing on data engineering and advanced analytics solutions using Spark Scala Python and PySpark. The position requires six to ten years of hands on development experience in hybrid work model with day shift schedule and no travel. The role drives scalable data pipelines that support business decisions and societal impact through better insights.
Responsibilities
Design robust distributed data processing solutions using Spark Scala Python and PySpark that transform complex raw data into reliable curated datasets for critical business analytics and reporting needs.
Develop highly efficient batch and near real time data pipelines optimized for performance and resource utilization ensuring stable delivery of trusted data to downstream applications and data science teams.
Implement clean modular and reusable code with strong focus on coding standards unit testing and maintainability so that data engineering assets remain stable and easy to evolve over time.
Collaborate closely with data analysts and business stakeholders to translate analytical requirements into technical designs that deliver clear measurable value for company objectives and customer outcomes.
Optimize Spark jobs through advanced tuning techniques including partitioning caching and efficient data formats to reduce execution time and infrastructure cost while maintaining accuracy and reliability.
Integrate data from diverse sources into unified data models using Spark and Python ensuring consistent data quality governance and traceability across the data lifecycle.
Troubleshoot complex production issues in data pipelines by analyzing logs performance metrics and data anomalies and implement durable fixes that prevent recurrence and improve system resilience.
Document technical solutions data flows and operational procedures in a structured and accessible manner so that other team members can support and enhance the platforms with confidence.
