Python Spark Lead

Job Description :

We are looking for a strong Python & Spark Lead with hands-on expertise in Python, PySpark, Scala, Apache Spark, and Big Data technologies. The ideal candidate should have strong coding skills and experience in building scalable data solutions, data pipelines, migration accelerators, and high-performance ETL/ELT frameworks. Candidates with experience in product development or Tier-1 organizations and strong engineering fundamentals are preferred.

Key Responsibilities :

- Lead the design and development of scalable, secure, and high-performance data pipelines using Python, PySpark, Scala, and Spark.

- Develop robust ETL/ELT pipelines for large-scale data processing and transformation.

- Design and implement data engineering solutions using Apache Spark and Big Data technologies.

- Develop migration automation frameworks and accelerators to improve data migration efficiency.

- Drive Dataiku-to-Python migration initiatives, including redesign, automation, and optimization of existing workflows.

- Develop reusable frameworks and components for data ingestion, transformation, validation, and processing.

- Work with Java and SQL for data processing, integration, and application development.

- Design and implement data integration solutions across multiple systems and data sources.

- Optimize Spark jobs, pipelines, queries, and applications for performance and scalability.

- Implement appropriate data quality, validation, error-handling, and monitoring mechanisms.

- Collaborate with architecture, product, QA, and engineering teams to deliver high-quality solutions.

- Lead technical design discussions, code reviews, testing, deployment, and production support.

- Mentor engineers and contribute to engineering best practices and coding standards.

Required Technical Skills :

- 7-10 years of experience in software/data engineering.

- Strong hands-on expertise in Python and PySpark.

- Strong knowledge of Apache Spark and Big Data technologies.

- Experience with Scala and good understanding of Java.

- Strong proficiency in SQL.

- Hands-on experience in ETL/ELT and data pipeline development.

- Experience building scalable and high-performance data processing solutions.

- Strong understanding of data integration, transformation, testing, and optimization.

- Experience developing migration tools, automation frameworks, or accelerators.

Skills

Data Engineering, Python, Apache Spark, ETL, Big Data, PySpark, Data Pipeline