Python Spark Lead

Job Description:

We are looking for an experienced Python Spark Lead responsible for designing, developing, and optimizing large-scale data processing solutions using Python, PySpark, Apache Spark, SQL, Airflow, and Control-M. The candidate will work on data engineering projects, build scalable ETL pipelines, optimize Spark jobs, and support enterprise data platforms.

Key Responsibilities:

- Develop and maintain scalable data processing pipelines using Python and PySpark.

- Design, build, and optimize Apache Spark applications for large-scale data processing.

- Write complex SQL queries for data extraction, transformation, and analysis.

- Develop and manage ETL workflows using Spark and Python.

- Schedule and monitor workflows using Apache Airflow and Control-M.

- Perform Spark performance tuning, debugging, and optimization.

- Work with large datasets and distributed computing environments.

- Troubleshoot production issues and ensure data pipeline reliability.

- Collaborate with data architects, analysts, and engineering teams.

- Follow best practices for coding, testing, and deployment.

Mandatory Skills:

- Python

- PySpark

- Apache Spark

- SQL

- Airflow

- Control-M

- ETL/Data Pipeline Development

- Spark Performance Optimization

Skills

Python, PySpark, Apache Spark, SQL, Control-M, ETL, Performance Tuning, Apache Airflow, Distributed Computing