Assistant Manager - Data Engineering

We are seeking an experienced Data Engineer with hands-on experience in ETL development and AWS-based data engineering solutions. The ideal candidate will have strong expertise in PySpark, Python (NumPy, Pandas), and large-scale distributed data processing, along with proven experience in deploying and managing AWS data services.

You will work closely with IT and business stakeholders to design, build, and optimize scalable data pipelines while ensuring cost efficiency, data quality, and high performance across the data ecosystem.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using PySpark and Python.
• Build and deploy AWS-based data solutions leveraging services such as Glue, S3, Lambda, SNS, and Step Functions.
• Perform large-scale data transformations and processing using distributed data frameworks.
• Implement and manage Delta Lake/Delta Tables within a Data Lake architecture.
• Optimize data pipelines for performance, scalability, and cost efficiency in AWS environments.
• Ensure data quality, reliability, and governance across integrated data systems.
• Collaborate with cross-functional IT and business stakeholders to resolve delivery issues and align on requirements.
• Orchestrate workflows using MWAA (Airflow) and manage end-to-end data pipeline automation is an added advantage.

Required Skills:

• 4 to 8 years of experience in Data Engineering.
• Strong expertise in AWS services (S3, Lambda, SNS, Step Functions).
• Hands-on experience with PySpark and Python.
• Strong proficiency in Python libraries such as NumPy and Pandas.
• Experience with large-scale data transformations.
• Strong communication and stakeholder management skills.

Good to Have:

• Metadata management, data lineage, data governance, and Data Lake architecture.
• Delta Tables / Delta Lake and MWAA (Airflow orchestration).
• Experience in cost optimization strategies in AWS.

Skills

AWS, Airflow, Data Engineering, Glue, Lambda, PySpark, Python, S3