Sr. Data Engineer

We are looking for a Senior Data Engineer to design, build, and maintain scalable, high-performance data pipelines that power our SaaS-based AI applications and Predictions-as-a-Service products. The role focuses on building reliable data infrastructure, optimizing data processing workflows, and ensuring data quality to support AI/ML solutions and business decision-making in the Consumer Packaged Goods (CPG) sector. The ideal candidate has strong hands-on experience with Python, PySpark, SQL, Apache Spark, and workflow orchestration tools, along with a proven track record of developing and maintaining production-grade data engineering solutions.

Responsibilities:

• Design, develop, and maintain scalable ETL/ELT data pipelines for data extraction, ingestion, transformation, and processing.
• Build, deploy, and orchestrate data workflows using Apache Airflow and/or Kubeflow.
• Own multiple data pipelines end-to-end, ensuring reliability, scalability, and performance.
• Develop and maintain production-grade Spark applications using PySpark.
• Monitor, troubleshoot, audit, and optimize existing pipelines to improve data quality, processing speed, and resource utilization.
• Implement real-time pipeline monitoring, alerting, and error-handling mechanisms.
• Design and maintain efficient data models and scalable big data processing infrastructure.
• Follow modular, configurable, and reusable programming practices using configuration files and parameterized workflows.
• Prepare and maintain High-Level Design (HLD), Low-Level Design (LLD), technical documentation, and implementation plans.
• Collaborate with AI/ML, Machine Learning Engineering (MLE), DevOps, product, and implementation teams to deliver effective data solutions.
• Support deployed pipelines through bug fixes, enhancements, and ongoing performance improvements.
• Contribute to engineering best practices, knowledge sharing, cross-team training, and hiring initiatives.

Requirements:

• Strong programming proficiency in Python, PySpark, and SQL.
• Hands-on experience writing, deploying, and maintaining Apache Spark code in production environments.
• Proven experience building and supporting scalable ETL/ELT pipelines and data processing workflows.
• Experience with data extraction, ingestion, transformation, monitoring, and alerting.
• Familiarity with workflow orchestration tools such as Apache Airflow and/or Kubeflow.
• Experience working with public cloud platforms such as AWS or Azure.
• Strong understanding of big data infrastructure, data modeling, and distributed data processing.
• Experience optimizing data loading, ingestion, Spark jobs, and pipeline performance.
• Working knowledge of developer tools such as Git, GitHub, Docker, and VS Code.
• Strong debugging, problem-solving, and analytical skills.
• Ability to work effectively in a fast-paced, deadline-driven environment and collaborate across multiple teams.

Preferred Candidate Profile:

• Strong ownership of production data pipelines, from development through deployment and maintenance.
• Ability to translate business requirements into scalable, reliable technical solutions.
• Good understanding of software engineering best practices, code quality, and technical documentation.
• Excellent communication and collaboration skills.
• Role Focus: Production Data Engineering, Python, PySpark, SQL, Apache Spark, ETL/ELT, Airflow/Kubeflow, AWS/Azure, Big Data.

Skills

Apache Spark, Data Pipelines, ELT, ETL, Pipelines, PySpark, Python, Spark