Data Engineer

JOB DESCRIPTION

Experience: 5+ Years

Job Type: Full-Time

Department: Data Engineering

Location: Hybrid / Gurgaon

About the Role

We are looking for an experienced Data Engineer with 5+ years of hands-on experience in designing, developing, and optimizing scalable data pipelines and data processing solutions. The ideal candidate should have strong expertise in Python, PySpark, SQL, Apache Spark, Databricks, Delta Lake, and cloud platforms, along with experience in batch and real-time data processing, data orchestration, and data quality management.

The candidate will be responsible for building reliable ETL/ELT pipelines, implementing scalable Lakehouse architectures, optimizing data processing performance, and ensuring data accuracy, consistency, and availability for downstream analytics and business applications.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark, SQL, Apache Spark, and Databricks.
• Implement Lakehouse architectures and Medallion Architecture (Bronze, Silver, and Gold layers) using Delta Lake.
• Build and maintain batch and real-time data processing workflows using Apache Kafka and Spark Structured Streaming.
• Develop incremental data loading, merge/upsert operations, and efficient data transformation strategies to minimize processing time and resource utilization.
• Optimize Spark jobs through partitioning, caching, query optimization, and performance tuning techniques.
• Develop and manage workflow orchestration using Apache Airflow, including DAGs, dependencies, retries, monitoring, and failure notifications.
• Implement data quality frameworks covering schema validation, null checks, duplicate detection, and data consistency.
• Work with cloud data services, particularly AWS S3, AWS RDS, CloudWatch, and other relevant cloud data storage and processing services.
• Design and optimize relational databases, SQL queries, stored procedures, and database-level business logic.
• Monitor pipeline performance, troubleshoot production issues, and ensure data reliability, scalability, and availability.
• Implement version control, CI/CD practices, and controlled releases across development, testing, and production environments.
• Collaborate with cross-functional teams and stakeholders to understand business requirements and deliver effective data engineering solutions.
• Participate in code reviews, technical documentation, and mentoring junior team members.

Required Skills and Qualifications

• Minimum 5+ years of professional experience in Data Engineering or a related role.
• Strong programming skills in Python, PySpark, and SQL.
• Hands-on experience with Apache Spark, Databricks, and Delta Lake.
• Strong understanding of ETL/ELT pipeline development, data warehousing, data modeling, and Lakehouse architecture.
• Experience with Apache Kafka and real-time data processing using Spark Structured Streaming.
• Hands-on experience with Apache Airflow or similar workflow orchestration tools.
• Experience with AWS data services such as S3, RDS, and CloudWatch.
• Strong understanding of incremental data processing, data quality, and pipeline performance optimization.
• Experience working with relational databases such as PostgreSQL, SQL Server, or equivalent platforms.
• Familiarity with Git, GitHub, CI/CD pipelines, Agile methodologies, and deployment processes.
• Strong analytical, problem-solving, communication, and collaboration skills.

Preferred Qualifications

• Experience with Azure Data Lake Storage (ADLS) and Azure Data Factory.
• Familiarity with Talend or other ETL tools.
• AWS or Databricks certifications.
• Experience working with large-scale datasets, production data pipelines, and enterprise data platforms.
• Experience mentoring junior engineers and collaborating with business stakeholders

What we are looking for

We are looking for a hands-on Data Engineer who can independently design and optimize complex data pipelines, troubleshoot data processing challenges, improve platform performance, and deliver reliable data solutions. The ideal candidate should demonstrate strong technical ownership, a problem-solving mindset, and the ability to work effectively in a collaborative engineering environment.