Senior Data Engineer

Role: Sr. Data Engineer

Location: Gurugram & Bangalore

Work Mode: Work from Office [5 Days from office]

Role Overview:

The Databricks Data Engineer will be responsible for designing, building, and optimizing scalable data pipelines and lakehouse solutions using Databricks. The role requires strong hands-on experience in data engineering, distributed data processing.

Key Role and Responsibilities:

• Design, build, and maintain ETL/ELT pipelines on Databricks using PySpark, Spark SQL, and Delta Lake.
• Develop and optimize data ingestion frameworks, data transformations, and end-to-end workflows for batch and streaming use cases.
• Implement Delta Lake-based architectures, including versioning, schema evolution, and ACID-compliant pipelines.
• Work with stakeholders to understand data requirements and translate them into scalable data engineering solutions.
• Manage and optimize Databricks clusters, jobs, and notebooks for performance and cost efficiency.
• Integrate Databricks pipelines with cloud services (Azure Data Lake / AWS S3, Event Hub/Kafka, Synapse/Redshift, etc.).
• Ensure data quality, reliability, and observability through validation frameworks and monitoring.
• Contribute to data modeling, metadata management, and best practices within the data platform.
• Collaborate closely with data scientists, analysts, and business teams to support analytics and ML workloads.

Must Have:

• 8+ years of experience in data engineering with strong expertise in Databricks.
• Hands-on experience with Spark (PySpark/Spark SQL) and distributed data processing.
• Solid SQL knowledge and experience working with large-scale datasets
• Strong understanding of Delta Lake, medallion architecture, and scalable lakehouse patterns.
• Good understanding of CI/CD, Git, and modern DevOps practices for data pipelines.
• Familiarity with structured/unstructured data, data quality frameworks, and performance tuning. Education: Bachelor’s degree in computer science, Software Engineering, MIS or equivalent combination of education and experience

Key Skills: Databricks, Pyspark, SQL

• Design, build, and maintain ETL/ELT pipelines on Databricks using PySpark, Spark SQL, and Delta Lake.
• Develop and optimize data ingestion frameworks, data transformations, and end-to-end workflows for batch and streaming use cases.
• Implement Delta Lake-based architectures, including versioning, schema evolution, and ACID-compliant pipelines.
• Work with stakeholders to understand data requirements and translate them into scalable data engineering solutions.
• Manage and optimize Databricks clusters, jobs, and notebooks for performance and cost efficiency.
• Integrate Databricks pipelines with cloud services (Azure Data Lake / AWS S3, Event Hub/Kafka, Synapse/Redshift, etc.).
• Ensure data quality, reliability, and observability through validation frameworks and monitoring.
• Contribute to data modeling, metadata management, and best practices within the data platform.
• Collaborate closely with data scientists, analysts, and business teams to support analytics and ML workloads.
• 8+ years of experience in data engineering with strong expertise in Databricks.
• Hands-on experience with Spark (PySpark/Spark SQL) and distributed data processing.
• Solid SQL knowledge and experience working with large-scale datasets
• Strong understanding of Delta Lake, medallion architecture, and scalable lakehouse patterns.
• Good understanding of CI/CD, Git, and modern DevOps practices for data pipelines.
• Familiarity with structured/unstructured data, data quality frameworks, and performance tuning. Education: Bachelor’s degree in computer science, Software Engineering, MIS or equivalent combination of education and experience