Senior Data Engineer - Databricks
First seen by Alion on Oct 27, 2026.
Position Overview:
Datakulture, Sedin's Data Analytics & Business Intelligence division, is looking for a highly experienced Senior Data Engineer with 8+ years of hands-on expertise, with a strong focus on Databricks, to join our Chennai team in an in-office capacity. In this role, you will architect and build enterprise-scale data platforms, Lakehouse environments, and complex data pipelines that power analytics and AI workloads for global clients.
Key Responsibilities:
- Architect, design, and build large-scale data pipelines and Lakehouse solutions on Databricks using PySpark, Spark SQL, and Delta Lake, following Medallion Architecture principles.
- Lead the design and implementation of batch and streaming data pipelines using Spark Structured Streaming, Apache Kafka, AWS Kinesis, or Azure Event Hubs.
- Develop and manage Databricks Workflows, Unity Catalog, Delta Live Tables (DLT), and cluster configurations for optimal performance and governance.
- Engineer robust data ingestion frameworks from diverse sources including RDBMS, REST APIs, flat files, NoSQL stores, and event streams.
- Design and optimize data models for analytical workloads; implement advanced performance tuning including Z-ordering, partitioning, compaction, and caching strategies.
- Build and maintain cloud data infrastructure on AWS and/or Azure alongside Databricks.
- Implement CI/CD pipelines for data engineering workflows; enforce code review standards and Git-based development practices.
- Define and enforce data quality frameworks, data observability standards, and data governance policies.
- Collaborate with data scientists, ML engineers, and BI developers to deliver feature stores, analytical datasets, and ML-ready data pipelines.
- Lead technical discussions, architecture reviews, and stakeholder engagements; mentor and guide junior and mid-level data engineers.
- Drive adoption of Infrastructure as Code (IaC) using Terraform or CloudFormation; manage cost optimization and security best practices.
Required Qualifications & Skills:
- B.E. / B.Tech / M.Tech / M.Sc. in Computer Science, Information Technology, Data Engineering, or a related discipline.
- 8+ years of proven experience in data engineering, with at least 3 - 4 years of deep, hands-on Databricks expertise in production environments.
- Databricks (Expert): PySpark, Spark SQL, Delta Lake, Unity Catalog, Delta Live Tables, Databricks Workflows, cluster tuning, and Databricks SQL.
- Cloud Platforms: Strong proficiency in AWS (S3, Glue, Athena, EMR, Redshift, Kinesis, Lambda, Step Functions) and/or Azure (ADLS Gen2, ADF, Synapse Analytics, Azure SQL, Event Hubs).
- Programming: Advanced Python (pandas, PySpark) and SQL (complex queries, window functions, performance optimization).
- Strong communication and leadership skills; demonstrated ability to drive technical decision-making, engage with senior stakeholders, and mentor engineering teams.
- Data Architecture: Deep understanding of Medallion Architecture, Lakehouse design, Data Vault, star/snowflake schema, and streaming data patterns.
- DevOps & IaC: Proficiency in CI/CD (GitHub Actions, Azure DevOps, Jenkins), Terraform or CloudFormation, and Git-based development workflows.
- Data Governance: Experience implementing data quality frameworks, data observability, and Unity Catalog-based governance.
Preferred Qualifications:
- Databricks Certified Data Engineer Professional or Associate; AWS Certified Data Engineer or Azure Data Engineer Associate.
- Experience delivering data platforms across BFSI, retail, healthcare, manufacturing, or other data-intensive industries.
- Exposure to dbt, Apache Iceberg, Apache Flink, Airflow/Prefect, or ML platform tooling (MLflow, Feature Store).
Skills
Data Engineering, Databricks, Apache Spark, Python, Delta Lake, AWS, Azure, Medallion Architecture, Cloud Infrastructure, PySpark