Data Engineer - Databricks/Spark
First seen by Alion on Nov 4, 2026.
Role : Data Engineer
Experience : 4 - 6 years
Location : Pune & Gurugram
Role and responsibilities :
- Design, build, and maintain robust, scalable, and high-performance ETL/ELT pipelines.
- Develop and optimize data processing workflows on Databricks.
- Partnering with data architects, analysts, and business stakeholders to translate data requirements into reliable, production-grade pipelines.
- Drive end-to-end data engineering across project phases - ingestion, transformation, quality, and delivery, ensuring alignment with business objectives and data governance.
- Implement and enforce data quality checks, validation frameworks, and monitoring/alerting for pipeline health.
- Adhere to defined data engineering standards, guidelines, and best practices across projects.
Required Experience & Skills :
- 4 - 6 years of hands-on experience designing and building enterprise-scale data pipelines and ETL/ELT solutions.
- Strong hands-on expertise with Databricks : notebooks, jobs/workflows, delta tables, unity catalog.
- Deep proficiency in Apache Spark (PySpark and/or Spark SQL) for data processing.
- Strong programming skills in Python and advanced SQL for data transformation and analysis.
- Solid understanding of data warehousing concepts, dimensional modeling, and the medallion (bronze/silver/gold) lakehouse architecture.
- Demonstrated ability to troubleshoot, tune, and optimize jobs for performance and cost benefits.
- Experience working in agile environments with CI/CD for data pipelines.
- Excellent communication and documentation skills.
Tech Stack & Expertise :
- Databricks platform : Delta Lake, Delta Live Tables (DLT), Unity Catalog, Databricks SQL, Workflows.
- ETL/ELT and orchestration.
- Data formats and storage : Parquet, JSON, Delta.
- Data quality frameworks.
- Cost optimization of data workloads : cluster right-sizing, job scheduling, and compute.
Skills
Data Engineering, Databricks, Apache Spark, Python, SQL, ETL, Data Warehousing, Data Catalogue