Databricks Data Engineer

The Databricks Data Engineer will design, build and maintain scalable data pipelines on the Azure Databricks Lakehouse platform, supporting BI and analytics consumption across the business. The role involves re-engineering legacy ETL/ELT workflows into modern code-based pipelines, ensuring data quality through automated validation and reconciliation. This is a hybrid position based in Bengaluru, suited to an experienced engineer who thrives in a fast-paced, multi-project environment.

Our client is a leading global professional services and accounting firm operating in the finance and assurance space, serving clients across multiple industries worldwide. They are known for a collaborative, learning-driven culture and a strong commitment to technology-led transformation of data and analytics capabilities.

Responsibilities

• Design, develop and maintain data pipelines using Azure Databricks, PySpark and SQL
• Build and orchestrate workflows and jobs using Databricks Workflows and manage data governance with Unity Catalog
• Implement CI/CD practices for Databricks notebooks and pipelines using GitHub
• Model data for the Lakehouse architecture (Delta Lake), including dimensional modeling for BI consumption
• Re-engineer rule-based and visual-workflow ETL/ELT logic into code
• Develop automated data validation and reconciliation between legacy and migrated pipelines
• Collaborate with analysts, BI developers and stakeholders, and produce clear technical documentation
• Work in Agile teams to plan and deliver multiple projects simultaneously

Requirements

• 6-9 years of experience in data engineering or related roles
• Strong hands-on experience with Azure Databricks and SQL
• Strong experience in Python and PySpark, and in building and maintaining data pipelines
• Exposure to Databricks Workflows/Jobs orchestration, Unity Catalog and CI/CD for Databricks
• Experience with data modeling and ETL/ELT re-engineering on Delta Lake
• Experience with one or more public clouds such as Azure, AWS or GCP (Azure preferred)
• Bachelor's degree in Computer Science, Electronics & Communication Engineering or a related discipline; advanced degree preferred
• Good knowledge of GitHub and Agile methodologies; strong communication and documentation skills
• Nice to have: Tableau, SSRS or Alteryx experience, AI tools (Databricks Genie, Copilot, OpenAI) and Data Science/AI/ML exposure

Skills

Azure Databricks, SQL, Python, PySpark, Databricks Workflows/Jobs orchestration, Unity Catalog, CI/CD for Databricks notebooks/pipelines, Data pipeline development, Data modeling for Lakehouse architecture (Delta Lake), ETL/ELT re-engineering, GitHub, Agile methodologies, Tableau, SSRS, Alteryx, Databricks Genie, Claude, Copilot, OpenAI, GPTs, Data Science, AI/ML, Azure, AWS, GCP