Senior Data Engineer - Azure Databricks
First seen by Alion on Nov 5, 2026.
Senior Data Engineer - Azure Databricks & Microsoft Fabric
Role Overview:
We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Python, PySpark, SQL, Azure Databricks, and Microsoft Fabric. The candidate will be responsible for designing and maintaining scalable batch and real-time data pipelines, developing lakehouse architectures, and enabling reliable data solutions for analytics and AI/ML applications.
Key Responsibilities:
Data Pipeline & Lakehouse Engineering:
- Design, implement, and maintain high-performance batch and real-time streaming data pipelines using Python, PySpark, and Spark SQL. Build multi-tier Medallion architectures (Bronze, Silver, and Gold) using Delta Lake principles.
Data Platform Delivery:
- Leverage OneLake, Direct Lake, Lakehouses, and Data Factory Gen2 to streamline data ingestion, optimise semantic models, and reduce redundant ETL processes.
API & Systems Integration:
- Build resilient API connectors, Change Data Capture (CDC) workflows, and data integration pipelines to ingest data from multiple heterogeneous sources.
AI & Analytics Enablement:
- Structure, clean, and optimise lakehouse data layers for Power BI reporting and AI/ML applications, ensuring data is validated and structured for AI grounding.
Data Quality & Governance:
- Implement data validation, access controls, data lineage, and metadata management using Unity Catalog, platform governance capabilities, and automated testing frameworks.
Technical Mentorship & CI/CD:
- Conduct code reviews, enforce engineering best practices, optimise query performance, and manage automated deployment pipelines.
Required Technical Skills:
- Advanced hands-on proficiency in Python, PySpark, and SQL, including Spark SQL and T-SQL.
- Strong experience with Azure Databricks, including Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, and Genie.
- Demonstrated experience developing data pipelines and workspace components within Microsoft Fabric.
- Expertise in Azure cloud services, including ADLS Gen2, Azure Data Factory (ADF), Azure Key Vault, and Azure SQL / Azure Synapse.
- Strong understanding of data engineering requirements for AI/ML workloads, including knowledge of vector embeddings, RAG data ingestion, and feature stores.
Education:
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related quantitative field.
Skills
Data Engineering, Azure Databricks, Microsoft Fabric, Python, PySpark, SQL, Delta Lake, Azure Data Factory, CI/CD, RAG, Machine Learning