First seen by Alion on Nov 17, 2026.
Role Overview :
We are seeking a high-caliber Technical Lead to spearhead the design and development on our enterprise-scale Databricks data platform. As a hands-on lead, you will be the primary authority on our Azure-based Lakehouse, ensuring that data pipelines are not just functional, but scalable, secure, and cost-optimized. You must have a proven track record of leading end-to-end implementations - from initial discovery and schema design to production deployment and monitoring.
Core Requirements :
- 10 - 12 + years of experience in Data Engineering
- Azure Mastery: Minimum 3 + years of lead experience specifically within the Azure cloud ecosystem (ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault, Azure functions).
- Databricks Expertise: Deep hands-on experience with Databricks (Spark/PySpark), specifically implementing Medallion Architecture (Bronze, Silver, Gold layers).
- Conversant with streaming and batch data processing.
- Proven Leadership: Must have led at least two full-lifecycle, end-to-end data platform implementations in a Tech Lead capacity.
- Proven Tech Lead experience on both Databricks and Azure
- Excellent understanding of Lakehouse architecture
- Very strong SQL skills and data modeling expertise is a must.
- Strong experience with GenAI-assisted development. Should have used GitHub Copilot, Claude Code, Codex or equivalent; AI-assisted code generation, refactoring, testing, documentation and debugging using GenAI tools.
- Should be conversant with LLM APIs, RAG, embeddings, vector databases, structured outputs, model selection, latency and token-cost optimization.
- Should be familiar with agentic development and loop engineering.
Key Responsibilities :
- Architectural Design: Lead the design and evolution of our Delta Lakehouse architecture, ensuring it supports both high-volume and complex batch processing.
- End-to-End Implementation: Take full ownership of the data lifecycle: ingestion from disparate sources (APIs, RDBMS, NoSQL), complex transformations in PySpark, and final exposure via Gold tables or semantic layers.
- Technical Mentorship: Guide a team of 3 - 5 data engineers through code reviews, design workshops, and the establishment of "best-in-class" Python and SQL coding standards.
- Optimization: Conduct performance tuning of Spark clusters by exploring all available options like optimizing shuffle partitions and managing "Small File" problems to reduce Azure consumption costs.
- Data Governance: Implement and manage Databricks Unity Catalog for centralized fine-grained access control, lineage tracking, and data discovery.
- Handle production issues, root-cause analysis, and long-term fixes.
Technical Skills & Toolset :
- Languages: Expert-level Python (PySpark) and Advanced SQL. Familiarity with .net framework or Java is a plus.
- Orchestration: Advanced use of Azure Data Factory (ADF) and Databricks Workflows/Jobs for complex dependency management.
- DevOps/DataOps: Strong proficiency in Azure DevOps or GitHub Actions, specifically building CI/CD pipelines for automated Databricks notebook deployment and Infrastructure as Code (Terraform/Bicep).
- Observability: Implementation of data quality frameworks (e.g., Great Expectations or Databricks Expectations) and monitoring via Azure Monitor/Log Analytics.
Preferred Qualifications :
- Certifications: Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer Professional.
- AI/ML Integration: Experience supporting ML workloads by building feature stores and integrating with MLflow.
- Agile: Experience working in a high-velocity Scrum and Kanban environment.
Soft Skills :
- Strong ownership mindset and accountability
- Excellent communication with technical and non-technical stakeholders
Skills
Data Engineering, Databricks, Azure, Python, SQL, Azure Data Factory, Generative AI, LLM, RAG, CI/CD, Synapse Analytics

