Data Engineering Lead - Azure Databricks
First seen by Alion on Aug 19, 2026.
Role Overview:
We are looking for an experienced Data Engineering Lead who can combine strong technical leadership with hands-on engineering expertise. The role will be responsible for driving the design, development, and maintenance of scalable data pipelines and platforms while remaining actively involved in technical implementation. The ideal candidate should have strong expertise in Databricks, Spark, PySpark, Python, SQL, and Data Architecture, along with the ability to make technical decisions, establish engineering standards, mentor team members, and solve complex production challenges.
Key Responsibilities:
Technical Leadership:
- Provide technical leadership for the design, development, and maintenance of scalable data pipelines and data platforms.
- Act as a technical SME for Databricks, Spark, PySpark, Python, SQL, and Data Architecture.
- Drive technical design decisions and establish scalable and effective engineering patterns.
- Conduct code reviews and design reviews to maintain technical quality and consistency.
- Define and enforce coding standards, development guidelines, and engineering best practices.
- Mentor and guide Data Engineers and Senior Data Engineers.
- Identify technical risks, dependencies, bottlenecks, and opportunities for improvement.
- Monitor and optimize data infrastructure performance.
- Lead troubleshooting and resolution of complex technical and production issues.
- Ensure appropriate testing, deployment, release, and rollback processes are followed.
- Collaborate with architecture and product teams to translate business requirements into effective technical solutions.
Individual Contributor Responsibilities:
- The Data Engineering Lead will remain approximately 60% hands-on / individual contributor, balancing technical leadership with direct engineering contribution.
- Participate directly in architecture and technical design.
- Contribute to coding and development activities.
- Perform debugging and troubleshoot complex technical issues.
- Drive performance optimization of data engineering solutions.
- Participate in code and design reviews.
- Contribute to production support and issue resolution.
- Remain actively involved in hands-on delivery while providing technical direction to the team.
Technical Skills & Requirements:
- Strong expertise in Databricks.
- Strong hands-on experience with Apache Spark and PySpark.
- Strong proficiency in Python and SQL.
- Strong understanding of Data Architecture.
- Experience designing and developing scalable data pipelines and platforms.
- Strong technical design and problem-solving capabilities.
- Experience conducting code reviews and design reviews.
- Ability to establish and enforce engineering standards and best practices.
- Experience mentoring and guiding Data Engineers and Senior Data Engineers.
- Strong understanding of production support, troubleshooting, and performance optimization.
- Ability to balance technical leadership with hands-on engineering.
Role Expectations:
- This role is suited for a technical leader who prefers to remain closely involved in engineering rather than moving completely into people management.
- The successful candidate will provide technical direction, support the development team, make architecture and design decisions, and continue contributing directly to coding, debugging, optimization, reviews, and production support.
Skills
Data Engineering, Azure Databricks, Apache Spark, Python, SQL, ETL, PySpark, Data Pipeline