Lead Data Engineer
First seen by Alion on Aug 20, 2026.
Role: Lead Data Engineer
Location: Chennai (Work from Office)
Experience: 8 - 12 years
About the Role:
We are looking for an experienced Lead Data Engineer to drive the design, development, and management of our data infrastructure while leading a team of 4 engineers. This role combines strong hands-on technical expertise with people management and delivery ownership and requires someone comfortable using modern AI-assisted ("vibe coding") tools to rapidly build and ship data products, while maintaining the infrastructure that powers them.
Key Responsibilities:
Technical Leadership & Development:
- Architect, build, and optimize scalable data pipelines using Databricks, PySpark, and Python.
- Write and review complex SQL queries for data transformation, validation, and performance tuning.
- Design and maintain data models and ETL/ELT workflows across Databricks environments.
- Support or lead development of ML models/pipelines for data products (feature engineering, model training, deployment, and monitoring).
- Build data product applications using AI-assisted "vibe coding" tools (Claude Code, Cursor, or similar) to rapidly prototype, develop, and ship internal tools, dashboards, and data-driven applications.
- Establish and enforce best practices for code quality, version control, and CI/CD using Git.
- Champion the adoption of AI-assisted development tools across the team to improve developer velocity, code review efficiency, and documentation.
- Conduct code reviews, architecture reviews, and technical design sessions.
Infrastructure & Platform Ownership:
- Own and maintain the infrastructure underlying data products including compute clusters, transactional databases, job orchestration, environments, deployment pipelines, and monitoring.
- Ensure data pipeline reliability, observability, and cost optimization across Databricks.
- Manage access controls, environment configuration, and platform scaling for data product apps.
- Troubleshoot infrastructure issues end-to-end (compute, storage, networking, connectivity).
Team & People Management:
- Lead, mentor, and grow a team of 4 engineers, providing technical guidance and career development support.
- Conduct 1:1s, performance reviews, and skill-development planning for team members.
- Foster a culture of continuous learning and rapid upskilling in response to evolving tech/business needs.
- Encourage and enable the team to adopt new tools and frameworks quickly.
Delivery & Process Management:
- Own sprint planning, backlog grooming, and sprint retrospectives for the team (Agile/Scrum).
- Break down business requirements into well-scoped technical tasks and user stories.
- Track sprint velocity, manage blockers, and ensure timely delivery of data engineering initiatives.
- Coordinate with cross-functional stakeholders (analytics, product, business teams) to align priorities.
Required Skills & Experience:
- 8+ years of experience in data engineering, with at least 2+ years in a technical leadership or team lead role.
- Strong hands-on expertise in Databricks (notebooks, clusters, job orchestration, Delta Lake).
- Proficiency in PySpark and Python for large-scale data processing.
- Advanced SQL skills (query optimization, stored procedures, performance tuning).
- Experience with Microsoft SQL Server (T-SQL, database design, troubleshooting).
- Experience with OCR data extraction (Azure Document Intelligence, AWS Textract).
- Solid understanding of Git workflows (branching strategies, PR reviews, merge conflict resolution).
- Hands-on experience building applications using AI coding tools / vibe coding (Claude Code, Cursor, GitHub Copilot, or similar).
- Experience maintaining and troubleshooting infrastructure supporting data products (compute, orchestration, deployment).
- Working knowledge of Machine Learning concepts and lifecycle (model training, deployment, monitoring).
- Proven experience running Agile sprint planning, backlog management, and team ceremonies.
- Experience managing and mentoring a small engineering team (3-5 members).
- Demonstrated ability to quickly learn and adopt new technologies in response to business needs.
Preferred Qualifications:
- Experience with cloud platforms (Azure preferred, Databricks stack).
- Familiarity with CI/CD pipelines (Azure DevOps, GitHub Actions, etc.).
- Experience with data governance frameworks and data quality tools.
- Bachelor's/Master's degree in Computer Science, Engineering, or related field.
Soft Skills:
- Strong communication skills to bridge technical and non-technical stakeholders.
- Ability to balance hands-on coding with people management responsibilities.
- High adaptability - comfortable navigating ambiguity and fast-changing tech/business priorities.
- Problem-solving mindset with a focus on scalable, maintainable solutions.
Skills
Data Engineering, Azure Databricks, PySpark, Python, SQL, Data Infrastructure, Vibe Coding, SQL Server, T-SQL