Senior Data Engineer - Databricks & AI Solutions
First seen by Alion on Sep 17, 2026.
Role Overview:
Seeking a highly skilled Senior Data Engineer with deep expertise in Databricks, Real-Time Data Processing, Lakehouse Architecture, and AI-driven solutions. The role involves designing, developing, and supporting scalable data platforms and streaming pipelines, leveraging modern Databricks capabilities such as Lakeflow, Unity Catalog, and Generative AI technologies.
Key Responsibilities:
- Design, develop, and maintain end-to-end streaming and batch data pipelines using Databricks and PySpark.
- Build, optimize, and manage Delta Lake tables within Unity Catalog.
- Develop and support Lakeflow pipelines with incremental processing, CDC, and data quality validations.
- Integrate enterprise systems into the Databricks Lakehouse.
- Ensure data quality, governance, security, and operational excellence across platforms.
- Monitor, troubleshoot, and optimize workflows for reliability and performance.
- Collaborate with stakeholders to deliver scalable, business-aligned solutions.
- Build and support AI-powered data solutions leveraging LLMs, RAG, and vector search.
- Implement CI/CD best practices and maintain deployment pipelines.
- Drive deliverables independently, proactively resolving challenges with minimal supervision.
Tech Stack & Requirements :
- Hands-on experience with Databricks and modern Lakehouse architecture.
- Expertise in PySpark, Spark Structured Streaming, SQL, NoSQL databases, Delta Lake.
- Proficiency with Unity Catalog for governance and security.
- Experience building AI agents, copilots, and generative AI solutions using LLMs, RAG, and vector search.
- Experience with Git-based source control, CI/CD pipelines, Databricks Repos, and Databricks Asset Bundles.
Educational Qualifications :
- BE/BTech/ME/MTech/MCA only.
Skills
Data Engineering, Generative AI, PySpark, Data Quality, SQL, NoSQL, LLM