Senior Data Engineer
First seen by Alion on Sep 17, 2026.
Role : Senior Data Engineer - Azure, Databricks & GenAI
About the Role :
We are looking for a Senior Data Engineer to design, build, and operate an enterprise-scale data platform on Microsoft Azure and Databricks, with strong hands-on experience in Generative AI, LLMs, and RAG-based applications.
The role will own the end-to-end lifecycle of data pipelines and AI-powered applications, working closely with engineering, product, and business teams to build scalable, reliable, and production-ready data and AI solutions.
Key Responsibilities :
- Design, develop, and maintain scalable data pipelines and data processing solutions using Azure and Databricks.
- Build and optimize data engineering workflows using PySpark and Python.
- Develop and manage enterprise data platforms, ensuring scalability, reliability, security, and performance.
- Design and implement ETL/ELT pipelines for structured and unstructured data.
- Build GenAI/LLM-powered applications, including RAG (Retrieval-Augmented Generation) solutions.
- Work with LLMs, embeddings, vector databases, prompt engineering, and retrieval pipelines.
- Integrate LLM capabilities with enterprise data and business applications.
- Develop, deploy, monitor, and optimize AI-powered applications in production.
- Implement data quality, validation, monitoring, and governance practices across pipelines.
- Optimize Databricks workloads and Spark jobs for performance and cost efficiency.
- Collaborate with cross-functional teams to translate business requirements into scalable data and AI solutions.
- Troubleshoot production issues and ensure high availability and reliability of data and AI workloads.
- Stay updated with emerging technologies across Data Engineering, GenAI, LLMs, and Azure.
Required Skills & Experience :
- Strong hands-on experience in Data Engineering and building enterprise-scale data platforms.
- Strong proficiency in Python and PySpark.
- Hands-on experience with Azure data services and Databricks.
- Experience designing and developing ETL/ELT data pipelines.
- Strong understanding of data engineering concepts, distributed computing, and data architecture.
- Hands-on experience with Generative AI, LLMs, and RAG architectures.
- Experience working with LLM application development, embeddings, vector search, and retrieval pipelines.
- Understanding of data modeling, data quality, and data governance.
- Experience with cloud-based data platforms and production-grade engineering practices.
- Strong problem-solving and analytical skills.
- Ability to work independently and collaborate effectively with technical and business stakeholders.
Preferred Skills :
- Experience with Azure services such as Azure Data Factory, Azure Data Lake, Azure OpenAI, and Azure AI services.
- Experience with MLflow and Databricks-based AI/ML workflows.
- Experience with vector databases and semantic search.
- Knowledge of LLM frameworks and orchestration tools.
- Experience deploying AI applications using APIs and microservices.
- Understanding of CI/CD, DevOps, monitoring, and cloud security.
Education :
- B.Tech / B.E. in Computer Science, Information Technology, Engineering, or a related field.
Skills
Data Engineering, Azure, Azure Databricks, Python, PySpark, Generative AI, RAG, ETL, Data Modeling, Data Quality, Data Governance