First seen by Alion on Nov 16, 2026.
Job Summary:
We are looking for a Senior Data Engineer to design, build, and operate our enterprise data platform on Azure and Databricks, with strong hands-on skills in GenAI/LLM application engineering. This role owns the full lifecycle of data pipelines and AI-powered applications from ingestion and orchestration to governance, API development, and deployment.
Key Responsibilities:
GenAI & LLM Application Engineering:
- Design and build RAG pipelines, including embedding-based and graph/metadata-grounded retrieval approaches.
- Orchestrate multi-step and multi-agent LLM workflows using LangGraph and LangChain.
- Build and integrate tools/agents using MCP for structured tool-calling and context management.
- Develop backend APIs for AI applications using FastAPI and Python Flask.
- Build and manage LLM/agent workflows using Azure AI Foundry, and implement semantic + vector search using Azure AI Search indexes.
- Implement prompt engineering, structured outputs, guardrails, and LLM evaluation/observability practices.
Data Engineering & Pipeline Development:
- Design and build scalable ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks (PySpark, Spark SQL).
- Develop and maintain Databricks Lakeflow (Delta Live Tables) pipelines across bronze/silver/gold medallion layers.
- Use Databricks Autoloader for incremental, schema-evolving file ingestion into Delta Lake.
- Build reusable, config-driven ingestion frameworks supporting multiple heterogeneous source systems.
- Implement CDC, SCD Type 1/2, and incremental loading strategies.
- Package and deploy Databricks workflows using Databricks Asset Bundles (DABs).
Platform, Infrastructure & DevOps:
- Own infrastructure provisioning and deployment using Terraform for Databricks, ADF, and related Azure resources.
- Build and maintain Azure DevOps CI/CD pipelines for automated testing, deployment, and release management.
- Deploy and manage applications on Azure Container Apps, using Docker for containerization.
- Develop and support Databricks Apps for internal data tooling.
Governance, Security & Reliability:
- Implement and maintain data governance using Databricks Unity Catalog.
- Design and enforce security policies across managed tables.
- Build monitoring, alerting, and SLA dashboards to ensure pipeline reliability and data quality.
- Perform Spark/Databricks performance tuning, cluster sizing, partitioning, and cost optimization.
Collaboration:
- Partner with data architects, analysts, and business stakeholders to translate requirements into scalable data and AI solutions.
- Document architecture, pipelines, and operational runbooks.
- Mentor junior engineers on Databricks, Python development, and platform engineering best practices.
Preferred Candidate Profile:
- Strong hands-on experience with Azure Data Factory, Databricks (PySpark, Spark SQL, Delta Lake, Autoloader, Lakeflow, DABs), and Unity Catalog.
- Proficiency in Databricks Apps, Azure Container Apps, Docker, and Terraform.
- Strong Python development skills, including API development with FastAPI and Flask.
- Hands-on experience with LLM application engineering (RAG, LangGraph, LangChain, MCP).
- Experience with Azure AI Foundry, Azure AI Search, and vector databases.
- Experience with CI/CD using Azure DevOps and API integration.
Skills
Data Engineering, Generative AI, Databricks, Python, Azure, LLM, PySpark, Terraform, LangChain, FastAPI

