Lead Data and AI Engineer

Lead Data & AI Engineer

About the role

You'll be the technical lead for a client-facing AI product team, owning the systems that take GenAI use cases from prototype to production. This is a hands-on role: you'll architect and build agentic and RAG systems on Databricks and Azure, set the engineering standards the team works to, and sit with clients to translate messy requirements into systems that hold up under real usage. You'll mentor engineers and lead technically.

What you'll do

• Architect and build agentic AI systems - multi-step orchestration, tool use, retrieval, memory, and human-in-the-loop patterns using LangGraph and LangChain
• Design RAG pipelines end to end chunking and indexing strategy, vector/AI Search, retrieval quality tuning, grounding and citation behavior
• Own AI evaluation as a first-class discipline - golden datasets, offline and online eval harnesses, LLM-as-judge where appropriate, regression gates in CI, and ongoing quality, cost, and latency monitoring
• Productionize models and agents via Databricks Model Serving and AI Gateway, with MLflow for tracking, registry, and lineage
• Build FastAPI services that expose agent and model capabilities to client applications, with Pydantic contracts, auth, versioning, and sensible performance characteristics
• Build the data foundation underneath: Databricks pipelines, workflows, Delta tables, and compute/storage optimization
• Lead architecture and design reviews; make build-vs-buy calls and defend them to both engineers and clients
• Work directly with clients - scoping, demos, technical discovery, and setting realistic expectations about what GenAI will and won't do for their problem
• Mentor engineers through code review, pairing, and raising the bar on testing, observability, and CI/CD

Must have

• 6-8 years overall in Data and AI, including 3+ years building, scaling, and productionizing AI use cases in production - not proofs of concept
• Deep hands-on experience with LangChain, LangGraph, and Pydantic
• Demonstrated ownership of AI evaluation strategy for LLM systems
• Strong Python and SQL
• FastAPI and API development
• Databricks DE stack: pipelines, workflows, Delta tables, compute and storage optimization
• Databricks AI stack: Model Serving, AI Gateway, MLflow, AI/vector Search
• Azure: ADLS, App Service, Entra ID
• Comfort in client-facing conversations - able to explain trade-offs to non-engineers without oversimplifying

Good to have

• Kubernetes and Docker
• Jenkins CI/CD pipelines
• Databricks Asset Bundles
• Azure networking (VNets, private endpoints, NSGs)
• React or comparable frontend experience for internal tooling and demos

What we look for

Judgment about what survives contact with production. Skepticism toward agent architectures that are more complicated than the problem requires. The instinct to measure before claiming something works, and the willingness to tell a client when a simpler non-AI solution is the right answer.