Lead Data engineer

HIRING: Lead Data Engineer – AI / Azure Databricks

Location: Cary, NC (On-site / Hybrid)
Employment Type: Full-Time
Experience: 12–18 Years

Any Visa

ABOUT THE ROLE

We are looking for a hands-on Lead Data Engineer to architect and build an enterprise-scale, AI-first Data Hub using Azure Databricks. The role combines advanced data engineering, lakehouse architecture, production-grade AI/LLM solutions, and technical leadership.

KEY RESPONSIBILITIES

• Design enterprise lakehouse architecture using Azure Databricks, ADLS Gen2, Delta Lake, and Bronze/Silver/Gold layers.
• Build metadata-driven ingestion frameworks and scalable ETL pipelines using Python, Scala, and PySpark.
• Optimize Spark jobs, cluster performance, partitioning, and processing costs.
• Develop batch, CDC, and streaming pipelines using Azure Data Factory, Event Hubs, Kafka, and Spark Structured Streaming.
• Build AI-assisted source-to-canonical mapping, automated data quality, anomaly detection, and reconciliation solutions.
• Design production LLM applications using RAG, agentic workflows, LangChain, LangGraph, or LlamaIndex.
• Implement semantic layers, knowledge graphs, and conversational data access where applicable.
• Establish CI/CD, automated testing, governance, security, and data quality standards.
• Lead architecture reviews, mentor engineers, and contribute hands-on production code.

REQUIRED SKILLS

• 12–18 years of data engineering or data platform experience.
• Expert Python, Scala, PySpark, SQL, and Apache Spark.
• Strong Azure Databricks, Delta Lake, Unity Catalog, and ADLS Gen2.
• Hands-on Azure Data Factory, Azure Event Hubs, and metadata-driven ingestion.
• 3+ years building production LLM solutions, including RAG, retrieval, embeddings, and agentic/tool-calling workflows.
• LangChain, LangGraph, or LlamaIndex with Azure OpenAI, OpenAI, or Databricks Model Serving.
• Azure DevOps, Terraform, Databricks Asset Bundles, and CI/CD.
• Strong Azure security, governance, data modelling, and Spark performance tuning.
• Proven enterprise-scale lakehouse architecture and technical leadership.

PREFERRED SKILLS

Knowledge graphs, text-to-SQL, AI-driven anomaly detection, LLMOps/MLOps, dbt, Great Expectations, and insurance or financial services experience.

Interested candidates, please share your resume at: