Lead II - Data Engineering

We are looking for strong Lead Databricks Developer with streaming expertise. Key Responsibilities:

• Design, develop, and lead enterprise-scale data platforms and pipelines using Databricks, PySpark, Delta Lake, and Spark SQL.
• Build and optimize scalable batch and real-time data ingestion solutions using Kafka, Event Hubs, Auto Loader, Structured Streaming, and Delta Live Tables (DLT).
• Develop and manage Lakehouse architectures leveraging Medallion (Bronze, Silver, Gold) design patterns and data engineering best practices.
• Implement and govern enterprise data assets using Unity Catalog, including data lineage, metadata management, security policies, and access controls.
• Collaborate with architects and cloud teams to build and deploy data solutions on AWS and Azure, leveraging services such as S3, ADLS, Event Hubs, IAM, and cloud-native security capabilities.
• Drive data quality, observability, monitoring, and validation frameworks to ensure trusted, reliable, and business-ready data products.
• Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency through workload tuning and FinOps best practices.
• Lead cloud modernization and migration initiatives from legacy/on-premises platforms to modern Databricks Lakehouse architectures.
• Implement and support GenAI, RAG, AI Agents, Vector Search, Mosaic AI, MLflow, and Databricks Genie use cases to accelerate analytics and business insights.
• Establish CI/CD, DataOps, Infrastructure as Code (Terraform), and DevOps best practices while providing technical leadership, mentoring team members, and ensuring delivery excellence. Domain Experience:
• Experience in Healthcare Payer domain (Claims, Membership, Coverage)
• Exposure to FHIR / healthcare data standards preferred
• Understanding of data privacy, security, and governance requirements (PII/HIPAA)
• Leadership Expectations
• Operate at a Lead level driving design, development, and delivery
• Mentor team members and ensure delivery quality
• Own modules independently and collaborate across distributed teams Experience:
• 8+ years in Data Engineering
• 5+ years hands-on experience with Databricks
• Prior experience in Tech Lead/Senior roles