Databricks Platform Engineering Lead / Technical Delivery Lead

Job Description

• Skill Set :

  azure databricks

• Total Experience :

  8.00 to 10.00 Years

• No of Openings :

  1

• Job Post Date :

  30/10/2026

• Job Expiry Date :

  30/12/2026

• Domain :

  IT

• Location :

  IRVINE [United States]

• Job Reference No :

  4115685

Job Summary

Databricks Platform Engineering Lead / Technical Delivery Lead

Warren, NJ

Project Overview

We are seeking an experienced Databricks Platform Engineering Lead to support an enterprise-wide pricing application within a global reinsurance organization. The application leverages Databricks as its data and analytics platform and integrates with a SQL Server-based pricing application.

The resource will lead the engineering, governance, and ongoing enhancement of the Databricks platform supporting the pricing application. The role requires a strong combination of hands-on platform engineering, Databricks administration, data engineering, and technical delivery leadership.

The ideal candidate must be capable of managing the Databricks platform foundation while providing technical oversight to data-engineering teams building ETL/ELT pipelines, transformations, and pricing data products.

Technical and Delivery Expertise

• 10+ years of overall technology experience.
• 5+ years of hands-on experience with Databricks, cloud data platforms, or platform engineering.
• 3+ years of experience in a Platform Lead, Databricks Lead, Cloud Platform Engineer, Data Platform Lead, or similar role.
• Strong hands-on experience with:
  • Azure Databricks
  • Databricks account and workspace administration
  • Unity Catalog
  • Apache Spark and PySpark
  • Python and SQL
  • Delta Lake and Lakehouse architecture
  • Azure Data Lake Storage
  • Azure Data Factory or comparable integration tools
  • Databricks Workflows and job orchestration
  • Infrastructure as code using Terraform
  • CI/CD and automated deployment practices
• Strong understanding of:
  • Databricks workspace provisioning and administration
  • Development, test, and production environment management
  • Compute provisioning and cluster configuration
  • Cluster policies, pools, runtimes, and serverless compute
  • Platform security, identity, and access management
  • Service principals, managed identities, and secrets management
  • Azure networking, private endpoints, and secure connectivity
  • Unity Catalog architecture, permissions, lineage, and governance
  • Platform monitoring, logging, auditability, and observability
  • Databricks performance, capacity, and cost optimization
  • Production support and platform-incident management
• Good understanding of data-engineering practices, including:
  • Batch and streaming data ingestion
  • ETL/ELT pipeline development
  • Medallion architecture
  • Data transformations and enrichment
  • Data quality, validation, and reconciliation
  • Schema management and data lineage
  • Spark workload and query optimization
• Experience leading globally distributed platform, data-engineering, application, QA, DevOps, and production-support teams.
• Ability to participate in architecture reviews, solution-design discussions, code reviews, security assessments, and production-readiness evaluations.
• Strong understanding of Agile delivery, release management, risk management, dependency management, and stakeholder governance.

Preferred Qualifications

• Prior experience implementing a Databricks-enabled pricing, underwriting, actuarial, insurance analytics, or financial-services application.
• Experience working with global insurance or reinsurance carriers.
• Understanding of reinsurance pricing, underwriting, treaty, and facultative business processes.
• Experience integrating Databricks with:
  • Pricing engines
  • SQL Server applications
  • Policy, claims, and accounting systems
  • Data warehouses and reporting platforms
  • Internal and third-party data sources
• Experience establishing reusable Databricks platform standards, engineering frameworks, and reference architectures.
• Experience implementing a hybrid solution of Databricks and SQL
• Experience defining the division of responsibility among platform engineering, data engineering, application development, DevOps, data science, and production-support teams.
• Familiarity with MLflow, model lifecycle management, model serving, or actuarial-model integration.
• Experience implementing data governance, metadata management, lineage, role-based access, audit controls, and regulatory controls.
• Familiarity with Azure DevOps, Terraform, Power BI, Microsoft Purview, Azure Monitor, or comparable tools.
• Databricks and Microsoft Azure certifications are preferred.
• Experience supporting business-critical production platforms with defined availability, recovery, security, and performance requirements.

The Resource Will Be Expected To:

• Own the engineering and operational management of the Databricks platform supporting the enterprise pricing application.
• Lead end-to-end technical delivery of platform enhancements, environment changes, integrations, releases, and production improvements.
• Provision and administer Databricks accounts, workspaces, compute resources, clusters, policies, and supporting Azure services.
• Define and enforce platform standards covering architecture, security, governance, compute, deployment, monitoring, and cost management.
• Design and maintain appropriate separation across development, test, UAT, and production environments.
• Implement infrastructure as code and automated deployment practices for Databricks and supporting cloud resources.
• Establish secure connectivity between Databricks, SQL Server, cloud storage, the pricing engine, the underwriting workbench, and other enterprise systems.
• Configure and maintain identity, access, security, secrets, networking, and governance controls.
• Define Unity Catalog structures, access models, data ownership, lineage, and auditability standards.
• Establish cluster policies, runtime standards, compute controls, and workload-management practices.
• Monitor and optimize platform availability, reliability, performance, capacity, and cost.
• Troubleshoot workspace, authentication, networking, compute, storage, access, and platform-performance issues.
• Lead root-cause analysis and corrective actions for platform incidents and recurring operational problems.
• Provide technical oversight to data engineers building ingestion pipelines, transformations, curated datasets, and pricing data products.
• Review pipeline designs, Spark workloads, data-processing patterns, deployment approaches, and production-readiness plans.
• Ensure that data-engineering solutions comply with platform, security, governance, quality, and performance standards.
• Coordinate platform engineering, data engineering, application development, QA, DevOps, infrastructure, security, and production-support teams.
• Independently identify technical and delivery impacts across Databricks, the SQL Server pricing engine, the underwriting workbench, and integrated systems.
• Support successful SIT, UAT, regression testing, release validation, deployment, cutover, and post-production stabilization.
• Communicate technical risks, delivery dependencies, architectural decisions, and recovery plans to business and technology stakeholders.
• Serve as the primary technical point of contact for the Databricks platform and its integration with the enterprise pricing application.
• Serve as a subject matter expert on Databricks platform engineering, administration, governance, security, reliability, and operational best practices.

The pay range for this role is $120k - $140k per annum including any bonuses or variable pay. Tech Mahindra also offers benefits like medical, vision, dental, life, disability insurance and paid time off (including holidays, parental leave, and sick leave, as required by law). Ask our recruiters for more details on our Benefits package. The exact offer terms will depend on the skill level, educational qualifications, experience, and location of the candidate.

Thanks & Regards

Tech Mahindra is an Equal Employment Opportunity employer. We promote and support a diverse workforce at all levels of the company. All qualified applicants will receive consideration for employment without regard to race, religion, color, sex, age, national origin or disability. All applicants will be evaluated solely on the basis of their ability, competence, and performance of the essential functions of their positions with or without reasonable accommodations. Reasonable accommodations also are available in the hiring process for applicants with disabilities. Candidates can request a reasonable accommodation by contacting the company ADA Coordinator at [email protected].”