Databricks Tech Lead

The Databricks Tech Lead will design, build, and optimize enterprise-scale data lakehouse solutions on the Databricks platform. You will lead the delivery of financial crime platforms covering AML, KYC, customer risk assessment, sanctions screening, transaction monitoring, fraud detection, and regulatory reporting. This is a hands-on technical leadership role with ownership of architecture, governance, and delivery quality.

Our client is a leading global technology and analytics company operating in the computer software industry at significant scale. They serve large enterprises with data-driven solutions and are known for a collaborative, engineering-led culture that values innovation and continuous learning.

Responsibilities

• Design, build, and maintain Databricks workspaces, clusters, and jobs across dev, test, and prod environments, including Unity Catalog governance and access control.
• Design Delta Lake tables and Medallion (Bronze/Silver/Gold) architectures, and implement Delta Live Tables pipelines with data quality expectations.
• Develop scalable batch and streaming pipelines using PySpark, Spark SQL, and Structured Streaming from Kafka, Event Hubs, or Kinesis.
• Support MLflow, Feature Store, and model serving workloads, including GenAI and RAG use cases.
• Build CI/CD pipelines and infrastructure-as-code using Databricks Asset Bundles or Terraform, integrated with Azure DevOps, GitHub Actions, or GitLab CI.
• Implement row-level security, column masking, performance tuning, and cost monitoring.
• Document architecture, design decisions, and operational runbooks.
• Lead and mentor engineers, driving code quality, best practices, and delivery standards.

Requirements

• 8+ years of total data or software engineering experience, with 5+ years of hands-on Databricks experience in production (3+ years minimum).
• Deep expertise in Databricks Workspaces, Jobs, Workflows, Unity Catalog, Delta Lake internals, and Delta Live Tables.
• 4+ years of PySpark development and expert-level SQL, including Spark performance tuning (AQE, partitioning, caching, query plans).
• Proficiency in Python for pipeline development and automation.
• Hands-on experience with at least one cloud platform (Azure ADLS Gen2/ADF/Azure Databricks, AWS S3/EMR/Glue, or GCP GCS/BigQuery/Dataproc), including networking, IAM, and service principal authentication.
• Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.
• Nice to have: working knowledge of MLflow, dbt, Kafka/Confluent, Terraform, Apache Iceberg or Hudi, and BI tools such as Power BI, Tableau, or Looker.
• Nice to have: Databricks certifications (Data Engineer Associate/Professional, Machine Learning, or Apache Spark Developer) or cloud data engineering certifications.

Skills

Databricks, Unity Catalog, Delta Lake, Delta Live Tables, PySpark, Spark SQL, Python, Structured Streaming, Cloud platform (Azure, AWS, or GCP), MLflow, Feature Store, dbt, Kafka/Confluent, Terraform, Apache Iceberg, Apache Hudi, Power BI, Tableau, Looker