Senior Lead GCP Databricks Platform Engineer

Role Overview :

This is a Senior Lead position focused on hands-on platform engineering, not an Architect role. The candidate will lead offshore implementation activities for the Databricks platform on GCP, including automation, configuration, governance, CI/CD pipeline development, FinOps support, monitoring, and operational documentation.

Key Responsibilities :

- Execute Databricks platform implementation tasks as per the approved project backlog.

- Configure Databricks workspaces, Unity Catalog, schemas, storage credentials, external locations, and other platform components.

- Develop and maintain Terraform modules, automation scripts, and reusable deployment assets.

- Build and manage CI/CD pipelines for notebooks, workflows, jobs, SQL assets, libraries, and Databricks Asset Bundles.

- Implement Unity Catalog configurations, including access controls, permissions, lineage validation, and governance settings.

- Configure and support Databricks SQL Warehouses, dashboards, and query performance optimization.

- Implement cluster policies, job policies, SQL warehouse policies, logging, monitoring, and health checks.

- Support FinOps activities by validating resource tags, analyzing usage, preparing cost reports, and identifying optimization opportunities.

- Troubleshoot workspace, cluster, job, SQL warehouse, deployment, permission, and connectivity issues.

- Prepare deployment guides, SOPs, runbooks, configuration documents, and knowledge transfer materials.

- Collaborate with onshore leads, customer platform teams, and internal delivery teams during implementation and handover.

- Provide L2/L3 platform support during rollout and stabilization.

Required Skills & Experience :

- 8+ years of experience in Data Platform Engineering, Cloud Engineering, DevOps, Analytics Engineering, or related roles.

- Strong hands-on experience with Databricks platform implementation and administration.

- Experience with :

1. Unity Catalog

2. Delta Lake

3. Databricks Workflows

4. Jobs & Clusters

5. SQL Warehouses

6. Permissions & Access Management

- Strong programming skills in Python, SQL, and scripting.

- Hands-on experience with Git, CI/CD pipelines, Databricks Asset Bundles, and deployment automation.

- Experience with Terraform or other Infrastructure as Code (IaC) tools.

- Good knowledge of Google Cloud Platform (GCP), including IAM, Service Accounts, GCS, Networking/VPC, and Monitoring.

- Understanding of platform governance, security, audit controls, and access management.

- Experience creating implementation guides, runbooks, operational documentation, and knowledge transfer materials.

- Ability to independently lead offshore execution while collaborating with onshore and customer teams.

Preferred Skills :

- Databricks Data Engineer Associate/Professional certification (or equivalent practical experience).

- Experience with MLflow, MLOps, Model Registry, or Model Serving.

- Exposure to FinOps, cost optimization, usage reporting, and dashboarding.

- Knowledge of enterprise data governance, platform observability, security controls, and production support.

- Experience supporting post-implementation stabilization and steady-state platform operations.

Skills

Data Engineering, Google Cloud Platform, Databricks, SQL, Python, Data Warehousing, Delta Lake