Databricks Platform Engineer

Role Overview :

We are seeking a hands-on Databricks Platform Engineer to provide offshore operational support, platform administration, automation, monitoring, and continuous improvement for an enterprise Databricks Lakehouse platform on AWS.

The engineer will work closely with the onshore Senior Data Platform Engineer while independently managing assigned platform engineering and operational responsibilities.

The role focuses on platform reliability, automation, incident management, security, governance, cost optimization, and supporting Data Engineering and Data Science teams.

Key Responsibilities :

- Provide day-to-day Databricks platform administration and production operational support.

- Monitor platform cost, usage, capacity, performance, and resource utilization.

- Identify and recommend opportunities for Databricks and AWS cost optimization.

- Implement approved platform architecture, engineering standards, security controls, and operational processes.

- Monitor platform health and respond to incidents, service requests, and operational alerts.

- Troubleshoot and restore platform services across Databricks, AWS, networking, security, compute, and workloads.

- Develop and maintain automation using Terraform, Databricks REST APIs, CLI, Python/Shell scripting, and other appropriate tools.

- Support platform deployments, configuration changes, environment promotion, and release activities.

- Administer Databricks workspaces, users, groups, permissions, compute, clusters, policies, and platform configurations.

- Manage Unity Catalog, including catalogs, schemas, external locations, storage credentials, grants, and access governance.

- Configure and manage compute policies and job policies.

- Support AWS-integrated Databricks services and troubleshoot connectivity and security issues.

- Assist Data Engineering and Data Science teams with platform-related issues, pipeline failures, compute problems, access issues, and workload troubleshooting.

- Perform initial investigation of complex platform issues and escalate to the appropriate engineering teams when required.

- Participate in proof-of-concept (POC) initiatives and evaluate emerging Databricks capabilities.

- Create and maintain runbooks, operational procedures, troubleshooting guides, and platform documentation.

- Provide effective shift handoffs and offshore/onshore operational coverage.

- Continuously identify opportunities to improve platform reliability, automation, operational efficiency, and support processes.

Required Qualifications :

- 3+ years of hands-on experience administering and supporting Databricks in a production environment.

- Strong experience supporting enterprise-scale Databricks environments on AWS.

- AWS certification and Databricks certification are required.

- Strong hands-on knowledge of Databricks Account and Workspace Administration, Unity Catalog, Compute and Job Policies, cost optimization, security, REST APIs, and Terraform.

- Strong AWS knowledge across IAM, VPC, KMS, DynamoDB, MSK, S3, PrivateLink, and CloudWatch.

- Experience with Python/Shell scripting, data pipelines, SQL, ETL, and incident management.

Good to Have :

- Experience with Databricks Jobs, Workflows, Delta Lake, SQL Warehouses, and serverless capabilities.

- Experience with CI/CD tools such as Git, GitHub/GitLab, Azure DevOps, or Jenkins.

- Experience with platform observability, monitoring solutions, and Databricks upgrades/migrations.

Skills

Databricks, AWS, IAC Terraform, Python, Shell Scripting, Configuration Management, SQL, ETL