Senior Databricks Engineer

We are seeking an experienced Senior Databricks Engineer to design, implement, automate, and support enterprise-scale Databricks platforms across AWS, Azure, and GCP. The role will focus on Databricks workspace management, Lakehouse architecture, cloud migrations, platform automation, governance, security, cost optimisation, and operational excellence. The candidate will work closely with Data Engineering, Decision Science, Product Engineering, Cloud Infrastructure, and Security teams to deliver scalable and secure data platforms.

Responsibilities:

• Databricks Platform Engineering: Own the design, deployment, administration, and lifecycle management of enterprise Databricks workspaces across AWS, Azure, and GCP.
• Cloud and Infrastructure Integration: Enable secure connectivity, authentication, networking, and integration between Databricks and enterprise cloud platforms and databases.
• Migration and Modernisation: Lead migration of legacy data platforms, analytics workloads, and databases to the Databricks Lakehouse architecture.
• Automation and DevOps: Drive Infrastructure-as-Code, CI/CD, and operational automation to improve platform scalability, consistency, and efficiency.
• Security and Governance: Implement enterprise-grade security, access controls, compliance, auditing, and data governance using Unity Catalogue and identity integrations.
• Performance and Cost Optimisation: Continuously optimise Databricks workloads, storage layouts, and compute resources to improve performance and reduce cloud spend.
• Monitoring and Operational Support: Establish proactive monitoring, incident management, troubleshooting, and platform reliability practices for business-critical workloads.
• AI and Innovation: Evaluate and implement Databricks AI, GenAI, and automation capabilities to enhance platform operations and business value.
• Stakeholder: Partner with architects, engineering teams, security teams, and business partners while mentoring engineers and driving platform standards.

Requirements:

• 3+ years of overall experience in Databricks.
• Solid experience with Databricks administration and platform engineering.
• Expertise in Spark, PySpark, SQL, Delta Lake, Unity Catalogue, and Workflows.
• Experience with AWS, Azure, and/or GCP.
• Strong Python scripting and automation experience.
• Experience with Terraform and Infrastructure as Code.
• Hands-on experience with CI/CD pipelines and Git-based deployments.
• Good understanding of networking, private link connectivity, SSO, SAML/OIDC, and security concepts.
• Experience with PostgreSQL, SQL Server, Oracle, or cloud-native databases.
• Knowledge of data governance, metadata management, and compliance controls.

Preferred Skills:

• Databricks Certified Professional Data Engineer and Cloud certifications (AWS/Azure/GCP)
• Experience with Lakehouse migrations.
• Knowledge of Kafka, ELK, Prometheus, Grafana, and cloud observability tools.
• Experience with MLflow, Databricks Genie, and GenAI solutions.
• Experience supporting large-scale enterprise analytics platforms.
• Exposure to multi-cloud environments and global data platforms.
• B. S. /B. E. in a related field or equivalent work experience required.
• Databrick/AWS certification is a plus.

Skills

apache spark, databricks, infrastructure as code, pyspark, spark, terraform