Senior Data Platforms Engineer

Databricks Platform Engineering Manager About the Role

We are looking for a hands-on Databricks Platform Engineering Manager to help establish and mature our data platform engineering capability.

This is the first dedicated platform engineering role within the Data Transformation function, so the successful candidate will play a foundational role in shaping how we build, operate, govern and continuously improve our modern data platform. The role combines hands-on engineering, operational ownership and technical leadership across Databricks, cloud infrastructure, platform automation, observability, security and developer enablement.

You will be responsible for ensuring that our Databricks and Lakehouse environments are secure, reliable, scalable, cost-effective and easy for data teams to use. You will work closely with Data Engineers, Analytics Engineers, Data Scientists, Product teams and business stakeholders to provide a governed, high-performing platform that enables faster delivery of data products, analytics and AI-enabled capabilities.

This is not a traditional support or administration role. It is a platform engineering role focused on building reusable capabilities, automation, standards and self-service patterns that allow teams to deliver safely and efficiently at scale.

What the Job Involves Platform Engineering and Architecture

• Design, build and maintain scalable, secure and highly available Databricks and Lakehouse platform environments.
• Establish the core engineering patterns, standards and guardrails for Databricks platform usage.
• Administer and optimise Databricks workspaces, clusters, SQL Warehouses, Unity Catalog, Delta Lake and related platform services.
• Develop reusable platform templates, reference architectures and self-service capabilities to accelerate adoption across data teams.
• Shape environment strategies across development, test, production and controlled experimentation.
• Evaluate emerging Databricks, cloud and data platform capabilities, and develop proof-of-concepts where they can improve platform performance, governance or usability.

Operations, Reliability and Observability

• Own the operational health of the Databricks platform, including reliability, performance, observability and incident response.
• Implement monitoring, logging, alerting and audit capabilities to provide clear visibility of platform performance, usage and issues.
• Create and maintain operational runbooks, standards and support processes.
• Troubleshoot platform issues and drive root-cause resolution.
• Continuously improve platform stability, resilience, scalability and cost efficiency.
• Define and track platform service levels, operational metrics and continuous improvement actions.

Automation, DevOps and Infrastructure

• Build and manage infrastructure-as-code for Databricks and related cloud platform components, using Terraform or equivalent technologies.
• Automate infrastructure provisioning, workspace configuration, access management and platform deployment activities.
• Develop and maintain CI/CD pipelines for platform services, infrastructure changes and reusable platform components.
• Embed quality, security and governance controls into automated deployment processes.
• Reduce manual operational effort by designing automated, repeatable and auditable platform processes.

Security, Governance and Compliance

• Implement secure and governed access patterns across Databricks, Unity Catalog, cloud storage and related services.
• Partner with architecture, security and data governance stakeholders to ensure the platform aligns with organisational and regulatory requirements.
• Define and enforce platform policies for identity, access management, data permissions, environment separation and auditability.
• Support the development of governance models for deploying data and analytics products across controlled environments.
• Ensure platform design supports appropriate data protection, lineage, discoverability and compliance requirements.

Enablement and Stakeholder Partnership

• Work closely with Data Engineers, Analytics Engineers, Data Scientists and Product teams to understand platform needs and remove delivery friction.
• Provide technical guidance, documentation and reusable patterns that help teams use the platform safely and effectively.
• Act as a trusted technical lead for Databricks platform best practice.
• Translate complex platform concepts into clear recommendations for both technical and non-technical stakeholders.
• Help shape the future operating model for platform engineering as the capability grows.

What We Are Looking For Essential Experience

• 7 to 10 years experience in data engineering, including strong hands-on experience designing and building production-grade data pipelines.
• Strong experience in Platform Engineering, Cloud Engineering, Data Platform Engineering or a closely related discipline.
• Hands-on experience administering, configuring and optimising Databricks environments.
• Good understanding of modern Lakehouse architecture, including Delta Lake, Databricks SQL and governed data access patterns.
• Experience with Unity Catalog or equivalent data governance and access control capabilities.
• Experience working with at least one major cloud platform, such as AWS, Azure or GCP.
• Strong infrastructure-as-code experience, ideally using Terraform or equivalent technologies.
• Experience designing and implementing CI/CD pipelines for infrastructure, platform services or data platform components.
• Strong understanding of platform security, identity management, networking, access controls and operational governance.
• Experience implementing monitoring, logging, alerting and observability for cloud or data platforms.
• Good scripting or automation skills, such as Python, PySpark, shell scripting or similar.
• Strong troubleshooting, problem-solving and analytical skills.
• Ability to work effectively with cross-functional engineering, data, product and business teams.
• Clear written and verbal communication skills, including the ability to explain technical concepts in concise business language.

Desirable Experience

• Relevant Databricks certification.
• Relevant cloud certification.
• Experience supporting distributed teams across multiple time zones.
• Experience working within a Product Development Life Cycle.
• Exposure to scaled agile delivery practices, such as PI Planning.
• Experience in financial services, benchmarking, analytics or data products.
• Experience supporting data science, machine learning or AI platform use cases.
• Experience defining platform operating models, service catalogues or platform product roadmaps.