Senior Data Engineering

Senior Data Engineering Manager About the Role

We are seeking a Senior Data Engineering Manager to provide technical leadership for the design, implementation, and evolution of our next-generation data platform on Databricks.

As the Lead Data Engineer, you will be accountable for establishing engineering best practices, shaping architectural decisions, and driving the successful delivery of enterprise-scale data solutions. You will act as the technical authority for data engineering, partnering closely with Architects, Product teams, Technology teams, and business stakeholders to ensure the platform remains scalable, secure, cost-effective, and aligned to strategic business objectives.

You will combine hands-on engineering expertise with technical leadership, guiding a team of data engineers through platform migration, solution design, and delivery excellence while helping transform the organisation's data capabilities.

What the Job Involves

Initially reporting to the Director of Data Transformation, you will be responsible for:

Technical Leadership & Platform Ownership

• Own the technical direction of the Databricks data platform and establish engineering standards across all data engineering initiatives.
• Lead architecture and design decisions for enterprise-scale data products and platform capabilities.
• Define reference architectures, reusable design patterns, and engineering guardrails for data ingestion, transformation, orchestration, governance, observability, and deployment.
• Act as the senior technical escalation point for complex engineering challenges.

Team Leadership & Engineering Excellence

• Lead, coach, and mentor a team of data engineers, fostering technical growth and engineering excellence.
• Drive engineering best practices through code reviews, technical design reviews, pair programming, and knowledge sharing.
• Promote a culture of quality, ownership, continuous improvement, and delivery excellence.
• Support recruitment, onboarding, capability development, and succession planning within the engineering team.

Databricks Platform Strategy

• Serve as the organisation's subject matter expert for Databricks and Lakehouse architecture.
• Drive adoption of Databricks capabilities including Delta Lake, Unity Catalog, Workflows, Lakeflow, SQL Warehouses, Serverless Compute, and emerging platform capabilities.
• Define the roadmap for platform optimisation, scalability, reliability, and cost efficiency.
• Partner with enterprise architecture to shape future-state data platform strategy.

Data Governance & Security

• Lead implementation of enterprise data governance standards using Unity Catalog and associated governance capabilities.
• Ensure secure, compliant, and auditable data access patterns across all environments.
• Define policies and standards for metadata management, lineage, data quality, and regulatory compliance.

Delivery Leadership

• Lead technical delivery of strategic data transformation initiatives and migration programmes.
• Work closely with Product Managers, Architects, and business stakeholders to translate business outcomes into technical solutions.
• Manage technical dependencies, risks, and architectural decisions across multiple workstreams.
• Ensure solutions are delivered to agreed standards for performance, reliability, scalability, and maintainability.

Engineering Operations & Automation

• Establish and maintain DevOps and DataOps standards across the platform.
• Lead CI/CD adoption, Infrastructure-as-Code implementation, automated testing, release governance, and operational monitoring.
• Drive platform observability, performance optimisation, reliability engineering, and operational excellence.

Requirements Essential Experience

• 8-12 years experience in Data Engineering, including significant experience leading engineering teams and driving large-scale data platform initiatives.
• Demonstrated experience as a Lead Data Engineer, Technical Lead, Principal Engineer, or equivalent senior engineering position.
• Proven experience architecting and delivering modern data platforms using Databricks.

Technical Skills

• Deep expertise in Apache Spark (PySpark, Spark SQL, Scala).
• Expert-level knowledge of Delta Lake and Lakehouse architecture patterns.
• Advanced proficiency in Python and SQL.
• Strong experience with cloud-native data platforms, ideally AWS.
• Proven expertise implementing Unity Catalog and enterprise governance frameworks.
• Experience with Infrastructure as Code using Terraform.
• Strong understanding of data warehousing, dimensional modelling, and data architecture principles.
• Experience implementing modern DataOps, DevOps, CI/CD, and automated testing practices.

Leadership Skills

• Ability to lead and influence engineering teams without compromising hands-on technical credibility.
• Experience shaping engineering operating models, standards, and governance processes.
• Strong stakeholder management skills across business, technology, architecture, and executive audiences.
• Proven ability to influence senior leadership and drive strategic technology decisions.

Nice to Have

• Databricks Certified Data Engineer Professional.
• Databricks Certified Lakehouse Platform Architect.
• Experience with MLOps and AI Engineering platforms.
• Experience leading enterprise migration programmes from legacy data platforms.
• Financial Services, Data Analytics, Benchmarking, or Information Services industry experience.
• Experience working in scaled Agile environments and PI Planning.