Data Engineering

We are seeking a Data Engineering Manager to help design, build, and optimise our next-generation data platform on Databricks. This is a hands-on engineering role for someone who can work across modern data architecture, scalable data pipelines, platform engineering, and data governance.

You will play a key role in migrating from legacy data platforms to Databricks, working closely with architecture, product, technology, and business teams to turn complex requirements into robust, secure, and scalable data solutions.

The successful candidate will bring deep technical experience in data engineering, strong Databricks capability, and the ability to influence engineering standards and delivery quality across a modern data platform environment.

Working as part of the Data Transformation Team, you will be responsible for:

• Designing, developing, and optimising scalable data pipelines using Databricks, Apache Spark, Delta Lake, Python, and SQL.
• Building reliable ETL and ELT processes that support analytics, reporting, benchmarking, and data product use cases.
• Contributing to the design and implementation of the Databricks Lakehouse architecture, including data modelling, data quality, performance, and scalability.
• Supporting the migration of data workloads from legacy platforms to Databricks, working with architects and engineering teams to recommend practical technical solutions.
• Implementing data governance and security controls using Databricks Unity Catalog, including access control, lineage, and environment-level governance.
• Improving the performance and cost efficiency of Databricks clusters, SQL warehouses, workloads, and serverless compute.
• Applying DevOps and DataOps practices, including CI/CD, automated deployment, version control, testing, and infrastructure as code.
• Working with tools such as Terraform, GitHub Actions, Azure DevOps, or equivalent engineering toolchains.
• Collaborating with Data Transformation, Architecture, Product, Technology, and business stakeholders to deliver high-quality data solutions.
• Supporting engineering best practice through code reviews, technical documentation, design input, and knowledge sharing.
• Helping to establish repeatable engineering patterns for data ingestion, transformation, orchestration, testing, monitoring, and release management.

We are looking for someone with:

• 7 to 10 years experience in data engineering, including strong hands-on experience designing and building production-grade data pipelines.
• Significant experience working with Databricks, ideally in a platform migration, modernisation, or Lakehouse implementation context.
• Deep expertise in Apache Spark, including PySpark, Spark SQL, and ideally Scala.
• Strong knowledge of Delta Lake and Lakehouse design patterns.
• Advanced proficiency in Python and SQL.
• Experience working with cloud data platforms and native cloud data services, ideally AWS.
• Practical experience implementing data governance patterns, ideally using Databricks Unity Catalog.
• Experience with infrastructure as code, particularly Terraform or similar tools.
• Good understanding of CI/CD, automated testing, deployment pipelines, and modern engineering practices.
• Knowledge of data warehousing concepts, including dimensional modelling, Kimball methodology, star schemas, and snowflake schemas.
• Experience integrating dbt or similar transformation frameworks with modern data platforms.
• Strong problem-solving skills and the ability to translate complex technical challenges into clear, practical solutions.
• Excellent communication skills, including the ability to explain technical topics to both engineering and non-technical stakeholders.
• Experience working in scaled Agile or product-led delivery environments.
• A strong focus on quality, maintainability, documentation, and engineering discipline.

It would be advantageous if you have:

• Databricks Certified Data Engineer Professional certification.
• Databricks Certified Lakehouse Platform Architect certification.
• Familiarity with MLOps or machine learning engineering workflows.
• Experience working within a Product Development Life Cycle.
• Experience supporting distributed teams across time zones.
• Exposure to Jobs-To-Be-Done or Human-Centred Design approaches.
• Industry experience in financial services, benchmarking, market analytics, or data analytics.
• Familiarity with scaled Agile ceremonies such as PI Planning.
• A team-oriented, proactive, analytical working style, with the ability to operate across detailed engineering work and broader platform context.