Senior Data Engineer

At Canopius, our delivery teams are responsible for ensuring that business users can effectively harness data insights to drive strategic decision making. Our data strategy is centred around an enterprise Lakehouse platform on Databricks, avoiding fragmented, ungoverned silos on legacy technologies that hamper creativity and scalability. We are building a governed, interoperable data estate that enables self-service for our business teams and provides the trusted foundation for advanced analytics, machine learning and AI to accelerate change across our industry.

This role is an opportunity to apply your expertise in Azure technologies, data storage and data engineering techniques to build, extend and maintain our data ecosystem, which is central to all reporting and analytics across Canopius. You will contribute to key business transformation projects, working closely with senior business stakeholders and leveraging modern technologies. The role includes line management responsibility for 2 junior and a mid-level data engineer, with the potential to grow over time as required. As a senior member of the team, you will also act as a role model and mentor to other engineers, providing guidance and support.

The ideal candidate is an experienced data engineer looking for a new challenge who is enthusiastic about using technology to accelerate change within our industry. You should have the ability to understand business problems and deliver efficient and reliable solutions tailored to our unique challenges, while keeping to a high degree of technical excellence and good data engineering practices. You should be comfortable collaborating and working as part of a dynamic multi-disciplinary team.

This role is central to delivering Canopius’ data strategy through development of new data capabilities and supporting the final transition from legacy SQL Server-based environments. Alongside hands-on delivery, the role provides technical leadership across data engineering initiatives, helping to establish a future-ready, cloud-native data ecosystem.

This is a hybrid role, working 2-3 days per week from our Manchester office.

Responsibilities will include:

• Provide technical leadership and direction across data engineering initiatives.
• Design solutions for production operation, including documentation, monitoring, alerting, logging, access and defined support responsibilities. Ensure solutions are secure, scalable, maintainable, and aligned with organisational standards and governance requirements.
• Design, build, and optimise data pipelines on Databricks using PySpark and Spark SQL, applying Delta Lake and a medallion/Lakehouse architecture.
• Implement data validation and reconciliation checks, and automated testing across pipelines to ensure trusted, accurate outputs.
• Support the migration and re-platforming of legacy SQL Server data warehouse processes as required.
• Challenge requirements constructively and provide guidance on what is technically feasible, scalable, and commercially viable.
• Communicate technical solutions, trade-offs, risks, and opportunities to both technical and non-technical audiences, managing expectations and influencing decisions with evidence-based recommendations.
• Lead technical planning activities, ensuring work is appropriately defined, estimated, and prioritised. Provide delivery estimates and communicate progress and blockers to stakeholders.
• Collaborate with Product Owners, Business Analysts, and Solution Architects to define product vision and technical roadmaps.
• Drive adoption of modern engineering practices, AI technologies, and automation opportunities across the team.
• Drive delivery of complex, end-to-end work items, seeking clarification of ambiguous specifications and proactively flagging risk of technical debt.
• Management of own task list and ensuring that plans are agreed.
• Line management of junior and mid-level data engineers in the team.
• Mentor and support engineers within the squad through pairing, peer review of pull requests, constructive feedback and knowledge sharing, ensuring successful delivery of sprint objectives and acting as a role model to the team.
• Undertake other ad-hoc duties as required.

Skills and Experience

• Strong understanding of data engineering principles, data architecture, and cloud-based data platforms.
• Proficient in Python (PySpark) and SQL for building and optimising data transformations at scale.
• Strong hands-on Databricks experience.
• Experience with Azure data storage technologies (Azure SQL, Data Lakes, Blob Containers).
• Experience with data transformation and modelling techniques.
• Thorough understanding and experience with data warehousing concepts and dimensional modelling.
• Experience with ETL and orchestration tooling (Data Factory, Databricks).
• Experience with data governance and cataloguing using Databricks Unity Catalog (access control, lineage, discoverability).
• Experience with Azure DevOps, including branching and release/deployment strategies; Git, CI/CD and DevOps Pipelines.
• Hands on experience developing and supporting commercial enterprise data solutions in a cloud hosted environment.
• Demonstrable experience of supporting internal Analytics, Data Science, BI and reporting functions.
• Experience working in an Agile and Scrum environment, with proven ability to work effectively in cross-functional development teams.
• Thorough understanding and experience with all aspects of SQL Server including the building of stored procedures to support complex data transformations.
• Experience leveraging AI-assisted development tools and copilots to improve engineering productivity and delivery outcomes is advantageous.
• Strong problem-solving skills with a continuous improvement mindset and willingness to challenge existing approaches and adopt new technologies where appropriate.
• Continuous improvement mindset - sets and maintains development standards, drives efficiencies in the E2E development process.
• Good Communication skills, with the ability to explain complex topics in an easy-to-understand manner for both technical and non-technical audiences. Able to translate ambiguous business needs into solutions, defining the appropriate data to use, communicating time/effort/risk/value of different approaches.
• Familiarity with specialty (re)insurance or Lloyd's market data such as bordereaux, delegated authority, underwriting and claims is advantageous.