Data Engineering Lead

Responsibilities Build scalable ETL/ELT pipelines to handle high-velocity streaming data and batch processing. Implement data integrity checks, metadata management, and security protocols. Provide the data backbone for the MDO, Commercialisation, and engineering teams to ensure seamless data flow. Maintain the highest standards for code quality, including conducting rigorous code reviews, developing comprehensive unit tests, and producing clear, thorough documentation Design and implement the data warehouse: raw data lake + structured analytical warehouse with clear separation of raw, cleaned, and modelled data layers Implement data lineage tracking: every number that enters a model output must be traceable back through transformations to its original source record, Design the data access layer for the appropriate teams and create experimental, isolated, and research sandboxes

Requirements Required Qualifications 8+ years in Data Engineering with experience leading engineering teams. Expertise in Apache Spark/Kafka, Airflow, and cloud infrastructure (AWS/Azure/GCP). Advanced knowledge of NoSQL, Relational, and Vector databases. Proven understanding of fundamental data structures and the ability to apply them to solve complex problems. Cloud data warehouse experience: Schema design, query optimisation, cost management Comfort with large, messy, multi-source datasets Experience with data quality frameworks Experience with Containerization (Docker/Kubernetes)

Preferred

Qualifications Experience at a data-intensive platform company where data pipelines directly powered business-critical decisions Exposure to time-series data pipelines: has built systems that ingest, store, and serve temporal data cleanly (traffic growth time-series, economic indicators) Understanding of mathematics, particularly linear algebra and probability theory