Data Engineer

ZAGENO is hiring a Data Engineer to own the operational data pipelines powering our life sciences catalogue. This is a high-ownership role: you'll be the primary engineer responsible for the reliability, correctness, and scalability of the pipelines our operations teams depend on daily. The work spans production pipeline reliability, reducing accumulated technical debt in existing systems, building observability and testing infrastructure, and partnering with Data Science and Analytics on clean data delivery. You'll work directly with CatalogOps and business stakeholders - turning evolving, often underspecified business rules into architectures that stay flexible without compromising data quality. You'll make architectural decisions, push back on requests that introduce heuristic debt, and own incident response end-to-end.

Responsibilities:

• Own reliability and performance of operational pipelines across our product catalogue infrastructure.
• Identify and reduce technical debt, replacing reactive patches with designed, testable logic.
• Build and maintain low-latency data APIs that serve downstream operational and analytics consumers.
• Implement CDC patterns to keep catalogue data synchronised across systems with minimal lag.
• Build monitoring, observability, and automated testing so failures surface before stakeholders report them.
• Design and implement unit standardisation and master data logic at catalogue scale.
• Translate business requirements from non-technical stakeholders into durable pipeline logic.
• Own code versioning, deployment, and incident response for your layer.
• Leverage your expertise in the tech stack: BigQuery, Databricks, Spark/PySpark, AWS/GCP.

Requirements:

• 3+ years as a Data Engineer, including solo or primary ownership of production pipelines.
• Strong Python - data engineering, transformation logic, testing discipline.
• Strong SQL with the ability to write correct queries, identify and refactor anti-patterns.
• Databricks, Delta Lake, Airflow for production orchestration.
• Experience with CDC patterns for real-time or near-real-time data synchronisation.
• Experience building low-latency APIs serving operational or analytical consumers.
• Test-driven development discipline - unit tests, integration tests, regression coverage as standard practice, not an afterthought.
• Operates independently under ambiguity; designs systems to be maintained, not just to run.
• Preferred: Kafka or equivalent event streaming platform experience.
• Experience with entity matching, deduplication, or master data management.
• Exposure to ML pipeline support in production.
• Familiarity with NLP techniques for entity resolution or text normalisation (tokenisation, similarity matching, named entity recognition).

What success looks like:

• Engineers ship data products: Outputs are documented, versioned, and designed for reuse across Analytics, Data Science, and operational consumers.
• Reliability: Pipelines run reliably with minimal manual intervention.
• Performance: Data latency and downtime decrease measurably over time.
• Data Quality: Analytics and Data Science teams receive clean, trustworthy data without ad hoc fixes.
• Scalability: Infrastructure scales with volume growth without proportional cost increase.
• Reduction of Debt: Technical debt in existing pipelines decreases measurably as heuristic patches are replaced with designed logic.
• Synchronisation: CDC-driven synchronisation eliminates the class of cross-pipeline identity drift bugs.