Data and AI Architect

We are looking for a Lead Data Architect to design, build, and scale our data pipelines and entity resolution systems. This role combines deep technical expertise in data engineering with hands-on experience in AI-assisted tooling, entity matching, and data integration from diverse sources. You will lead architectural decisions for our data platform, mentor engineers, and ensure our pipelines are reliable, scalable, and production-grade.

Responsibilities:

• Architect, build, and maintain robust, scalable data pipelines that ingest, transform, and serve data from multiple internal and external sources.
• Own the end-to-end orchestration of data workflows using tools like Dagster, ensuring observability, reliability, and maintainability of pipelines.
• Design and implement entity resolution workflows including matching, merging, and survivorship logic using tools such as Splink to produce clean, deduplicated, golden records.
• Build and maintain web scrapers to source data from external providers, ensuring resilience to source changes, rate limits, and data quality issues.
• Integrate and reconcile data coming from multiple, often inconsistent, sources into unified, trustworthy datasets.
• Design and maintain data models and schemas across transactional and analytical systems, ensuring consistency, scalability, and performance.
• Leverage AI/LLM-based tools and techniques to enhance data pipeline capabilities, e. g., intelligent data extraction, automated data quality checks, or AI-assisted entity matching.
• Define and enforce best practices around pipeline design, testing, monitoring, and documentation.
• Collaborate closely with data engineers, product managers, and other stakeholders to translate business requirements into scalable data architecture.
• Provide technical leadership and mentorship to the data engineering team.

Requirements:

• Strong hands-on experience building and maintaining production-grade data pipelines at scale.
• Practical experience with Dagster (or similar orchestration tools like Airflow/Prefect) for pipeline orchestration.
• Experience with Splink or similar probabilistic/deterministic record linkage tools for entity matching, merging, and survivorship.
• Strong proficiency in Python, including experience writing and maintaining web scrapers.
• Proven experience integrating and maintaining data pipelines that pull from multiple, heterogeneous data sources.
• Experience applying AI/ML tools within data engineering workflows (e. g., LLM-assisted data cleaning, extraction, or matching).
• Hands-on experience with relational and distributed databases such as PostgreSQL and Google Cloud Spanner.
• Strong understanding of data modelling principles (normalisation, dimensional modelling, schema design) across OLTP and OLAP systems.
• Experience with cloud data warehousing platforms such as BigQuery, Redshift, and cloud platforms (GCP/AWS/Azure).
• Strong communication skills and experience working cross-functionally with engineering and product teams.
• Experience with distributed data processing frameworks (e. g., Spark, Dask).
• Familiarity with data governance, lineage, and cataloguing tools.
• Prior experience in a lead or architect-level role guiding a data engineering team.

Skills

Airflow, Dagster, ETL, Prefect, Spark, Splink, data, orchestration, pipeline, pipelines, python