Data Engineer

Responsibilities:

• AI-Augmented Pipeline Architecture: Design and build self-healing, schema-validated data pipelines on GCP (BigQuery, Dataflow, Pub/Sub) that power real-time analytics. Pipelines must be idempotent, observable, and auto-recover from failures without human intervention.
• Streaming and Batch Processing at Scale: Implement and optimise large-scale processing workflows using Apache Spark, Flink, or Beam. Own both real-time Kafka consumer pipelines and scheduled batch jobs, ensuring sub-hour SLAs for data freshness across all critical business domains.
• Schema Contracts and Data Integrity: Implement and enforce schema registry and data contract standards across all producers and consumers: target 100% data quality scores, no silent failures, and no downstream surprises.
• ML Pipeline Integration: Build and maintain feature stores and data pipelines that feed production ML models (recommender systems, demand forecasting, pricing). Collaborate with data science to ensure training-serving skew is eliminated.
• Data Governance and Cost Engineering: Champion data cataloguing, lineage tracking, and access-control policies. Monitor BigQuery slot usage and pipeline costs, and implement optimisations that reduce cloud spend by measurable percentages each quarter.

Requirements:

• DE2 3-6 years; end-to-end ownership of production pipelines at scale.
• Ability to ship data products independently with clear quality and latency guarantees.

Tech Stack:

• Languages: Python, Scala, Java, SQL.
• Processing: Apache Spark, Apache Beam / Dataflow, Apache Flink, Kafka, Databricks.
• Cloud and Storage: GCP (BigQuery, Dataflow, Pub/Sub, DataProc, and GCS) and AWS (S3 and EMR).
• Orchestration: Apache Airflow (Cloud Composer).
• Data Quality: Great Expectations, dbt.
• Infrastructure: Kubernetes, Docker, Terraform, Databricks.
• AI Productivity: Cursor, Claude.

Skills

airflow, beam, bigquery, databricks, dataflow, flink, gcp, java, kafka, pub/sub, python, scala, spark, sql