Senior Data Engineer

We are looking for a hands-on Senior Data Engineer to design and build scalable data solutions and reusable data platform capabilities on Google Cloud Platform (GCP). This role combines data engineering and data platform engineering. You will provide technical leadership for complex data-processing capabilities while remaining actively involved in architecture, design, and implementation. You will help define how data products are transformed, orchestrated, governed, and operated at scale, with a strong focus on reusability, automation, performance, reliability, and engineering standards.

Responsibilities:

• Own the technical design and architecture of scalable data-processing and transformation solutions on GCP.
• Design and build production-grade batch, incremental, and streaming data pipelines.
• Develop and optimize complex transformations using BigQuery, SQL, and Dataform/dbt.
• Build reusable, metadata-driven, and configuration-driven frameworks that reduce bespoke pipeline development.
• Design orchestration patterns using Cloud Composer / Apache Airflow, including dynamic workflows, dependencies, retries, and recovery.
• Develop distributed and streaming processing solutions using Dataflow / Apache Beam and Dataproc / Spark.
• Define standards for data modeling, schema design, schema evolution, and data contracts.
• Design and evolve modern analytical storage patterns, including Apache Iceberg.
• Establish engineering patterns for data quality, reconciliation, metadata, lineage, and observability.
• Optimize large-scale workloads for performance, scalability, latency, and GCP cost.
• Define reusable engineering standards for testing, versioning, CI/CD, and production readiness.
• Lead technical design and code reviews and guide engineers on complex implementation decisions.
• Identify systemic technical debt and drive improvements to shared platform capabilities.
• Work closely with data engineering, platform engineering, DevOps, architecture, governance, and product teams.
• Take technical ownership from solution design through production deployment and operational stability.

Requirements:

• 6+ years of hands-on experience designing and building enterprise-scale data platforms and data products.
• Strong production experience with Google Cloud Platform (GCP).
• Deep hands-on experience with BigQuery, including data modeling, partitioning, clustering, query optimization, and cost optimization.
• Advanced SQL skills for complex transformation and analytical workloads.
• Strong programming experience with Python, including modular development, testing, and automation.
• Strong experience with Dataform and/or dbt.
• Hands-on experience with Cloud Composer / Apache Airflow and complex DAG/workflow design.
• Hands-on experience with Google Cloud Dataflow / Apache Beam.
• Experience with Dataproc and Spark / PySpark for distributed processing.
• Experience with Apache Iceberg or modern lakehouse table formats.
• Strong understanding of batch, incremental, idempotent, and streaming processing patterns.
• Experience designing reusable data-processing frameworks rather than only individual pipelines.
• Experience with metadata-driven and configuration-driven processing.
• Strong data-modelling skills, including dimensional, normalized, and denormalized models.
• Experience defining and implementing data contracts and schema-management patterns.
• Strong understanding of data quality, metadata, lineage, and observability.
• Experience with Git, automated testing, and CI/CD for data workloads.
• Experience designing solutions for reliability, recoverability, and production operations.
• Ability to independently make architectural decisions and communicate technical trade-offs.

Technical Skills:

• Cloud and Storage: Google Cloud Platform, BigQuery, Google Cloud Storage, Apache Iceberg.
• Transformation and Orchestration: Advanced SQL, Dataform/dbt, Cloud Composer/Apache Airflow.
• Distributed and Streaming Processing: Dataflow, Apache Beam, Dataproc, Spark / PySpark.
• Programming and Engineering: Python, Git, CI/CD, automated testing, monitoring, and observability.
• Data Platform Capabilities: Batch and incremental processing, streaming and event processing, data modelling, data contracts and schema management, metadata-driven processing, configuration-driven processing, data quality and reconciliation, metadata and lineage, performance and cost optimisation, Reusable platform frameworks.

Good to Have:

• Practical experience with data product/data mesh principles.
• Experience developing self-service or internal data-platform capabilities.
• Experience with change data capture and event-driven architectures.
• Exposure to semantic and metric modelling.
• Experience with Terraform / Infrastructure as Code.
• Experience working with multi-domain or multi-market enterprise data platforms.

Strong Interpersonal Skills:

• Strong technical ownership and ability to drive complex initiatives from concept to production.
• Ability to lead technical discussions and communicate architectural decisions clearly.
• Strong problem-solving and production troubleshooting skills.
• Ability to mentor engineers and improve engineering practices across a team.
• Comfortable collaborating with engineering, architecture, product, and business stakeholders.
• Ability to work effectively within distributed and multicultural teams.

Skills

GCP, Google Cloud Platform, Python