DevOps Lead (GCP, AWS) – Data Engineering

We are looking for a highly skilled and motivated DevOps Lead to design, build, and operate scalable, secure, and highly available cloud and data platforms, primarily on Google Cloud Platform (GCP) with Amazon Web Services (AWS) as a secondary environment.

This is a hybrid platform role. Alongside core DevOps ownership - Terraform, Kubernetes, CI/CD, cloud security, and observability - you will also contribute substantially to our data engineering function: building and operating production Python data pipelines, orchestrating batch and streaming workflows, and owning the infrastructure that sits underneath our data platform.

The ideal candidate has spent several years building data pipelines as a data engineer and has since moved into a DevOps / SRE / Platform Engineering role, or has run platform engineering for data-intensive products. We are looking for someone who can credibly wear both hats: infrastructure and automation first, data engineering a close second.

DevOps & Cloud Infrastructure Responsibilities:

• Design, implement, and manage cloud infrastructure across GCP and AWS using Terraform and other Infrastructure-as-Code practices.
• Deploy and maintain Kubernetes clusters and manage application workloads using Helm and/or ArgoCD.
• Build and enhance secure, scalable CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI, and ArgoCD.
• Implement best practices in containerization using Docker, including image hardening and container runtime security.
• Integrate quality gates in CI/CD, including automated testing, code coverage, and SonarQube configuration.
• Ensure secure CI/CD processes, including secrets management, least-privilege access, and hardened build agents.
• Implement and maintain cloud security best practices: IAM roles and policies, network segmentation, encryption, and vulnerability management.
• Collaborate with security teams to embed DevSecOps practices (shift-left security, scanning in pipelines, policy-as-code).
• Contribute to system architecture, monitoring, and observability using modern tools and practices.
• Participate in incident response, post-incident reviews, and root cause analysis.

Data Engineering & Data Platform Responsibilities:

• Design, build, and maintain production-grade ETL/ELT pipelines in Python for both batch and streaming workloads.
• Build and operate workflow orchestration using Airflow / Cloud Composer, including scheduling, dependency management, retries, and alerting.
• Develop and optimise data models and transformations in BigQuery (and equivalents such as Snowflake or Redshift), with attention to partitioning, clustering, and query cost.
• Own the infrastructure and deployment lifecycle of data services - provisioning, autoscaling, upgrades, and cost management.
• Implement data quality, validation, lineage, and observability checks so pipeline failures and data drift are caught early.
• Apply software engineering discipline to data workloads: version control, code review, automated testing, and CI/CD for pipelines and transformations.
• Work with analytics, ML, and product teams to make data reliably available, secure, and well governed.

Must have skills:

-8+ years of total professional experience in software, data, or infrastructure engineering.

-Minimum 4 years in a dedicated DevOps / SRE / Platform Engineering role.

-Substantial prior hands-on data engineering experience at a senior level (pipeline development, data warehousing, and orchestration).

-Strong proficiency in Google Cloud Platform (GCP) and solid working experience with AWS.

-Hands-on experience with Terraform and Infrastructure as Code (IaC).

-Strong experience with Kubernetes, Docker, and Helm in production.

-Expertise with CI/CD tools: Jenkins, GitLab CI, GitHub Actions, ArgoCD.

-Strong production-grade Python - not scripting only, but maintainable pipeline and tooling code - plus solid Bash.

-Advanced SQL and hands-on experience with a cloud data warehouse (BigQuery preferred; Snowflake or Redshift also considered).

-Experience with workflow orchestration (Airflow / Cloud Composer) and a distributed processing framework (Dataflow/Apache Beam, Spark, or dbt).

-Experience with streaming or event-driven data (Pub/Sub, Kafka, or Kinesis).

-Experience with SonarQube and static code analysis integration.

-Practical cloud security experience (IAM, VPC, firewalls/security groups).

-Secrets management experience (Secret Manager / Secrets Manager / Vault).

-Experience with image and dependency scanning.

-Knowledge of secure software development practices.

-Excellent written and verbal communication skills.

Good to Have:

-Certifications: GCP Professional DevOps Engineer, GCP Professional Data Engineer, GCP Architect, AWS DevOps Engineer, etc.

-Experience with dbt and modern analytics engineering practices.

-Data modelling experience (dimensional modelling, lakehouse formats such as Iceberg or Delta Lake).

-Data quality and testing frameworks (Great Expectations, Soda, dbt tests).

-Exposure to ML platform support (Vertex AI, MLflow, feature stores, model deployment pipelines).

-Knowledge of compliance frameworks (ISO 27001, CIS Benchmarks, NIST) and data governance / PII handling.

-Experience with monitoring tools (Prometheus, Grafana).

-Experience with logging stacks (ELK, Loki, etc.).

-Exposure to policy-as-code and security tools (OPA, scanners).

-Cloud cost optimisation / FinOps experience, particularly for data workloads.

-Scripting or service development in Go.