Data Engineer GCP

Project description

The project focuses on building and enhancing a data platform on Google Cloud Platform. The team is responsible for integrating data from multiple GCP-based data assets, transforming it into usable formats, and delivering high-quality data pipelines for downstream analytics and business use cases. The environment is cloud-native and leverages modern data engineering practices, including CI/CD, orchestration, and scalable data processing frameworks.

Scope of work / tasks

Design and develop scalable data pipelines on Google Cloud Platform using Python and SQL

Integrate data from various GCP data sources such as BigQuery and Cloud Storage (buckets)

Implement data transformations and ensure efficient data processing workflows

Collaborate with Data Architect, Data Delivery Lead, and Business Analysts to translate business requirements into technical solutions

Propose and implement robust data models and pipeline architectures

Orchestrate workflows using Apache Airflow / Cloud Composer

Ensure pipelines meet high quality standards including auditability, rerun capability, modularity, reusability, and unit testing

Contribute to CI/CD processes for deployment and pipeline automation

Support development of data lakes, data warehouses, and dimensional models (star schema / Snowflake)

Requirements (must have)

Strong experience with Google Cloud Platform

Very good knowledge of Python and SQL

Hands-on experience with BigQuery and Cloud Storage

Experience with Apache Airflow or Cloud Composer

Experience in building and maintaining data pipelines in production environments

Strong understanding of data modeling (data lake, DWH, star schema, Snowflake)

Experience with CI/CD and pipeline orchestration

Ability to design scalable and maintainable data solutions

Experience with data quality, testing, and observability practices

Nice to have

Experience with PySpark

Knowledge of Spark and/or Hadoop ecosystems

Experience with streaming or big data processing solutions

Background in backend services development (Python)