Expert Data Engineer GCP
Remote: 4 days/month from the office in Cracow
Project description
The project focuses on building and enhancing a data platform on Google Cloud Platform. The team is responsible for integrating data from multiple GCP-based data assets, transforming it into usable formats, and delivering high-quality data pipelines for downstream analytics and business use cases. The environment is cloud-native and leverages modern data engineering practices, including CI/CD, orchestration, and scalable data processing frameworks.
Scope of work / tasks
Design and develop scalable data pipelines on Google Cloud Platform using Python and SQL
Integrate data from various GCP data sources such as BigQuery and Cloud Storage (buckets)
Implement data transformations and ensure efficient data processing workflows
Collaborate with Data Architect, Data Delivery Lead, and Business Analysts to translate business requirements into technical solutions
Propose and implement robust data models and pipeline architectures
Orchestrate workflows using Apache Airflow / Cloud Composer
Ensure pipelines meet high quality standards including auditability, rerun capability, modularity, reusability, and unit testing
Contribute to CI/CD processes for deployment and pipeline automation
Support development of data lakes, data warehouses, and dimensional models (star schema / Snowflake)
Requirements (must have)
Strong experience with Google Cloud Platform
Very good knowledge of Python and SQL
Hands-on experience with BigQuery and Cloud Storage
Experience with Apache Airflow or Cloud Composer
Experience in building and maintaining data pipelines in production environments
Strong understanding of data modeling (data lake, DWH, star schema, Snowflake)
Experience with CI/CD and pipeline orchestration
Ability to design scalable and maintainable data solutions
Experience with data quality, testing, and observability practices
Nice to have
Experience with PySpark
Knowledge of Spark and/or Hadoop ecosystems
Experience with streaming or big data processing solutions
Background in backend services development (Python)