GCP BigQuery Developer

The candidate will have responsibilities across the following functions:

Data Warehousing and BigQuery Development:

• Design and implement data warehouse solutions using Google BigQuery.
• Develop, optimise, and maintain complex SQL queries and data models.
• Create efficient partitioning, clustering, and indexing strategies.

Data Pipeline Development:

• Build scalable data pipelines using GCP services (Dataflow, Dataproc, Cloud Composer).
• Perform data ingestion from various sources (APIs, databases, cloud storage).
• Ensure data transformation, cleansing, and validation.

Performance Optimisation:

• Optimise BigQuery queries for cost and performance.
• Monitor query execution and tune workloads.
• Implement best practices for storage and compute efficiency.

Integration and ETL/ELT:

• Implement ETL/ELT workflows using BigQuery, Cloud Storage, and Pub/Sub.
• Integrate data from multiple cloud and on-prem systems.
• Automate workflows and scheduling.

Collaboration and Reporting:

• Work closely with data analysts, data scientists, and business teams.
• Support reporting tools like Looker, Data Studio (Looker Studio), and Tableau.
• Translate business requirements into scalable data solutions.

The core requirements for the job include the following:

Core Technical Skills:

• 3-5 years of experience in data engineering / BigQuery development.
• Strong expertise in SQL (mandatory).
• Hands-on experience with Google BigQuery.
• Experience with data modelling (star/snowflake schemas).

GCP Tools and Services:

• BigQuery
• Cloud Storage
• Dataflow / Dataproc
• Pub/Sub
• Cloud Composer (Airflow)

Programming Skills:

• Proficiency in Python / SQL scripting.
• Experience with Spark (PySpark) is a plus.