Data Engineer

Responsibilities:

• Designing data platform architecture and design for (a) migrating open-source or other public cloud-based data platforms to to GCP cloud-native technologies (not just lift-and-shift migration but re-engineering to native services) (b) Designing greenfield data platforms on the cloud.
• Define solution architecture and detailed design and perform hands-on implementation.
• Data pipelines for batch and event-driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc.
• Data pipelines for real-time data ingestion and processing.
• Automatic transpilation of legacy code (HIVE, Teradata, python logic etc. ) to BQ SQL.

Requirements:

• In-depth knowledge and experience of GCP data and analytics technologies.
• BQ query performance optimization.
• CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, and Terraform.
• Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog).
• Tools and languages experience required.
• Must have GCP Dataflow, Dataproc, Pub/Sub, Cloud Composer, Cloud Workflow, BigQuery, Cloud Run, and CloudBuild.
• Must have programming knowledge and willingness to be hands-on - Python and Java.

Skills

BigQuery, Cloud Composer, Dataflow, Dataproc, GCP, Google Cloud Platform, Java, Pub/Sub, Python