Senior Data Engineer

Responsibilities:

• Design, build, and maintain scalable batch and streaming data pipelines on GCP.
• Ingest data from multiple sources, including traditional databases and API-based integrations, into our cloud data platform.
• Develop data and feature engineering workflows that support analytics, BI, and machine learning use cases.
• Orchestrate and schedule pipelines with tools such as Cloud Composer/Airflow and Vertex AI Pipelines.
• Contribute to data architecture design, choosing the right storage and compute options for each use case.
• Optimize pipeline and cluster performance for reliability, scalability, and cost.
• Package and deploy data workloads with Docker and CI/CD practices.
• Work with data scientists, analysts, and business teams to understand how data is used and ensure it is accurate, available, and fit for purpose.

Requirements:

• Strong programming skills in Python, SQL, and PySpark.
• Hands-on experience with at least one cloud platform (GCP preferred).
• Experience working with Docker.
• Experience managing environments and dependencies (e. g., venv, pip).
• Experience with orchestrators such as Vertex AI Pipelines or Airflow.
• Solid knowledge of data engineering and feature engineering techniques.
• Proficiency in at least one of Apache Spark, Apache Beam, or Apache Flink.
• Understanding of streaming concepts such as windowing, late-arriving data, and triggers.

Should have:

• Hands-on experience with distributed computing.
• Working experience in data architecture design.
• Knowledge of storage and compute options and when to choose each.
• Good understanding of cluster and pipeline optimization strategies.
• Experience using GCP tools to build end-to-end data pipelines for different scenarios, including ingestion from traditional databases and API-based sources.
• Experience with CI/CD pipelines and deployment methodologies.
• A business mindset: understanding the data and how it will be used for BI and analytics.
• Good to have: Hands-on experience with Kubernetes.

Tools and Technologies (GCP):

• Storage: Cloud SQL, Cloud Storage, Cloud Bigtable, BigQuery.
• Ingestion: Pub/Sub, Cloud Functions, Google Kubernetes Engine (GKE).
• Scheduling and Orchestration: Cloud Composer, Airflow, and Vertex AI Pipelines.
• Processing: Cloud Dataproc, Cloud Dataflow, Apache Spark.
• DevOps: Docker, CI/CD (e. g., GitHub/GitLab Actions).

Skills

Apache Spark, PySpark, Python, SQL, Spark