Data Engineer

A Data Engineer for Enterprise Data and Analytics role focuses on unifying complex, siloed data from internal ERPs (like SAP) and Product Life Cycle Manager systems like Laboratory Information Systems with external retail signals (POS data, shipment tracking) to optimise every stage from production to the shelf.

Responsibilities:

• CPG Data Harmonisation: Designing pipelines to ingest, harmonise, and curate disparate data sources, including ERP (eg, SAP), PLM (eg, Laboratory Information Management System), Point of Sale (POS) data from global retailers (e. g., Walmart, Costco, Carrefour, Tesco, etc. ), and internal WMS/TMS systems into a unified Delta Lake.
• Lakehouse Implementation: Developing a Medallion Architecture (Bronze, Silver, Gold) in Databricks to ensure high-quality, trusted aggregated data model and curated data products.
• Supply Chain Orchestration: Using Databricks Workflows or Azure Data Factory to automate complex ETL/ELT processes that power demand forecasting and inventory replenishment models.
• Operational Monitoring: Implementing real-time streaming (Spark Structured Streaming) to provide immediate visibility into supply chain disruptions, such as logistics bottlenecks or stockouts.
• Governance and Security: Managing fine-grained access control and data lineage for multi-brand or regional datasets using Unity Catalogue to ensure compliance with global data privacy standards.

Requirements:

• Education: Bachelor's degree in engineering, computer science, or related field.
• Programming: 5 - 8 years of total work experience, with at least 3 years of experience in advanced SQL, PySpark or Python.
• Core Platform: Good grasp of Databricks (Delta Lake, DLT, Workflows) and Azure/Cloud services (ADLS Gen2 Key Vault).
• Data Lineage: Experience with leveraging Unity Catalogue for ensuring lineage from raw to curated data assets.
• DevOps/DataOps: Proficiency in Git, CI/CD (Azure DevOps/GitHub Actions), and Infrastructure as Code (Terraform) for stable, repeatable deployments.
• Agile Development: Experience working in a collaborative agile environment.
• Cloud Architecture: Understanding of cloud architecture principles and best practices.
• Experience in the design and build of end-to-end solutions that meet business requirements and adhere to scalability, reliability, and security standards.

Desired Certifications:

• Microsoft Certified: Azure Fundamentals.
• Microsoft Certified: Azure Data Engineer Associate.
• Databricks Certified Data Engineer Associate.

Skills

databricks, pyspark, python, sql