Sr. Data Engineer

The Data Engineer (Level III) designs, builds, and operates the data pipelines and lakehouse data products that power P& SC intelligence across device supply chain, reverse logistics, procurement, network supply chain, and real-time control tower capabilities. Operating within a squad focused on a specific SC domain or cross-cutting platform function, the Data Engineer is a hands-on technical contributor who builds to production quality, owns pipeline reliability and KTLO, and contributes to the engineering standard of the team.

Responsibilities:

• Design and build production-grade data pipelines spanning source ingestion, bronze landing, silver transformation, and gold-layer data product delivery within the squad's domain scope.
• Own pipeline KTLO (Keep the Lights On) for assigned data products, including monitoring, alerting, incident response, and ongoing reliability improvements.
• Implement data ingestion patterns for assigned source systems, including batch file ingestion, API-based ingestion, and event-driven streaming (Kafka, Azure Event Hub) depending on squad scope.
• Apply medallion architecture (bronze, silver, gold) and Fabric IQ certification standards consistently across all data product builds.
• Collaborate with System Analysts to implement field-level transformations, business rule logic, and data quality checks as specified in product requirements documentation.
• Participate in and contribute to pipeline design reviews, ensuring solutions align with the organisation's Databricks and Fabric engineering standards.
• Support the migration and deprecation of legacy platforms including SCOpsBI SQL Server and SAP boundary systems, following the organisation's extract, validate, rebuild, cutover, and decommission pattern.
• Write and maintain comprehensive pipeline documentation including data lineage, transformation logic, SLA definitions, and dependency maps.
• Contribute to the organisation's DevOps and engineering reliability practices, including CI/CD pipeline setup, testing frameworks, and incident runbooks.
• Support and collaborate with junior data engineers within the squad, sharing knowledge and providing guidance on day-to-day engineering tasks.

Requirements:

• 5-8 years of data engineering experience with a track record of delivering production-grade data pipelines in enterprise environments.
• Strong experience with Python, PySpark, Spark SQL, and T-SQL for pipeline development and transformation logic.
• Strong Proficiency in Azure Data Factory for building and maintaining ELT/ETL pipelines, including parameterisation, triggers, linked services, and error handling
• Hands-on experience with Azure Databricks, including Delta Lake, Unity Catalogue, and workflow orchestration.
• Hands-on experience with dbt (Data Build Tool), writing models, tests, sources, and documentation within an established project structure.
• Solid SQL skills for data validation, transformation logic, and ad-hoc source system analysis.
• Expert-level data modelling skills across 3NF, Dimensional (Kimball), and Data Vault 2.0 patterns.
• Strong understanding of data quality frameworks, including implementing checks, alerting on anomalies, and maintaining SLA-compliant pipeline health.
• Experience with DevOps practices for data pipelines, including version control (Git), CI/CD, and automated testing.
• Agile or Scrum practitioner certification or equivalent.
• Ability to operate with moderate independence on complex technical problems, escalating appropriately and maintaining clear technical documentation.

Nice to Have:

• Experience with real-time streaming technologies including Kafka, Azure Event Hub, Delta Live Tables, or Spark Structured Streaming.
• Working experience with Snowflake table design, query performance tuning, views, stored procedures, and data loading patterns (Snowpipe, COPY INTO)
• Experience with enterprise batch scheduling and orchestration tools such as Control-M or Autosys is a plus.
• Experience with real-time streaming technologies including Kafka, Azure Event Hub, Delta Live Tables, or Spark Structured Streaming.
• Experience with Microsoft Fabric or Azure Synapse Analytics; familiarity with Fabric IQ and OneLake is a plus.
• Background in supply chain, reverse logistics, procurement, or network infrastructure data domains.
• Experience working within Agile delivery frameworks including PI planning, sprint-level technical governance, and cross-squad dependency management.
• Familiarity with AI-assisted development tools such as GitHub Copilot or Claude, and experience integrating AI/ML-ready data preparation into ETL/ELT pipeline design.
• Working knowledge of SOX and USGCI compliance requirements as they apply to enterprise data platform design and deployment.
• Microsoft Azure certifications: DP-203 AZ-305 DP-900 or the Microsoft Fabric Analytics Engineer Associate.
• Python development skills beyond PySpark, including utility scripting and framework contributions.