M28 - Data Engineer

Role Overview

We are seeking an experienced Data Engineer to design, build, and deploy a scalable and reliable data lakehouse platform on AWS. The role will focus on developing end-to-end data pipelines, establishing robust data quality frameworks, and delivering production-ready solutions that form the foundation of the organisation's data infrastructure.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using AWS services such as Glue, Step Functions, Lambda, and S3.
• Architect and implement data lakehouse solutions using Apache Iceberg, S3 Tables, or similar open table formats.
• Implement lakehouse capabilities including schema evolution, partition evolution, and ACID transactions.
• Optimise data pipelines and storage solutions for performance, scalability, reliability, and cost efficiency.
• Define and implement automated data quality validation frameworks to ensure data accuracy, completeness, and consistency.
• Establish data quality metrics, monitoring, and alerting mechanisms across the data platform.
• Implement and maintain data governance standards and ensure compliance with relevant policies and requirements.
• Develop production-quality code and deploy solutions on AWS cloud infrastructure.
• Implement CI/CD practices to support automated, repeatable, and reliable deployments.
• Use infrastructure-as-code tools such as Terraform or CloudFormation to provision and manage cloud infrastructure.
• Work closely with cross-functional teams to design and deliver data engineering solutions.
• Communicate technical designs and concepts clearly to both technical and non-technical stakeholders.
• Produce comprehensive technical documentation covering architecture, pipelines, deployment, operations, and support procedures.
• Support knowledge transfer and handover of developed solutions to Day 2 operations and support teams.

Qualifications & Experience

• Degree in Computer Science, Data Engineering, Information Systems, or a related discipline.
• 3–5 years of relevant experience in data engineering, ETL/ELT development, or data platform engineering.
• Strong hands-on experience with AWS cloud services, particularly Glue, Step Functions, Lambda, and S3.
• Strong experience designing and developing scalable data pipelines and data lakehouse architectures.
• Hands-on experience with Apache Iceberg, S3 Tables, or similar open table formats.
• Good understanding of Apache Iceberg capabilities including schema evolution, partition evolution, ACID transactions, and table optimisation.
• Experience implementing automated data quality frameworks, validation rules, monitoring, and data governance controls.
• Experience developing and deploying production-grade applications or data solutions in cloud environments.
• Experience with CI/CD practices and infrastructure-as-code tools such as Terraform or CloudFormation.
• Experience working in a Government Commercial Cloud (GCC) environment is strongly preferred.
• Strong analytical, problem-solving, communication, and stakeholder management skills.
• AWS certifications such as AWS Certified Data Engineer, AWS Certified Solutions Architect, or equivalent certifications would be an advantage.

Skills

Snowflake Cloud Data Warehouse, Design, Database Schemas, Data Quality Standards, Data Pipeline, AWS, Optimisation Techniques, Information Systems, Data Solutions, Cloudformation, ETL, Infrastructure as Code (IaC), Data Governance, Computer Science, Cloud Deployment, Data Engineering, Data Infrastructure, AWS Lambda, S3