Senior Data Engineer | Data Integration Developer

Introduction

Centurion  |  Gauteng  | Permanent

7+ years experience required - Senior

Hands on Engineer  | Data Integration Developer

This role is for someone that is not afraid to get their hands dirty.

 

Hands-on role that owns the full data engineering lifecycle:

• from ingestion and transformation through to platform architecture and production deployment – designing, building and running complex data pipelines on AWS.

 

Technical Skills Required:

• Python (advanced) – data pipeline development, automation and scripting.
• SQL (advanced) – complex query writing, performance optimisation and stored procedures.
• AWS (mid–advanced) – S3, Glue, Lambda, Redshift, RDS, IAM, CloudWatch, Athena.
• Data warehousing and data modelling – star/snowflake schemas, dimensional and analytics-oriented design.
• ETL/ELT pipeline development – batch, incremental and near-real-time patterns.
• Data lakehouse architecture on cloud-native platforms.
• CI/CD and Git – version control, automated testing and deployment pipelines.
• Power BI – supporting BI teams, data model integration and semantic layer understanding.
• Experience ingesting data from relational databases (e.g. SQL Server, Oracle), cloud storage, APIs, SFTP, MongoDB and file-based sources.

 

Highly Valued (Differentiating Skills):

• Apache Spark / PySpark for large-scale distributed processing.
• dbt for analytics engineering and transformation layer management.
• Apache Airflow or AWS Step Functions for orchestration.
• Terraform or CloudFormation (infrastructure-as-code).
• Real-time streaming – Kafka, Kinesis or similar.
• Machine learning workflow support – feature engineering and model data pipelines.
• Microsoft Azure or GCP as secondary cloud exposure.

 

Senior Data Engineer, AWS, Python, SQL, Data Engineering, ETL/ELT, Data Lakehouse, Data Warehousing, Power BI, CI/CD, Agile, AI-enabled engineering, Financial Services, Centurion, Hybrid

Duties & Responsibilities

Delivery Ownership within Agile Execution

• Contribute to data engineering outcomes aligned to sprint goals and programme-level commitments.
• Take part in Agile ceremonies – planning, estimation, prioritisation and continuous improvement.
• Decompose data features into implementable tasks with reliable effort estimates.
• Ensure outputs meet agreed functional, performance and data quality expectations.

Data Engineering & Pipeline Development

• Design and implement scalable ingestion and transformation pipelines across multiple systems and data domains, supporting batch and incremental processing.
• Build in error handling, monitoring and alerting to keep pipelines robust.
• Implement data validation and reconciliation mechanisms to maintain confidence in data assets.

Platform Design & Architecture (AWS)

• Design solutions aligned to the data platform strategy and target architecture, and contribute to the evolution of the cloud-based data lakehouse environment.
• Assess the impact of design choices on security, performance, cost and supportability.
• Identify integration points, upstream/downstream dependencies and risks early in the delivery lifecycle.

Technical Leadership, Coaching & Standards

• Provide guidance and technical oversight to less experienced data engineers.
• Support analytics, BI and data science teams with clarity on data structures, availability and pipeline behaviour.
• Review code and configuration, and help define shared standards, patterns and best practices.
• Optimise and simplify existing pipelines and data models.

AI-Enabled Engineering (Governance-Led)

• Promote responsible use of AI to improve engineering productivity and solution quality, within group and business-unit governance frameworks.
• Use only enterprise-approved AI tooling, and ensure no sensitive, proprietary or client information is exposed to public or unapproved AI platforms.
• Validate all AI-assisted outputs and retain accountability for correctness, compliance and production readiness.
• Apply AI to understand data flows and lineage, improve pipeline logic, support query formulation and test scenario creation, and accelerate documentation.

DevOps, Operational Support & Incident Response

• Apply version control, automated testing, CI/CD and structured releases.
• Contribute to operational readiness, deployment pipelines and monitoring.
• Diagnose and resolve data incidents, perform root cause analysis and implement corrective actions.

Security, Risk, Compliance & Governance

• Build solutions with privacy, access control and regulatory considerations embedded by design.
• Identify and mitigate data risks, and support remediation of audit findings and compliance gaps.
• Maintain accurate documentation of pipelines, data models and operational considerations, and keep work tracking up to date in Jira.
• Adhere to delivery governance, change management and release processes.

Stakeholder Engagement

• Communicate technical progress, constraints and decisions clearly to business and technical stakeholders.
• Collaborate across technology and analytics teams and contribute positively to team culture.

Desired Experience & Qualification

Skills & Experience:

• 7+ years’ professional experience in data engineering or related roles.
• Demonstrated delivery of production-ready, end-to-end data pipelines and cloud data platforms.
• Strong exposure to AWS-based data architectures.
• Experience across the full stack – ingestion, transformation, modelling and serving – not just a single layer.
• Hands-on experience working within Agile or SAFe delivery environments.
• Experience in financial services, insurance, investment management or other regulated data environments (preferred).
• Practical experience applying AI tooling in engineering workflows – including using AI responsibly and validating its outputs – to support pipeline understanding, optimisation, testing or documentation.
• Experience coaching or guiding other engineers.

Interested?

Apply to [email protected]  or on PNET