{"id":3945244,"url":"https://alion.io/job/active-recruitment-senior-data-engineer-data-integration-developer","title":"Senior Data Engineer | Data Integration Developer","company":{"id":4527027,"name":"Active Recruitment","domain":"activerecruitment.co.za","url":"https://alion.io/company/active-recruitment-2","size_band":null,"is_staffing_agency":true,"employer_type":"agency","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Pretoria, South Africa"],"countries":["ZA"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"Airflow","optional":false},{"name":"Amazon CloudWatch","optional":false},{"name":"Amazon Kinesis","optional":false},{"name":"Amazon Redshift","optional":false},{"name":"Amazon S3","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"AWS Lambda","optional":false},{"name":"AWS Step Functions","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"CloudFormation","optional":false},{"name":"dbt","optional":false},{"name":"ETL/ELT","optional":false},{"name":"GCP","optional":false},{"name":"Git","optional":false},{"name":"IAM","optional":false},{"name":"Jira","optional":false},{"name":"Machine Learning","optional":false},{"name":"Power BI","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false}],"status":"live","first_seen_at":"2026-10-31T16:01:03Z","employer_posted_date":null,"last_verified_at":"2026-10-31T16:01:03Z","board_verified":false,"closed_at":null,"days_open":8,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":8},"description":"Introduction\nCenturion | Gauteng | Permanent\n\n7+ years experience required - Senior\n\nHands on Engineer | Data Integration Developer\n\nThis role is for someone that is not afraid to get their hands dirty.\n\nHands-on role that owns the full data engineering lifecycle:\n\n from ingestion and transformation through to platform architecture and production deployment – designing, building and running complex data pipelines on AWS.\n\nTechnical Skills Required:\n\n Python (advanced) – data pipeline development, automation and scripting.\n\n SQL (advanced) – complex query writing, performance optimisation and stored procedures.\n\n AWS (mid–advanced) – S3, Glue, Lambda, Redshift, RDS, IAM, CloudWatch, Athena.\n\n Data warehousing and data modelling – star/snowflake schemas, dimensional and analytics-oriented design.\n\n ETL/ELT pipeline development – batch, incremental and near-real-time patterns.\n\n Data lakehouse architecture on cloud-native platforms.\n\n CI/CD and Git – version control, automated testing and deployment pipelines.\n\n Power BI – supporting BI teams, data model integration and semantic layer understanding.\n\n Experience ingesting data from relational databases (e.g. SQL Server, Oracle), cloud storage, APIs, SFTP, MongoDB and file-based sources.\n\nHighly Valued (Differentiating Skills):\n\n Apache Spark / PySpark for large-scale distributed processing.\n\n dbt for analytics engineering and transformation layer management.\n\n Apache Airflow or AWS Step Functions for orchestration.\n\n Terraform or CloudFormation (infrastructure-as-code).\n\n Real-time streaming – Kafka, Kinesis or similar.\n\n Machine learning workflow support – feature engineering and model data pipelines.\n\n Microsoft Azure or GCP as secondary cloud exposure.\n\nSenior Data Engineer, AWS, Python, SQL, Data Engineering, ETL/ELT, Data Lakehouse, Data Warehousing, Power BI, CI/CD, Agile, AI-enabled engineering, Financial Services, Centurion, Hybrid\n\nDuties & Responsibilities\nDelivery Ownership within Agile Execution\n\n Contribute to data engineering outcomes aligned to sprint goals and programme-level commitments.\n\n Take part in Agile ceremonies – planning, estimation, prioritisation and continuous improvement.\n\n Decompose data features into implementable tasks with reliable effort estimates.\n\n Ensure outputs meet agreed functional, performance and data quality expectations.\n\nData Engineering & Pipeline Development\n\n Design and implement scalable ingestion and transformation pipelines across multiple systems and data domains, supporting batch and incremental processing.\n\n Build in error handling, monitoring and alerting to keep pipelines robust.\n\n Implement data validation and reconciliation mechanisms to maintain confidence in data assets.\n\nPlatform Design & Architecture (AWS)\n\n Design solutions aligned to the data platform strategy and target architecture, and contribute to the evolution of the cloud-based data lakehouse environment.\n\n Assess the impact of design choices on security, performance, cost and supportability.\n\n Identify integration points, upstream/downstream dependencies and risks early in the delivery lifecycle.\n\nTechnical Leadership, Coaching & Standards\n\n Provide guidance and technical oversight to less experienced data engineers.\n\n Support analytics, BI and data science teams with clarity on data structures, availability and pipeline behaviour.\n\n Review code and configuration, and help define shared standards, patterns and best practices.\n\n Optimise and simplify existing pipelines and data models.\n\nAI-Enabled Engineering (Governance-Led)\n\n Promote responsible use of AI to improve engineering productivity and solution quality, within group and business-unit governance frameworks.\n\n Use only enterprise-approved AI tooling, and ensure no sensitive, proprietary or client information is exposed to public or unapproved AI platforms.\n\n Validate all AI-assisted outputs and retain accountability for correctness, compliance and production readiness.\n\n Apply AI to understand data flows and lineage, improve pipeline logic, support query formulation and test scenario creation, and accelerate documentation.\n\nDevOps, Operational Support & Incident Response\n\n Apply version control, automated testing, CI/CD and structured releases.\n\n Contribute to operational readiness, deployment pipelines and monitoring.\n\n Diagnose and resolve data incidents, perform root cause analysis and implement corrective actions.\n\nSecurity, Risk, Compliance & Governance\n\n Build solutions with privacy, access control and regulatory considerations embedded by design.\n\n Identify and mitigate data risks, and support remediation of audit findings and compliance gaps.\n\n Maintain accurate documentation of pipelines, data models and operational considerations, and keep work tracking up to date in Jira.\n\n Adhere to delivery governance, change management and release processes.\n\nStakeholder Engagement\n\n Communicate technical progress, constraints and decisions clearly to business and technical stakeholders.\n\n Collaborate across technology and analytics teams and contribute positively to team culture.\n\nDesired Experience & Qualification\nSkills & Experience:\n\n 7+ years’ professional experience in data engineering or related roles.\n\n Demonstrated delivery of production-ready, end-to-end data pipelines and cloud data platforms.\n\n Strong exposure to AWS-based data architectures.\n\n Experience across the full stack – ingestion, transformation, modelling and serving – not just a single layer.\n\n Hands-on experience working within Agile or SAFe delivery environments.\n\n Experience in financial services, insurance, investment management or other regulated data environments (preferred).\n\n Practical experience applying AI tooling in engineering workflows – including using AI responsibly and validating its outputs – to support pipeline understanding, optimisation, testing or documentation.\n\n Experience coaching or guiding other engineers.\n\nInterested?\nApply to  or on PNET","description_format":"text","description_chars":6009,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-11-08T18:59:55Z"}],"visa":[],"liveness":{"score":51,"band":"ok","label":"Likely open","p_open":1,"p_active":0.511,"p_room":1,"age_days":8,"expected_fill_days":29,"reasons":["seen:8","agency","win:early"],"computed_at":"2026-11-08T20:32:41Z"},"pay":null,"html_url":"https://alion.io/job/active-recruitment-senior-data-engineer-data-integration-developer","json_url":"https://alion.io/job/active-recruitment-senior-data-engineer-data-integration-developer.json","meta":{"generated_at":"2026-11-08T20:32:41Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":42925,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-09T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":4527027},"rest":"https://alion.io/mcp/rest/get_company?id=4527027"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Factive-recruitment-senior-data-engineer-data-integration-developer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Factive-recruitment-senior-data-engineer-data-integration-developer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Factive-recruitment-senior-data-engineer-data-integration-developer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/active-recruitment-senior-data-engineer-data-integration-developer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Factive-recruitment-senior-data-engineer-data-integration-developer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3945244}},{"tool":"search_jobs","arguments":{"company":"activerecruitment.co.za","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3945244},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}