{"id":1932401,"url":"https://alion.io/job/job-mail-senior-data-engineer-jhb","title":"Senior Data Engineer - JHB","company":{"id":701519,"name":"Job Mail","domain":"jobmail.co.za","url":"https://alion.io/company/job-mail","size_band":null,"is_staffing_agency":true,"employer_type":"agency","is_intermediary":true,"listed_via":"jobmail.co.za","ats_vendor":"Schema","truth_index":{"grade":"C","score":64,"open_postings":6738,"ghost_share":0.497,"stale_share":0,"repost_share":0.723,"time_to_fill_p50_days":7,"computed_at":"2026-10-20T05:45:00Z"}},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Johannesburg, South Africa"],"countries":["ZA"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":38000,"max_usd":72000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":10},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Apache Hudi","optional":false},{"name":"Apache Kafka","optional":false},{"name":"Apache NiFi","optional":false},{"name":"CI/CD","optional":false},{"name":"Druid","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Flink","optional":false},{"name":"GDPR","optional":false},{"name":"Linux","optional":false},{"name":"Machine Learning","optional":false},{"name":"MySQL","optional":false},{"name":"PostgreSQL","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"Superset","optional":false}],"status":"live","first_seen_at":"2026-10-05T22:19:40Z","employer_posted_date":"2026-10-14","last_verified_at":"2026-10-14T15:20:30Z","board_verified":false,"closed_at":null,"days_open":14,"trust":{"level":"ok","repost_count":1,"flags":[],"days_open":14},"description":"Hire Resolve's client is looking for a Senior Data Engineer to join their team in Johannesburg, GP. The Data Engineer will be responsible for designing, building, and maintaining scalable data pipelines to support telecommunications CDR processing, real-time data ingestion, and analytical workloads.Thisrole requires expertise in data modeling, ETL development, stream processing, and distributed data systems. The ideal candidate will work closely with developers, DevOps, and analytics teams to transform raw network data into clean, structured, and query-ready datasets that power dashboards, machine learning models, and business logic. The candidate will provide technical leadership, optimize data workflows for performance and reliability, and drive best practices in data engineering methodologies\n\nResponsibilities:\nBuild ETL/ELT pipelines for ingesting, cleansing, and transforming CDRs and telecommunications logsfrom multiple network elements (5G/4G/3G/2G).\n\nDesign and maintain real-time data flows using Kafka, Apache NiFi and Apache Flink.\nWork with large-scale distributed file systems for batch and streaming ingestion.\nIntegrate and structure data for analytics platform like Apache Druid, Hudi and Superset.\nDevelop CI/CD pipelines for deploying data workflows and transformation logic.\nEnsure data quality, schema validation and compliance with retention and security policies.\nMonitor data pipeline health and optimize performance throughput and cost efficiency.\nWrite complex and performant queries for data validation, transformation, aggregation, and analyticsacross relational and distributed platforms.\n\nDevelop and optimize big data processing workflows in platforms like Apache Spark, Hive and Druid.\nEstablish efficient issue tracking and workflow processes, enhancing productivity and collaborationacross engineering teams.\n\nImplement security best practices and compliance frameworks to safeguard infrastructure, data, andapplications from vulnerabilities and threats.\n\nMaintain secure role-based access control mechanisms, encryption strategies, and identity managementsolutions to protect sensitive data and ensure regulatory compliance.\n\nMapping data flows from source to transformation to consumption.\nDesign and implement full-text search and indexing solutions for querying and retrieval of structured and unstructured telecommunications data using Apache tools or similar search engines.\nAnalyze and estimate and implement storage requirements and strategies for large-scale CDR datasetsand real-time data streams, ensuring optimal resource allocation and scalability across environments.\n\nEnsure data integrity and consistency across ingestion, transformation and storage layers throughvalidation checks, schema enforcement and robust error handling mechanisms.\n\nDevelop and maintain quality monitoring tools to proactively detect anomalies, missing records or datacorruption across pipelines.\n\nPerform other duties as assigned\n\nRequirements:Bachelor’s degree in Computer Engineering, Software Engineering, Computer Science, or a related field.\nStrong experience in building data pipelines using tools like Apache NiFi, Kafka, Airflow or similar.\nProficiency inSQL, Python, database administration and management like PostgreSQL, MySQL.\nSolid understanding of distributed data systems like Hive and Hudi and Spark.\n\nExperience with streaming frameworks like Kafka Streams, Apache Flink and Apache Beam.\n\nFamiliarity with data serialization formats like JSON.\n\nKnowledge of SFTP and secure data transfer mechanisms for ingesting remote files.\nProficient with Linux environments, shell scripting and storage systems like Ceph.\nExperiencewith data governance, including data privacy and regulatory compliance like GDPR andimplementing access control, auditing and data usage policies.\n\nExperience in maintaining central inventory of data assets, managing metadata and enabling searchablediscovery across structured and unstructured datasets.\n\nExperience in data lineage tracking to map data flows, visualize and track dependencies.\nExperience with OLAP systems,analytical modelling and columnar databases and designing andquerying multidimensional cubes.\n\nStrong problem-solving skills, ability to work in a fast-paced environment, and manage multiple projectsefficiently.\n\nStrong collaboration skills, adaptability, and a commitment to continuous learning\nHow to Apply: If you would like to apply for this role, kindly forward your CV to Gaby Turner at  or you may forward your CV to ","description_format":"text","description_chars":4576,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["Continuous learning"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-10-05T22:19:40Z"},{"event":"close","at":"2026-10-06T02:56:51Z"},{"event":"reopen","at":"2026-10-14T15:20:30Z"},{"event":"close","at":"2026-10-15T19:42:59Z"},{"event":"restore","at":"2026-10-16T04:59:13Z"}],"visa":[],"liveness":{"score":12,"band":"cold","label":"Long shot","p_open":1,"p_active":0.352,"p_room":0.35,"age_days":14,"expected_fill_days":7,"reasons":["seen:5","agency","velocity","win:tail","comp:brand"],"computed_at":"2026-10-20T05:45:00Z"},"pay":null,"html_url":"https://alion.io/job/job-mail-senior-data-engineer-jhb","json_url":"https://alion.io/job/job-mail-senior-data-engineer-jhb.json","meta":{"generated_at":"2026-10-20T16:07:21Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":18305,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-21T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":701519},"rest":"https://alion.io/mcp/rest/get_company?id=701519"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fjob-mail-senior-data-engineer-jhb"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fjob-mail-senior-data-engineer-jhb"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fjob-mail-senior-data-engineer-jhb"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/job-mail-senior-data-engineer-jhb\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fjob-mail-senior-data-engineer-jhb"}]}