Location
In office (Johannesburg)
Seniority
Senior
Employment
Full-Time
First seen by Alion on Nov 18, 2026.
Overview
Company
Impact
Profile match
About the role
- Our client in Fourways is looking for a Senior Data Engineer to join their team. The Data Engineer will provide hands-on technical delivery and optimisation support for the Digital Agriculture Agri Data Hub, developed on Azure Databricks. The platform integrates structured data from multiple on-premises databases and unstructured SharePoint content to support business intelligence, agronomic analytics and AI-driven capabilities. The project has progressed into User Acceptance Testing (UAT), and the resource will focus on resolving defects, stabilising data pipelines, optimising performance and supporting production readiness.
- The role requires a technically strong, delivery-focused engineer who can operate independently within an established architecture and contribute to ongoing MVP improvements.
- The Data Engineer will join an existing Digital Agriculture implementation focused on delivering a centralised Agri Data Hub within Azure Databricks. The solution incorporates structured data ingestion from on-premises databases into a Medallion Architecture, curated Gold-layer data products supporting BI reporting, and a Generative AI/LLM knowledge agent (Agripedia) consuming unstructured SharePoint information.
- Current project position: Analysis and design, core architecture, Azure networking, CI/CD pipelines, Gold-layer data modelling and initial BI and LLM development have been completed. The BI reports and Agripedia solution are currently in UAT, with defect resolution and production readiness activities underway.
- Technical delivery and stabilisation: Review and understand the existing Databricks implementation, ingestion pipelines, data transformations and Gold-layer models. Investigate and resolve defects identified during UAT, troubleshoot integration and data-quality issues, and implement improvements required to stabilise the solution.
- Pipeline development and optimisation: Develop, maintain and optimise ingestion pipelines for structured data from on-premises databases. Implement scalable transformation logic using Spark, PySpark and SQL, ensuring that pipelines are performant, secure, cost-efficient and supportable. Apply Delta Lake optimisation techniques and maintain consistency across the established Medallion Architecture.
- Production readiness and deployment: Support the transition from UAT into production, including validation of data accuracy, pipeline execution, error handling, monitoring and recovery processes. Work within existing Azure DevOps and CI/CD deployment processes, ensuring changes are appropriately tested and deployed.
- Cost, performance and operational improvements: Monitor and optimise Databricks compute usage, pipeline performance and associated cloud costs as the solution moves into production. Identify opportunities to improve operational efficiency, stability and maintainability.
- AI and unstructured data integration: Support the existing Agripedia LLM knowledge-agent solution where data engineering input is required, particularly relating to SharePoint data ingestion, integration, data preparation and ongoing optimisation. Previous experience supporting AI/LLM data pipelines would be advantageous.
- Governance, documentation and knowledge transfer: Adhere to existing enterprise governance and architecture standards, contribute to data-quality controls, and update technical documentation where required. Collaborate with the Data Architect and internal technical teams to support knowledge transfer, troubleshooting and further enhancements.
- Ongoing MVP enhancements: As the solution is being delivered iteratively, support further development and refinement of data pipelines, curated data products and platform capabilities based on business priorities, testing feedback and production requirements.
What they're looking for
- Databricks, Microsoft Azure, Spark, PySpark, SQL, Delta Lake, Medallion Architecture (Bronze/Silver/Gold), Azure DevOps, CI/CD, Data Integration, Data Quality, Performance and Cost Optimisation. Exposure to Generative AI/LLM solutions and SharePoint integration would be advantageous.
- Strong intermediate to senior-level engineer with proven hands-on Databricks experience in production environments. Must demonstrate strong troubleshooting and problem-solving capabilities, with the ability to understand existing solutions, identify root causes and implement practical improvements. Comfortable working within an established architecture and governed enterprise environment, collaborating with architects, technical teams and business stakeholders. Must be proactive, adaptable and capable of taking ownership of technical activities with minimal supervision. Experience supporting UAT, production deployments and iterative MVP delivery is important.
Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
2,382,526 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Free forever. No card. Under a minute.
Your match
How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.
Recommended for you based on this role
Data Science
Similar stack
Same company
Johannesburg
Senior Data Engineer
2 hours ago
In office · Full-Time · South Africa
Python
Java
C#
C#
.NET
Databases
Apache Kafka
AI/ML
Hadoop
Spark
Analytics
ETL/ELT
Apply
Enterprise Data Architect
2 hours ago
In office · Full-Time · Johannesburg
Python
SQL
AI/ML
Feature Store
Machine Learning
DevOps
Azure
AWS
AWS Lambda
Analytics
ETL/ELT
Master Data Management
Apply
Data Science & AI Enablement Executive
8 days ago
In office · Contractor · 12+ years exp · Master's Degree · Cape Town
Python
SQL
AI/ML
Machine Learning
Analytics
Master Data Management
Apply
Senior Data Scientist (Full Stack)
1 day ago
In office · Full-Time · Johannesburg
Python
SQL
Databases
Cassandra
AI/ML
Hadoop
Kubeflow
TensorFlow
PyTorch
Machine Learning
DevOps
GCP
Azure
AWS
Analytics
Power BI
Microsoft Excel
Apply
Senior Data Scientist (Full Stack)
1 day ago
In office · Full-Time · Johannesburg
Python
SQL
Databases
Cassandra
AI/ML
Hadoop
Kubeflow
TensorFlow
PyTorch
Machine Learning
DevOps
GCP
Azure
AWS
Analytics
Power BI
Microsoft Excel
Apply
up to $1.1M per year (gross) · In office · Full-Time · Johannesburg
TypeScript
C#
C#
.NET
DevOps
Azure
AWS
Apply
Apply
Apply
Apply
Apply
This is one of many
2,382,526 more open roles from verified company boards, updated every day.

