Data Engineering Lead – Data Science and AI
DVT is a leading technology consulting and software engineering company delivering innovative solutions across Africa and internationally. We partner with clients to solve complex business challenges through software engineering, cloud platforms, data, artificial intelligence, and digital transformation. Our teams combine deep technical expertise with a strong consulting mindset to create measurable business value.
Our Data Practice is looking for an experienced Data Engineering Lead to design, build and govern the data and technology architecture underpinning our Data Science, Machine Learning, Generative AI and agentic AI initiatives. You will provide the engineering capability required to move analytical and AI solutions from experimentation into secure, scalable, production-ready services.
This hands-on technical leadership role combines architecture ownership, engineering delivery and coordination of engineering resources, with a core focus on establishing the Azure-based architecture used by the team, including governed data pipelines between our AWS data environment and Azure-based Data Science and AI platforms.
Requirements
Key Responsibilities:
Define and maintain the target data engineering architecture for Data Science and AI workloads, including data preparation, model deployment, GenAI applications and agentic workflows.
Design the architecture and integration patterns for bringing data from AWS into the Azure Data Science environment, working with IT, Data Platform, Cloud and Security teams.
Own data engineering delivery, building reusable pipelines, transformations and curated datasets, and standardised frameworks for ingestion, transformation, orchestration and publishing.
Define the approach to reusable features and data products, including feature-store capabilities, lineage, versioning and monitoring.
Partner with Data Scientists and Engineers to establish production-grade MLOps and LLMOps capabilities, including model deployment, versioning, monitoring and retraining.
Provide the data engineering foundations for Generative AI and agentic AI solutions, including ingestion, chunking, indexing and governance of enterprise information.
Ensure data and AI solutions align with governance, privacy, responsible AI, security and data quality standards.
Provide technical leadership to Data Engineers supporting the Data Science portfolio, and translate the roadmap into a prioritised engineering and architecture plan.
Establish monitoring, operational support, incident management and cost-optimisation practices, and act as primary technical liaison with architecture, infrastructure, cybersecurity and governance teams.
Essential Experience & Qualifications Required
Proven, senior-level experience (7+ years) in data engineering, architecture, and cloud platforms.
Degree in Computer Science, Information Systems, Data Engineering, Software Engineering, Mathematics or a related field.
Microsoft Azure and/or AWS architecture or engineering certifications advantageous.
Strong practical experience with Microsoft Azure data and AI services (e.g. Azure Data Factory, Azure Functions, Azure Storage, Azure Machine Learning), and experience integrating data across AWS and Azure environments.
Experience building and operating production data pipelines (batch, incremental, streaming or API-based), with data orchestration, automated testing, monitoring and deployment practices.
Experience supporting Data Science, machine learning or AI teams, including implementing MLOps or model deployment capabilities.
Strong Python and SQL skills, with experience in data modelling and transformation frameworks.
Experience with Git-based source control, CI/CD pipelines and Infrastructure as Code.
Experience working with data governance, information security and enterprise architecture teams, and leading technical delivery or coordinating engineers.
Advantageous Experience
Experience with Microsoft Fabric, Synapse Analytics or Databricks.
Experience with AWS services such as S3, Glue, Athena, Redshift, Lambda or SageMaker.
Experience building or supporting feature stores, model registries and experiment-tracking platforms.
Experience with containerisation, orchestration technologies, and relational, document, vector or analytical databases.
Exposure to FinOps and cloud cost optimisation practices.
Important to Know
This is a hands-on Data Engineering and Cloud Architecture leadership role embedded within the Data Science and AI function, not a pure infrastructure or platform-operations role. The focus is establishing the data architecture, pipelines and engineering foundations that enable Data Science, ML and AI delivery.