Senior Data Engineer

MUST BE LOCATED IN CAPE TOWN

We are looking for a highly skilled Senior Data Engineer with a strong background in Databricks, Azure Data Services, and modern data engineering tools to design, build, and optimize scalable data pipelines. The ideal candidate will have at least 5 years of hands-on experience with Databricks and 5+ years of experience in data engineering, with proven expertise in ETL/ELT development, Spark-based processing, and Python/SQL programming.

• Design, build, and manage robust, scalable data pipelines using Databricks, Delta Live Tables, and Azure Data Factory (ADF) or Fivetran.
• Implement advanced ETL/ELT processes to ingest and transform data from multiple sources, including SAP.
• Develop and maintain reusable code using Python, SQL, and PySpark.
• Work with Unity Catalog to manage data governance, access controls, and data lineage across environments.
• Use Databricks notebooks and Git for collaborative development, version control, and CI/CD.
• Leverage Azure services and Terraform for provisioning, automation, and cloud infrastructure management.
• Optimize data models for performance, scalability, and operational efficiency in a cloud-native architecture.
• Collaborate closely with Data Analysts, Scientists, and Business Stakeholders to deliver clean, trusted datasets.
• Document technical solutions, data flows, and transformation logic.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, Information Systems, or related field.
• Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional) preferred.

Experience:

• 5+ years in data engineering or big data development roles.
• 5+ years hands-on experience with Databricks, including DLT and Unity Catalog.
• Experience working with Azure Data Factory (ADF) or Fivetran for ETL/ELT.
• Strong programming skills in Python and SQL.
• Proficiency in PySpark or Apache Spark for distributed data processing.
• Experience with Git, notebook-based development, and CI/CD pipelines.
• Familiarity with Azure administration and infrastructure as code (Terraform).
• Exposure to SAP systems as a data source is a plus.

Soft Skills:

• Strong problem-solving skills and attention to detail.
• Ability to work collaboratively in cross-functional teams.
• Excellent communication and documentation skills.
• Comfortable working in Agile environments.

R540 p/h or R85 000 perm depending in experience