Data Engineer

Expleo South Africa is where bold ideas meet real-world impact. As a global leader in engineering, quality services, and management consulting, we help businesses accelerate digital transformation and shape the future. Joining Expleo means more than just a job - it's an opportunity to grow, innovate, and collaborate on projects that redefine industries. From smart factories and next-generation banking to cutting-edge digital solutions, you'll tackle complex challenges alongside inspiring colleagues. At Expleo, your career isn't just developed - it's turbo-charged. Together, let's push boundaries and change the game.

We are seeking a highly skilled Data Engineer to join our team in Johannesburg (Hybrid).

The ideal candidate will be responsible for designing, developing, and maintaining our data infrastructure to ensure the efficient processing and storage of large datasets. This role involves collaborating with cross-functional teams to deliver high-performance data solutions, enabling informed decision-making across the organization. You will play a key role in optimizing data pipelines, ensuring data integrity, and implementing best practices for scalability and performance in a dynamic environment.

Responsibilities

Data Pipeline Development:

• Design, build, and maintain scalable, efficient, and reliable data pipelines using Ab Initio, ensuring data flows seamlessly between various systems.
• Develop robust ETL (Extract, Transform, Load) processes to handle large volumes of data from diverse sources, enabling timely and accurate data delivery.
• Continuously improve and optimize pipeline architectures for future scalability and enhanced performance.

Data Integration and Management:

• Integrate structured and unstructured data from multiple sources to create a unified and consistent data environment.
• Ensure the integrity, accuracy, and consistency of data across all platforms and systems, implementing validation processes and data quality checks.
• Maintain data management policies and procedures to ensure compliance with industry standards and organizational goals.

Performance Optimization:

• Monitor the performance of data pipelines and ETL workflows to identify bottlenecks and areas for improvement.
• Implement best practices for data storage, retrieval, and archiving to ensure fast and reliable access to critical information.
• Utilize performance-tuning techniques to optimize data transformations and processing, ensuring minimal downtime and efficient resource utilization.

Collaboration and Communication:

• Act as a liaison between data scientists, business analysts, and other technical stakeholders to gather requirements and provide technical solutions that align with business objectives.
• Collaborate with cross-functional teams, including DevOps, system engineers, and business units, to ensure seamless data integration and project alignment.
• Communicate complex technical concepts in a clear and understandable manner, facilitating better decision-making for stakeholders.

Troubleshooting and Support:

• Proactively identify, troubleshoot, and resolve data-related issues, including inconsistencies, discrepancies, and performance issues, to ensure smooth operations.
• Provide timely support for data-related queries and requests from various teams, offering insights and technical assistance to optimize workflows.
• Maintain and update documentation for data processes, ensuring that all procedures and solutions are well-documented for future reference.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field.

Experience

Required Experience:

• 5+ years of experience in data engineering and ETL development.
• Extensive experience with Ab Initio, including developing and maintaining data pipelines.
• Proficiency in SQL and other database technologies.
• Experience with big data technologies such as Hadoop, Spark, and Hive.
• Knowledge of data modeling, data warehousing, and data integration best practices.
• Strong problem-solving and analytical skills.
• Excellent communication and collaboration abilities.

Preferred Experience:

• Experience with cloud platforms such as Azure.
• Familiarity with scripting languages like Python or R.
• Knowledge of data governance and security practices