Senior Data Engineer

Responsibilities:

• Design, develop, test, and maintain optimal data pipeline and ETL architectures.
• Map out data systems and define/design required integrations, ETL, BI, and AI systems/processes.
• Prepare and optimise data for predictive and prescriptive modelling.
• Collaborate with teams to integrate ERP data into the enterprise data lake, ensuring seamless flow and quality.
• Enhance cloud data infrastructure on AWS or Azure for scalability and performance.
• Utilise big data tools and frameworks to optimise data acquisition and preparation.
• Build architectures to move data to/from data lakes and data warehouses for advanced analytics.
• Develop and curate data models for analytics, dashboards, and reports.
• Conduct code reviews, maintain production-level code, and implement testing approaches.
• Monitor, troubleshoot, and resolve data ingestion workflows to maintain reliability and uptime.
• Drive innovation and implement efficient new approaches to data engineering tasks.

Requirements:

• Bachelor's degree in Computer Science, Mathematics, Engineering, or a related field.
• 5+ years of experience working with enterprise data platforms, including building and managing data lakes.
• QX | Public: 3-5 years of experience designing and implementing data warehouse solutions.
• Expertise in SQL, including developing stored procedures (SPs) and applying advanced data design concepts
• Proficiency in Spark (Python/Scala) and Spark Streaming for real-time data pipelines.
• Experience with AWS or Azure services (e. g., AWS Glue, Azure Data Factory, Redshift, Snowflake).
• Familiarity with big data tools such as Apache Kafka, Apache Spark, or Flink.
• Hands-on experience with orchestration tools (e. g., Apache Airflow, Prefect).
• Knowledge of CI/CD processes, version control (e. g., Git, Jenkins), and deployment automation.
• Strong problem-solving, communication, and collaboration skills.

Good-to-Have Skills:

• Experience in integrating ERP data into data lakes.
• Experience with traditional ETL tools (e. g., Talend, Pentaho).

Skills

apache spark, data lake, data warehouse, etl, redshift, snowflake, spark, sql