Data Engineer

Responsibilities:

• Lead enterprise-scale data engineering initiatives and cloud modernisation programs.
• Design scalable batch and real-time data pipelines across AWS and GCP platforms.
• Drive Data Lake and Lakehouse architecture implementations using Databricks and Snowflake.
• Mentor engineering teams and establish engineering best practices.
• Collaborate with architects, DevOps, business stakeholders, and cross-functional teams.
• Design, develop, and maintain scalable ETL/ELT pipelines.
• Build batch and real-time data ingestion frameworks.
• Develop optimised SQL transformations and scalable data processing solutions.
• Implement distributed data processing using Spark/PySpark.
• Perform query optimisation, partitioning, clustering, and workload tuning.
• Implement orchestration, monitoring, alerting, and operational support.
• Ensure data quality, governance, lineage, and metadata management.
• Support troubleshooting and production issue resolution.
• Enable business intelligence and analytics use cases through scalable curated datasets and optimised data models.
• Collaborate with BI and analytics teams supporting reporting platforms such as Power BI, AWS QuickSight, or equivalent visualisation tools.

Requirements:

• Skills: AWS, GCP, Databricks, Snowflake, Spark, Delta Lake, Python, SQL, PySpark, Airflow, Kafka, AWS Glue, Dataflow, Redshift, Athena, Power B/Quicksight, AWS.
• Good to have QuickSight, or equivalent analytics/reporting platforms, ETL/ELT frameworks and distributed data processing, CI/CD, Terraform, Docker, Kubernetes, Data governance, security, monitoring, and cost optimisation.