Data Lead - Data Engineering

JOB DESCRIPTION

Data Lead

Purpose of the Role :

The Data Lead will own the design, development, governance and delivery of the organisation's data platform and data engineering capabilities. The role will lead data architecture and engineering on AWS and Databricks, ensuring that data is ingested, transformed, secured and made available for reporting and business use in a reliable, scalable and cost-effective manner.

Key Responsibilities :

- Define and maintain the data platform architecture, including lakehouse/data lake design and integration patterns.

- Lead the design and implementation of batch and incremental data pipelines across APIs, files and databases.

- Design and manage data storage, access and security configurations on AWS.

- Establish data cataloguing, data dictionary, ownership, stewardship and lineage practices, using Databricks Unity Catalog.

- Work with BI and reporting teams to define trusted reporting data models and support effective dashboard design.

- Establish standards for source control, code reviews, testing, release management and automated deployments.

- Lead, coach and develop data engineers and other team members.

Tech Stack :

- Databricks, Apache Spark/PySpark, SQL, Python, AWS (S3, IAM, Glue, CloudWatch), Git/GitHub, Terraform.

Experience & Knowledge Required :

- 15 - 20 years of relevant experience across data engineering, data architecture, data platforms or related technology functions.

- Demonstrated experience designing and delivering enterprise data platforms and production-grade data pipelines.

- Experience leading teams and delivering complex data initiatives.

Qualifications :

Bachelor's degree in Computer Science, Information Technology, Engineering or a related discipline, or equivalent practical experience.

Skills

Data Engineering, Data Analytics, DataLake, Data Integration, Databricks, Apache Spark, PySpark, SQL, Python, AWS, Data Pipeline