Senior Data Engineering

Job Description for Data Engineer – Databricks Data PlatformRole Overview

Experienced Data Engineer to design, build, and maintain scalable data pipelines and analytics-ready data products on Databricks and AWS. End-to-end ownership of data ingestion, modelling, transformation, quality, and performance optimisation.

Key Responsibilities

·Build and maintain data pipelines using Databricks, Spark, and Delta Lake on AWS.

·Ingest data from databases, APIs, event streams, and flat files.

·Design and implement Medallion Architecture (Bronze / Silver / Gold layers).

··Write data transformation logic in PySpark, Spark SQL, and Python.

··Implement data quality checks, validation rules, and reconciliation processes.

·Optimise Spark workloads for performance, cost, and scalability.

·Integrate AWS services (S3, Glue, Lambda, Redshift, IAM) into data solutions.

·Support CI/CD for pipelines using Git, DevOps tooling, and release processes.

·Monitor pipelines, triage failures, and restore services within agreed SLAs.

·Maintain technical documentation: data models, pipeline flows, runbooks.

·Adhere to data security, privacy, and regulatory compliance standards.

Must-Have Skills

··Databricks: Workflows, Delta Live Tables, Unity Catalog, Databricks SQL.

·Apache Spark: PySpark and Spark SQL for large-scale data processing.

·Delta Lake: ACID transactions, schema evolution, time travel, merge.

·Medallion Architecture or equivalent layered data platform design.

·Python for pipeline development and transformation logic.

··AWS: S3, Glue, IAM, EC2, Lambda.

··Batch, micro-batch, and streaming ingestion (Structured Streaming / Kafka).

·Data modelling: star schema, snowflake schema, Data Vault.

·CI/CD, Git, version control, and Agile delivery practices.

Good-to-Have Skills

·Unity Catalog for data governance and lineage.

·AWS Kinesis for real-time streaming.

·Data Warehouse and Informatica PWC Knowledge

·dbt for SQL-based transformation workflows.

··AWS Redshift or Athena for cloud data warehousing.

··Terraform or AWS CloudFormation for infrastructure-as-code.

··AWS Step Functions for workflow orchestration.

·Data observability tools (Monte Carlo, AWS Glue Catalog, Atlas).

·Industry experience in insurance or financial services.

·Databricks or AWS Data Analytics certification.

·

Technical Environment

·Platform: Databricks Lakehouse, Delta Lake, Unity Catalog.

·Languages: Python, PySpark, SQL, Spark SQL.

··Ingestion: Auto Loader, APIs, JDBC, CDC tools.

··Processing: Apache Spark, Structured Streaming, Delta Live Tables.

·Cloud: AWS.

·Orchestration: Databricks Workflows, AWS Step Functions.

·DevOps: Git, CI/CD, Terraform, GitHub Actions.

·Analytics: Power BI, Tableau, Looker.

Experience and Qualifications

·Bachelor's in Computer Science, Engineering, IT, or equivalent.

·4-8 years in data engineering, ETL/ELT, and cloud data platforms.

··Production-grade pipeline delivery on AWS.

··Agile or DevOps-driven delivery environment.