Senior Data Engineer - AWS & Databricks (Public Sector)

At Xtremax, we're looking for a Senior Data Engineer to design, build and operate dependable data pipelines and reusable data products on AWS and Databricks. You'll help establish the data foundations that support analytics, reporting and machine learning, ensuring that information remains reliable, well-governed and ready for production use.

You'll lead engineering work across data ingestion, transformation, modelling, quality validation, orchestration, lineage and performance optimisation. Using Python, PySpark, SQL, Apache Spark and Delta Lake, you'll build production-grade pipelines that handle complex data sources, incremental updates and operational recovery while maintaining clear data contracts and ownership.

This role combines hands-on engineering with technical leadership. You'll work with data owners, analysts, data scientists, architects and platform teams to translate business needs into maintainable data solutions, improve engineering standards and develop reusable patterns that make data services easier to operate and scale.

Responsibilities

• Lead the design and implementation of reliable batch and streaming pipelines across AWS and Databricks.
• Build lakehouse architectures using Delta Lake and layered Bronze, Silver and Gold data patterns.
• Develop reusable data models, curated datasets and data products with clear definitions, ownership and service expectations.
• Implement source integration and incremental processing for databases, files, APIs and event streams, including change data capture, replay and recovery.
• Establish data contracts, schema enforcement, automated quality checks, reconciliation and anomaly detection.
• Build and maintain workflows using Databricks Workflows, Lakeflow and relevant AWS orchestration services.
• Implement metadata, lineage, classification and access controls using Unity Catalog, AWS Glue Data Catalog, Lake Formation or approved platform capabilities.
• Develop monitoring, dashboards, alerts and operational runbooks for pipeline health, data freshness, quality, latency and downstream impact.
• Optimise Spark and SQL workloads, partitioning, clustering, caching and cluster configuration to improve performance and cost efficiency.
• Apply cloud security practices covering least-privilege access, encryption, secrets management, environment separation and audit logging.
• Review technical designs and code, establish engineering standards, mentor engineers and promote reusable components.
• Collaborate with data owners, analysts, data scientists and governance stakeholders to define data requirements and acceptance criteria.
• Maintain technical documentation and transfer operational knowledge to support sustainable internal ownership.

Requirements

• Degree or diploma in Computer Science, Software Engineering, Data Engineering, Information Systems or a related technical discipline.
• Typically 6-10 years of relevant data or software engineering experience, including production data pipeline design and operation.
• At least two years of experience working with cloud-based data platforms.
• Strong Python, PySpark and SQL skills, including distributed processing, complex transformations, query optimisation, debugging and automated testing.
• Hands-on experience with Databricks, Apache Spark and Delta Lake for production-scale batch or streaming data engineering.
• Practical AWS data engineering experience across relevant services such as S3, Glue, Lake Formation, IAM, KMS, Lambda, Step Functions, EventBridge, CloudWatch or Redshift.
• Experience designing lakehouse or data-lake architectures, layered data patterns, dimensional or domain data models and reusable datasets.
• Experience with incremental ingestion, change data capture, idempotency, late-arriving data, backfills, replay and recovery.
• Experience implementing data quality validation, schema enforcement, reconciliation, metadata and lineage in production pipelines.
• Familiarity with workflow orchestration, Git-based development, CI/CD, infrastructure as code such as Terraform and environment promotion.
• Experience implementing monitoring, operational dashboards, alerting, runbooks, incident diagnosis and service recovery.
• Working knowledge of cloud security, data classification, access controls, encryption and audit logging.
• Ability to review designs and code, establish engineering standards and guide less-experienced engineers.

Preferred

• Experience with Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming or Delta Sharing.
• Familiarity with Apache Airflow, dbt, Kafka, Amazon MSK/Kinesis or AWS DMS.
• Experience with data products, data contracts, semantic layers or data mesh-aligned operating models.
• Experience supporting feature engineering, ML training datasets, feature stores or MLOps pipelines.
• Familiarity with Singapore public-sector security, data classification and governed data-sharing requirements.
• Databricks or AWS data engineering certifications.

Benefits

By submitting your resume/CV, you consent and agree to allow the information provided to be used and processed by or on behalf of Xtremax Pte Ltd for purposes related to your registration of interest in current or future employment with us and for the processing of your application for employment.

You also represent to us that you have obtained the consent of your referees when you disclose to us their personal data for the purpose of conducting reference checks.

The personal data held by us relating to your application will be kept strictly confidential and in accordance with the PDPA. You may also refer to our Privacy Policy for more details here:

We regret to inform you that should you not consent to providing the necessary data required for us to process your application, your application will be considered void.