Data Engineer

Responsibilities:

• Legacy Data Extraction: Design, build, and optimise high-throughput data extraction pipelines from legacy IBM DB2 transactional databases without locking production environments.
• Cloud Platform Build: Develop production-grade ETL pipelines and data lake storage architectures directly on the target AWS data platform.
• Schema and Code Modernisation: Collaborate with the Solution Architect and Tech Lead to migrate complex legacy schemas, handle large object (LOB) data types, and convert legacy stored procedures into cloud-native equivalents.
• Performance Tuning: Troubleshoot and resolve pipeline bottlenecks, optimise PySpark/Glue jobs, and handle massive data volumes under a strict Time and Materials delivery model.

Requirements:

• Enterprise Relational Databases: 4+ years of deep, hands-on experience with IBM DB2 (understanding of DB2 utilities, transaction logs, locking behaviour, and Change Data Capture/CDC mechanisms).
• Core AWS Data Stack: Advanced proficiency with AWS Glue, AWS DMS (Database Migration Service), AWS SCT (Schema Conversion Tool), and Amazon S3 data lake patterns.
• Big Data Frameworks: Heavy experience writing optimised PySpark, Spark SQL, or Scala jobs within AWS Glue or EMR.
• Data Warehousing: Direct exposure to target cloud data warehouses like Amazon Redshift, AWS Athena, or modern open table formats (Delta Lake, Apache Iceberg).
• Infrastructure as Code (IaC): Basic familiarity with Terraform or AWS CloudFormation to deploy data pipelines safely.

Skills

AWS DMS, AWS Glue, AWS SCT, DB2, DMS, Glue, PySpark, Scala, Spark, Spark SQL