Data Engineer

To strategically lead and support the API and Data Engineering function, enabling secure, scalable, and high-performance integration and data processing capabilities across the organisation. This role is responsible for hands-on design, development, and deployment of APIs and data pipelines using modern technologies such as FastAPI, REST APIs, Databricks, and SQL, while leveraging containerization and cloud-native platforms including Docker, Azure Container Apps (ACA), and AKS.

The role ensures seamless data movement and transformation through tools like Azure Data Factory (ADF) or equivalent, aligns API-led integration and data engineering initiatives with business and digital priorities, and provides expert guidance on microservices architecture, data platforms, and best practices within a scalable Lakehouse ecosystem.

Responsibilities:

• Design, develop, and maintain API-led integration solutions and data engineering pipelines to support business applications.
• Build and enhance scalable REST APIs using frameworks such as FastAPI, ensuring high performance and reliability.
• Develop and optimise data processing workflows using Databricks (PySpark, SQL) for efficient data transformation and analytics.
• Collaborate with business and technical teams to understand requirements and translate them into robust API and data solutions.
• Implement containerised applications using Docker and deploy/manage services on cloud platforms such as Azure Container Apps (ACA) and AKS.
• Develop and manage data integration workflows using Azure Data Factory (ADF) or equivalent to enable seamless data movement across systems.
• Ensure adherence to best practices in API design, microservices architecture, security, and performance optimisation.
• Troubleshoot, debug, and resolve issues related to APIs, data pipelines, and cloud deployments.
• Stay updated with emerging technologies and continuously improve technical skills in API engineering, cloud-native development, and data platforms.

Decision Making Authority:

• Decisions made alone - Management/maintenance/bug fixes/ stakeholders management.
• All items requiring key architectural calls, investments, or additional expense should be consulted with the manager and above.
• List of internal and external stakeholders the role is expected to interact with to execute duties effectively.
• Internal Stakeholders: Business Team Members - Wheels, Collections, emerging business and Non- Wheels.
• External Stakeholders: FSS Subsidiary members and Data vendors.

Key Challenges:

• Ensuring high performance, scalability, and reliability of REST APIs under varying workload conditions.
• Managing secure API integrations and data exchange across multiple systems with proper authentication and governance.
• Deploying, monitoring, and optimising containerised applications on Docker, ACA, and AKS environments.
• Handling large-scale data processing and optimisation in Databricks (PySpark, SQL) while maintaining efficiency.
• Troubleshooting and maintaining stability across APIs, data pipelines, and cloud-native infrastructure.

Requirements:

• Develop scalable and high-performance REST APIs using FastAPI.
• Strong hands-on experience with Databricks (PySpark, SQL) for data processing.
• Experience in building and managing data pipelines and integrations.
• Proficiency in SQL for data querying, transformation, and optimisation.
• Hands-on experience with Docker and deployment on ACA/AKS environments.
• Ability to develop and manage workflows using Azure Data Factory (ADF) or equivalent.
• Good understanding of database design and data architecture.
• Strong debugging, troubleshooting, and performance tuning skills.
• Python programming for API development and data engineering use cases.