Data Engineer - API & Platform Integration

To strategically lead and support the API and data engineering function, enabling secure, scalable, and high-performance integration and data processing capabilities across the organisation. This role is responsible for hands-on design, development, and deployment of APIs and data pipelines using modern technologies such as FastAPI, REST APIs, Databricks, and SQL while leveraging containerisation and cloud-native platforms including Docker, Azure Container Apps (ACA), and AKS. The role ensures seamless data movement and transformation through tools like Azure Data Factory (ADF) or equivalent, aligns API-led integration and data engineering initiatives with business and digital priorities, and provides expert guidance on microservices architecture, data platforms, and best practices within a scalable Lakehouse ecosystem.

Responsibilities:

• Design, develop, and maintain API-led integration solutions and data engineering pipelines to support business applications.
• Build and enhance scalable REST APIs using frameworks such as FastAPI, ensuring high performance and reliability.
• Develop and optimise data processing workflows using Databricks (PySpark, SQL) for efficient data transformation and analytics.
• Collaborate with business and technical teams to understand requirements and translate them into robust API and data solutions.
• Implement containerised applications using Docker and deploy/manage services on cloud platforms such as Azure Container Apps (ACA) and AKS.
• Develop and manage data integration workflows using Azure Data Factory (ADF) or equivalent to enable seamless data movement across systems.
• Ensure adherence to best practices in API design, microservices architecture, security, and performance optimisation. Troubleshoot, debug, and resolve issues related to APIs, data pipelines, and cloud deployments.
• Stay updated with emerging technologies and continuously improve technical skills in API engineering, cloud-native development, and data platforms.

Decision-Making Authority:

• Decisions made alone - Management/maintenance/bug fixes/stakeholder management.
• All items requiring key architectural calls, investments, or additional expense should be consulted with the manager and above.
• List of internal and external stakeholders the role is expected to interact with to execute duties effectively.
• Internal Stakeholders: Business Team Members - Wheels, Collections, Emerging Business and Non-Wheels.
• External Stakeholders: FSS subsidiary members and data vendors.

Key Challenges:

• Ensuring high performance, scalability, and reliability of REST APIs under varying workload conditions.
• Managing secure API integrations and data exchange across multiple systems with proper authentication and governance.
• Deploying, monitoring, and optimising containerised applications on Docker, ACA, and AKS environments.
• Handling large-scale data processing and optimisation in Databricks (PySpark, SQL) while maintaining efficiency.
• Troubleshooting and maintaining stability across APIs, data pipelines, and cloud-native infrastructure.

Requirements:

• Bachelor's or master's degree in computer science, information technology, engineering, or a related field with 4-8 years of experience in API development and data engineering.
• Knowledge of Gen AI tools.
• Strong understanding of REST API design principles and microservices architecture.
• Knowledge of FastAPI (or similar frameworks) and API development best practices.
• Understanding of Databricks (PySpark, SQL) and data engineering concepts.
• Knowledge of data pipelines, ETL processes, and data flow design.
• Familiarity with Docker, containerisation, and cloud platforms (ACA, AKS).
• Understanding of Azure Data Factory (ADF) or equivalent for data orchestration.
• Knowledge of SQL for querying, optimisation, and database design.
• Awareness of API security (OAuth, JWT, authentication/authorisation mechanisms).
• Banking/NBFC domain knowledge will be an added advantage.
• Knowledge of Gen AI tools and their application in development.
• Develop scalable and high-performance REST APIs using FastAPI.
• Strong hands-on experience with Databricks (PySpark, SQL) for data processing.
• Experience in building and managing data pipelines and integrations.
• Proficiency in SQL for data querying, transformation, and optimisation.
• Hands-on experience with Docker and deployment on ACA/AKS environments.
• Ability to develop and manage workflows using Azure Data Factory (ADF) or equivalent.
• Good understanding of database design and data architecture.
• Strong debugging, troubleshooting, and performance tuning skills.
• Python programming for API development and data engineering use cases.

Competencies:

• Strong execution focus with ownership of deliverables.
• Problem-solving mindset with attention to detail.
• Ability to collaborate effectively with cross-functional teams.
• Adaptability to evolving technologies in the API and data engineering space.
• Continuous learning and improvement mindset.

AI Skills:

• Understanding of how AI tools can enhance API development and data engineering workflows.
• Regular usage of GenAI tools (Copilot / ChatGPT / other LLMs) for writing and optimising API code, SQL queries, and data transformations.
• Ability to create effective prompts for generating APIs, debugging Python code, and automating data workflows.
• Use of AI tools to accelerate development (API documentation, test case generation, data validation).
• Exposure to AI-enabled features in Databricks (e. g., Genie) for faster development and data exploration.
• Applying AI for anomaly detection, data quality checks, and pipeline monitoring.
• Understanding the role of clean and structured data in enabling AI-driven systems.