Senior Data Engineer

Job Purpose - We are seeking a highly skilled Python & Cloud Platform Engineer to support, modernize, and migrate Python-based applications currently running on Linux servers with load balancers to containerized environments on Kubernetes, specifically Azure Kubernetes Service (AKS) or Amazon Elastic Kubernetes Service (EKS). Log extraction and RCA debug using Azure Data Explorer.

Key Roles -

• Support and maintain Python applications running on Linux environments

• Manage load-balanced application architectures to ensure high availability and performance

• Analyse existing Linux/VM-based workloads and plan migration to Kubernetes

• Containerize Python applications using Docker

• Deploy, manage, and scale applications on Azure Kubernetes Service (AKS) or Amazon Elastic Kubernetes Service (EKS)

• Design and configure Kubernetes resources (Deployments, Services, Ingress, ConfigMaps, Secrets)

• Perform application and infrastructure migration from on-prem / VM setups to AKS and EKS

• Implement CI/CD pipelines for automated build and Kubernetes deployments

• Monitor and troubleshoot application, container, and cluster-level issues

• Optimize performance, scalability, and reliability in cloud-native environments

• Implement security best practices for Kubernetes and cloud platformsDuties and Responsibilities

Key Responsibilities -

• Translate business requirements into technical solutions in collaboration with the PMO team.

• Own end-to-end delivery of data projects, ensuring on-time execution and adherence to quality standards.

• Design technical architecture and guide development efforts for enhancements and new projects.

• Develop and maintain robust ETL pipelines and data integration modules across systems.

• Ensure high data quality, platform stability, and resolution of critical process issues.

• Monitor and resolve performance bottlenecks in data workflows and programs.

• Establish best practices, standard operating procedures, and drive their implementation across teams.

• Act as a liaison with business users and product managers to support daily data needs and strategic initiatives.

• Coordinate with internal and external development teams to troubleshoot and resolve issues efficiently.

• Manage workload through effective planning, prioritization, and progress tracking.Key Decisions / Dimensions

Key Decisions / Dimensions::

• Making critical decisions during production issues, including root cause analysis, quick fixes, and long-term resolutions.

• Prioritizing and escalating support tasks effectively to minimize downtime and business impact.

• Driving decisions around technical design, architecture, and optimization to ensure performance, scalability, and maintainability of solutions.Major Challenges

Major Challenges

• Balancing development timelines with on-time delivery in a dynamic and evolving technical environment.

• Coordinating with multiple internal and external stakeholders to align priorities, resolve dependencies, and ensure smooth execution.

• Ensuring code quality, data integrity, and performance while scaling data solutions across diverse systems and platforms. Required Qualifications and Experience

Required Skills & Experience -

• Strong experience with Python development (FastAPI, Pyspark) - Must Have

• Strong Linux system administration skills. - Must Have

• Hands-on experience with Docker and Kubernetes.

• Proven experience with AKS and EKS.

• Experience with load balancers and traffic routing.

• Understanding of microservices architecture.

• Experience with Git, CI/CD pipelines, and automation tools.

• Knowledge of cloud networking concepts.

• Azure Databricks - PySpark, SQL

• Azure Data Explorer as Time Series Database with Kusto Query Language (KQL) As Programming Language - Good to Have

Soft Skills:

• Strong problem-solving and analytical thinking

• Good communication and stakeholder management

• Ability to work independently and as part of a team in a fast-paced environment

• Strong attention to detail and data integrity