Sr. Site Reliability Engineer

Responsibilities:

• Implement and maintain observability, monitoring, and alerting systems for AI platforms and mission-critical backend services serving the T-Mobile Supply Chain domain.
• Collaborate in designing telemetry pipelines, logging infrastructure, and metrics dashboards using tools such as Splunk, Prometheus, Grafana, and OpenTelemetry
• Contribute to the development of SLOs, SLIs, and real-time health indicators across platform services and APIs.
• Participate in on-call rotations and lead resolution of high-impact incidents, including root cause analysis and postmortem reporting.
• Work closely with platform engineering teams to enforce governance, compliance, and security standards in production environments.
• Improve deployment pipelines, CI/CD workflows, and infrastructure automation (e. g., GitLab).
• Tune and scale infrastructure components such as Kafka, HAProxy, RMQ, databases, and distributed APIs.
• Support capacity planning, cost analysis, and system tuning to optimise platform performance.
• Advocate for automation-first operations, reducing toil through scripting and system reliability tooling.
• Contribute to documentation, runbooks, and knowledge sharing across SRE and engineering teams.
• Mentor junior engineers and participate in the culture of technical rigour and continuous improvement.

Requirements:

• Bachelor's degree in computer science, Engineering, or a related field (master's preferred).
• 7+ years of experience in SRE, DevOps, or operations engineering in cloud-based environments.
• Hands-on experience with monitoring, alerting, and incident response in distributed systems.
• Strong coding/scripting ability in Python, Java, or shell scripting languages like Bash or PowerShell.
• Proficiency in CI/CD pipelines and GitLab workflows.
• Strong working knowledge of SQL and NoSQL databases, including Oracle DB and MongoDB.
• Working knowledge of AI/ML systems, APIs, and modern LLM tooling is a strong plus.
• Familiarity with observability tools such as Splunk, Grafana, and Prometheus.
• Experience with Kubernetes, container orchestration, and hybrid/multi-cloud deployments (Azure, AWS, GCP, OCI).
• Demonstrated ability to work in fast-paced, incident-driven environments with high stakes and uptime requirements.

Engineering an Operations:

• CI/CD for integrations and infrastructure.
• Automated testing (integration/regression).
• Performance & scalability engineering.
• SRE fundamentals (SLIs, SLOs, RCA).
• Platform cost visibility & optimisation.
• Security-first design & compliance readiness.

Preferred Qualifications:

• Experience supporting AI workloads, model inference systems, or LLM-enabled platforms.
• Exposure to AIOps or related ML platform for observability and reliability practices.
• Experience in the highly regulated finance industry with an understanding of compliance and audit controls
• Familiarity with platforms such as OpenAI or AI Gateway patterns.
• Background in building secure, zero-downtime platforms with enterprise-scale SLAs.

Must Have Skills:

• Strong grasp of SRE best practices including SLOs, SLIs, postmortems, and chaos engineering.
• Proficiency in diagnosing system bottlenecks across infrastructure, application, and network layers.
• Experience driving automation across observability, configuration, and deployment domains.
• Strong communicator and collaborator in cross-functional technical teams.
• Curiosity-driven mindset with a passion for learning emerging AI technologies and systems reliability.