Site Reliability Engineer (AI Factory)

Description

The Site Reliability Engineer is responsible for engineering reliability, scalability, automation and operational resilience across the AI Factory. This is a hands-on role for an engineer who can work across Linux systems, GPU platforms, storage, Kubernetes, infrastructure APIs and data centre networking. The SRE works closely with AI Cloud Operations Engineers, AI Engineering teams and vendors to ensure that performance does not constrain AI workloads.

The role must maintain a deliberate balance between operational response and engineering work. Repetitive operational activity should be documented, measured and progressively reduced through automation or platform improvement.

KEY RESPONSIBILITIES

Reliability Engineering

Engineer reliability across AI Factory services, including GPUaaS, BMaaS, AI training and other platform services.

Identify systemic weaknesses that cause outages, workload instability, performance degradation or recurring incidents.

Maintain end-to-end visibility across facilities, compute, GPU, storage, network, Kubernetes and orchestration layers.

Assess reliability risks introduced by platform changes, new services, configuration changes and capacity growth.

Compute and GPU Platform Engineering

Administer and troubleshoot compute and network infrastructure in a large-scale, high-availability production environment.

Investigate operating-system, kernel, process, memory, SXM, disk, firmware and hardware-related faults.

Monitor and troubleshoot NVIDIA GPU health, drivers, CUDA, firmware, temperature, throttling, memory and interconnect conditions.

Maintain configuration and version consistency across cluster nodes, including approved operating-system, firmware and driver baselines.

Execute or support hardware acceptance, cluster validation and performance testing for new and replacement infrastructure.

Work with vendors and engineering teams to isolate hardware, firmware, software and integration failures.

Storage and Data-Path Reliability

Support high-performance storage used by AI workloads, including WEKA or comparable distributed storage platforms.

Troubleshoot local NVMe, pooled disk, shared storage, capacity, latency, throughput and data-path issues.

Analyse data-ingress and data-egress bottlenecks that affect training or inferencing workloads.

Create repeatable health checks and validation for storage availability, performance and integration dependencies.

DevOps, Automation and API Integration

Identify repetitive, error-prone and high-effort operational activities and replace them with controlled automation.

Use Ansible for configuration management, provisioning, validation, patching and remediation workflows.

Develop maintainable Python scripts for operational tooling, health checks, data collection and recovery actions.

Use APIs to automate workflows and integrate infrastructure, monitoring, ticketing, hardware-management and orchestration platforms.

Apply software engineering practices to infrastructure automation, including source control, peer review, testing, structured logging, error handling, documentation and controlled release through CI/CD pipelines.

Observability and Monitoring Engineering

Design and maintain Grafana dashboards for availability, capacity, utilisation, GPU health, storage performance and network behaviour.

Use Prometheus and PromQL to develop metrics, recording rules, alerts, capacity views and reliability indicators.

Develop or maintain exporters and metric integrations for infrastructure components where required.

Correlate metrics, logs, events and traces during incident investigation.

Ensure alerts are actionable, severity-aligned and mapped to documented operational responses.

Reduce alert noise and improve signal quality so that emerging issues are identified before customer impact.

Network and Fabric Reliability

Troubleshoot Ethernet and InfiniBand connectivity supporting distributed AI workloads.

Apply solid L2/L3 knowledge to investigate routing, VLAN, bonding, addressing, DNS and firewall issues.

Investigate latency, packet loss, congestion, retransmissions, oversubscription and throughput degradation.

Trace host-to-host communication across server interfaces, fabrics, switching and cluster communication layers.

Automate network data collection, configuration validation and operational checks where appropriate.

Work with specialist engineers and vendors on advanced fabric, routing, switching and firewall incidents.

Incident Engineering and Escalation Support

Provide engineering-level escalation support for complex production incidents while retaining primary accountability for eliminating recurring failure patterns through automation, observability and platform improvement.

Participate in an on-call rotation for critical escalations and maintain clear technical communication during high-severity incidents.

Gather evidence from systems, metrics, logs, hardware, storage and network components, then test evidence-based hypotheses.

Drive or contribute to post-incident reviews focused on root cause, learning, prevention and measurable follow-through.

Convert recurring failure patterns into automation, monitoring improvements, runbooks, backlog items or design changes.

Capacity, Change and Operational Security

Contribute to capacity modelling, scaling strategies and performance baselines across compute, GPU, storage and network resources.

Plan and execute controlled firmware, driver, operating-system and platform maintenance with appropriate validation and rollback controls.

Coordinate changes to minimise risk to active customer workloads and maintain formal change records.

Apply infrastructure security principles including least privilege, segmentation, secure multi-tenancy, IAM and RBAC.

Collaborate with Security teams on patching, vulnerability remediation and response to security-related events.

Work with facility providers to manage dependencies on power, cooling, physical access and cabling.

Requirements

ROLE REQUIREMENTS

Significant hands-on experience in Site Reliability Engineering, DevOps, platform engineering, production engineering, Linux systems engineering or large-scale infrastructure operations.

Advanced Linux systems administration and troubleshooting in production environments.

Practical Python scripting and Ansible automation experience, including the ability to create, maintain and troubleshoot production automation.

Hands-on experience with Grafana, Prometheus, metrics, alerting, logging and operational dashboards.

Experience operating distributed systems, high-availability platforms or complex infrastructure at scale.

Familiarity with Kubernetes, containers, orchestration concepts and Git-based engineering workflows.

Solid understanding of storage concepts, including NVMe, SAN, NAS or distributed storage platforms.

Solid understanding of Ethernet, InfiniBand fundamentals and L2/L3 troubleshooting.

Experience investigating complex incidents across multiple technology layers and reducing recurrence through engineering.

Hands-on experience integrating platforms through REST APIs, including authentication, JSON payloads, pagination, response validation, error handling and secure credential management.

Experience in Site Reliability Engineering, Platform Engineering or Production Engineering roles.

Background in large-scale platforms, cloud providers, HPC environments, data centre environments or complex infrastructure.

Demonstrated experience reducing incidents through engineering rather than manual processes.

Demonstrated ability to use software engineering and automation to improve infrastructure operations.

Ability to troubleshoot confidently across Linux, compute, GPU, storage, orchestration and network layers.

Ability to work effectively with internal engineering teams, operational support tiers, facility providers and vendors.