Platform Reliability Engineer

Role Summary

Firmus runs large-scale,state-of-the-art AI infrastructure built on the latest generation of GPUrack-scale systems and operated as one estate to power the next generation ofAI innovation. The Platform Reliability Engineer is part of the team that operatesthe platform services Firmus AI Cloud runs on in production: the GPU computefleet, exabyte-scale storage, the multi-tenant control and management plane,and the virtualisation and shared services that host it. Together, these arethe foundation every customer workload depends on.

This is a hands-on site reliabilityengineering role that works across the stack, from Linux and Kubernetes tostorage, virtualisation and the compute fleet. The team runs each service to adeclared service level, measures reliability against it, and treats operationsas a software problem, engineering repetitive manual work out of the systemrather than absorbing it. The role picks up incidents escalated from firstresponse, owns each one through to resolution, and works across teams andvendors to restore service, bringing in specialist engineers when a fault needsdeeper investigation. Automation is a first-class part of the role: faults arediagnosed using the observability stack, post-incident reviews feed fixes backinto the platform, and every resolved fault is a candidate for a runbook or anautomated fix.

Key Responsibilities

·      Triage, diagnose and resolve incidents acrossthe stack, including Kubernetes clusters and workloads, storage,virtualisation, network connectivity and the underlying Linux hosts, owningeach incident from escalation through to resolution.

·      Operate the multi-tenant control and managementplane that Firmus' AI and infrastructure services depend on, including thetenancy, quota and access controls that support separation between tenantworkloads and data.

·      Operate the virtualisation platform that hoststhe management plane, and the shared services that run on it, to their declaredservice levels.

·      Operate Firmus' exabyte-scale distributed andhigh-performance filesystems and S3-compatible object storage to their declaredservice levels, handling capacity, performance and availability incidents andescalating to specialist engineers and vendors (for example VAST or WEKA) withclear evidence when further investigation is needed.

·      Operate the GPU compute fleet in production:node health and readiness, GPU and node fault detection and handling, firmwareand driver currency to the supported baselines, remediation and return toservice, and support for cluster provisioning and validation activities.

·      Reduce toil by converting repeat fixes intorunbooks and automated remediation delivered as controlled code, contributingto the team's automation and remediation tooling.

·      Diagnose performance problems across the datapath between compute and storage, gathering the evidence needed to resolve themand working with network and storage specialists where required.

·      Deliver change as code through the agreed changeprocess, with peer review, testing, staged rollout where applicable, and arollback plan.

·      Use the observability stack (for exampleGrafana, Prometheus, Loki and Tempo) to track service level indicators againsttheir objectives, query metrics, logs and traces, correlate signals acrossservices and diagnose faults, and improve dashboards and alerting so that eachalert is actionable and tied to a runbook.

·      Execute the patching cycle and urgentvulnerability remediation across the services the team operates.

·      Manage vendor support cases across the hardwareand software portfolio, raising and tracking cases under the relevant supportagreements, supplying logs and diagnostic evidence, and following cases throughto agreed response and resolution times.

·      Take part in technical recovery during majorincidents and in blameless post-incident reviews, feed recurring issues back tothe engineering teams as clear requirements, share the follow-the-sun on-callroster, and keep incident tickets, operational procedures and runbooks accurateand current so that others can execute them successfully.

Skills & Experience

Required Skills

·       Solidexperience operating production infrastructure in a 24/7 environment using sitereliability engineering practices, including service level objectives, toilreduction, owning incidents from escalation to resolution, on-callparticipation, post-incident review and writing runbooks that others canexecute successfully.

·       Strongworking knowledge of Linux systems administration and troubleshooting,including storage, host networking, processes, memory and I/O, and a systematicapproach to isolating faults.

·       Workingexperience with scale-out, parallel or enterprise storage (for example VAST,WEKA, Ceph, Lustre, GPFS or NetApp), including diagnosing performance andcapacity problems using evidence rather than assumption.

·       Familiaritywith a virtualisation platform (for example Proxmox, VMware or KVM).

·       Workingexperience using observability tooling to diagnose faults (for example Grafana,Prometheus, Loki, Tempo or Elasticsearch), including writing queries acrossmetrics and logs (such as PromQL and LogQL) and correlating signals acrossservices.

·       Workingexperience with infrastructure-as-code and GitOps practices (for exampleOpenTofu or Terraform, Ansible, Argo CD), and writing scripts or small toolsfor operational automation in a language such as Python or Go.

·       Familiaritywith server hardware operation and support, including working with hardware andsoftware vendors to raise support cases and supply diagnostic evidence.

·       Familiaritywith cyber security principles and how they apply to operating productioninfrastructure.

·       Clearcommunication skills, with the ability to write incident updates, ticket notesand escalations that others can act on, and a working style suited to a teamthat shares accountability for a broad portfolio across regions.

PreferredExperience

·       Experiencein infrastructure operations for GPU or HPC environments and the demands oflarge-scale distributed training and inference, including GPU health and faultsignals (for example DCGM health checks, NCCL tests and XID errors).

·       Experiencein multi-tenant service provider, cloud or colocation environments withcontractual response commitments and customer-facing reporting.

·       Workingknowledge of networking fundamentals, including TCP/IP, DNS, routing, VLANs,load balancing and firewalls, and the ability to trace a connectivity faultacross hosts, switches and services.

·       Knowledgeof high-performance fabrics and data-path technologies, such as InfiniBand,RoCE, RDMA and GPUDirect Storage, and DPU or SmartNIC-based networking.

·       Familiaritywith bare-metal lifecycle and provisioning automation (for example NVIDIA InfraController, MAAS, Ironic or Metal3), and standard operating environments withknown-good-state enforcement.

·       Hands-onexperience operating and troubleshooting Kubernetes, including workloads,networking, storage and cluster health, and multi-tenant and GPU-enabledKubernetes patterns.

·       Experiencewith ITSM and incident management tooling (for example Jira Service Management,ServiceNow or PagerDuty).

·       Experienceworking under ISO 27001 or SOC 2 controls.

·       Relevantcertification, such as Certified Kubernetes Administrator (CKA), a Linuxcertification (for example RHCSA or LFCS), or a vendor certification.

·       ABachelor's degree in computer science, engineering or a related discipline, oran equivalent combination of relevant experience and training.

Skills

Report Writing, Performance Testing, Engineering Management, Technical Skills, Code Remediation, Fault Isolation, GPU, VMware, resolved escalations, Log Analysis, Distributed File Systems, Virtualization Platform, implement process changes, Developing Skills, Quotas, Storage Area Networks