{"id":2185069,"url":"https://alion.io/job/liquid-intelligent-technologies-site-reliability-engineer-ai-factory","title":"Site Reliability Engineer (AI Factory)","company":{"id":57361,"name":"Liquid Intelligent Technologies","domain":"liquid.tech","url":"https://alion.io/company/liquid-intelligent-technologies","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"MCI DirectHire","truth_index":null},"role":"DevOps","role_family":"DevOps","seniority":null,"employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["South Africa"],"countries":["ZA"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Ansible","optional":false},{"name":"CI/CD","optional":false},{"name":"Configuration Management","optional":false},{"name":"CUDA","optional":false},{"name":"DNS","optional":false},{"name":"Git","optional":false},{"name":"Grafana","optional":false},{"name":"HPC","optional":false},{"name":"IAM","optional":false},{"name":"InfiniBand","optional":false},{"name":"Kubernetes","optional":false},{"name":"Least Privilege","optional":false},{"name":"Linux","optional":false},{"name":"Platform Engineering","optional":false},{"name":"Prometheus","optional":false},{"name":"Python","optional":false},{"name":"Rest API","optional":false},{"name":"VLAN","optional":false}],"status":"closed","first_seen_at":"2026-10-09T12:00:00Z","employer_posted_date":"2026-10-09","last_verified_at":"2026-10-19T14:27:27Z","board_verified":false,"closed_at":"2026-10-19T14:27:27Z","days_open":10,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":10},"description":"Description\nThe Site Reliability Engineer is responsible for engineering reliability, scalability, automation and operational resilience across the AI Factory. This is a hands-on role for an engineer who can work across Linux systems, GPU platforms, storage, Kubernetes, infrastructure APIs and data centre networking. The SRE works closely with AI Cloud Operations Engineers, AI Engineering teams and vendors to ensure that performance does not constrain AI workloads.\nThe role must maintain a deliberate balance between operational response and engineering work. Repetitive operational activity should be documented, measured and progressively reduced through automation or platform improvement.\nKEY RESPONSIBILITIES\nReliability Engineering\nEngineer reliability across AI Factory services, including GPUaaS, BMaaS, AI training and other platform services.\nIdentify systemic weaknesses that cause outages, workload instability, performance degradation or recurring incidents.\nMaintain end-to-end visibility across facilities, compute, GPU, storage, network, Kubernetes and orchestration layers.\nAssess reliability risks introduced by platform changes, new services, configuration changes and capacity growth.\nCompute and GPU Platform Engineering\nAdminister and troubleshoot compute and network infrastructure in a large-scale, high-availability production environment.\nInvestigate operating-system, kernel, process, memory, SXM, disk, firmware and hardware-related faults.\nMonitor and troubleshoot NVIDIA GPU health, drivers, CUDA, firmware, temperature, throttling, memory and interconnect conditions.\nMaintain configuration and version consistency across cluster nodes, including approved operating-system, firmware and driver baselines.\nExecute or support hardware acceptance, cluster validation and performance testing for new and replacement infrastructure.\nWork with vendors and engineering teams to isolate hardware, firmware, software and integration failures.\nStorage and Data-Path Reliability\nSupport high-performance storage used by AI workloads, including WEKA or comparable distributed storage platforms.\nTroubleshoot local NVMe, pooled disk, shared storage, capacity, latency, throughput and data-path issues.\nAnalyse data-ingress and data-egress bottlenecks that affect training or inferencing workloads.\nCreate repeatable health checks and validation for storage availability, performance and integration dependencies.\nDevOps, Automation and API Integration\nIdentify repetitive, error-prone and high-effort operational activities and replace them with controlled automation.\nUse Ansible for configuration management, provisioning, validation, patching and remediation workflows.\nDevelop maintainable Python scripts for operational tooling, health checks, data collection and recovery actions.\nUse APIs to automate workflows and integrate infrastructure, monitoring, ticketing, hardware-management and orchestration platforms.\nApply software engineering practices to infrastructure automation, including source control, peer review, testing, structured logging, error handling, documentation and controlled release through CI/CD pipelines.\nObservability and Monitoring Engineering\nDesign and maintain Grafana dashboards for availability, capacity, utilisation, GPU health, storage performance and network behaviour.\nUse Prometheus and PromQL to develop metrics, recording rules, alerts, capacity views and reliability indicators.\nDevelop or maintain exporters and metric integrations for infrastructure components where required.\nCorrelate metrics, logs, events and traces during incident investigation.\nEnsure alerts are actionable, severity-aligned and mapped to documented operational responses.\nReduce alert noise and improve signal quality so that emerging issues are identified before customer impact.\nNetwork and Fabric Reliability\nTroubleshoot Ethernet and InfiniBand connectivity supporting distributed AI workloads.\nApply solid L2/L3 knowledge to investigate routing, VLAN, bonding, addressing, DNS and firewall issues.\nInvestigate latency, packet loss, congestion, retransmissions, oversubscription and throughput degradation.\nTrace host-to-host communication across server interfaces, fabrics, switching and cluster communication layers.\nAutomate network data collection, configuration validation and operational checks where appropriate.\nWork with specialist engineers and vendors on advanced fabric, routing, switching and firewall incidents.\nIncident Engineering and Escalation Support\nProvide engineering-level escalation support for complex production incidents while retaining primary accountability for eliminating recurring failure patterns through automation, observability and platform improvement.\nParticipate in an on-call rotation for critical escalations and maintain clear technical communication during high-severity incidents.\nGather evidence from systems, metrics, logs, hardware, storage and network components, then test evidence-based hypotheses.\nDrive or contribute to post-incident reviews focused on root cause, learning, prevention and measurable follow-through.\nConvert recurring failure patterns into automation, monitoring improvements, runbooks, backlog items or design changes.\nCapacity, Change and Operational Security\nContribute to capacity modelling, scaling strategies and performance baselines across compute, GPU, storage and network resources.\nPlan and execute controlled firmware, driver, operating-system and platform maintenance with appropriate validation and rollback controls.\nCoordinate changes to minimise risk to active customer workloads and maintain formal change records.\nApply infrastructure security principles including least privilege, segmentation, secure multi-tenancy, IAM and RBAC.\nCollaborate with Security teams on patching, vulnerability remediation and response to security-related events.\nWork with facility providers to manage dependencies on power, cooling, physical access and cabling.\nRequirements\nROLE REQUIREMENTS\nSignificant hands-on experience in Site Reliability Engineering, DevOps, platform engineering, production engineering, Linux systems engineering or large-scale infrastructure operations.\nAdvanced Linux systems administration and troubleshooting in production environments.\nPractical Python scripting and Ansible automation experience, including the ability to create, maintain and troubleshoot production automation.\nHands-on experience with Grafana, Prometheus, metrics, alerting, logging and operational dashboards.\nExperience operating distributed systems, high-availability platforms or complex infrastructure at scale.\nFamiliarity with Kubernetes, containers, orchestration concepts and Git-based engineering workflows.\nSolid understanding of storage concepts, including NVMe, SAN, NAS or distributed storage platforms.\nSolid understanding of Ethernet, InfiniBand fundamentals and L2/L3 troubleshooting.\nExperience investigating complex incidents across multiple technology layers and reducing recurrence through engineering.\nHands-on experience integrating platforms through REST APIs, including authentication, JSON payloads, pagination, response validation, error handling and secure credential management.\nExperience in Site Reliability Engineering, Platform Engineering or Production Engineering roles.\nBackground in large-scale platforms, cloud providers, HPC environments, data centre environments or complex infrastructure.\nDemonstrated experience reducing incidents through engineering rather than manual processes.\nDemonstrated ability to use software engineering and automation to improve infrastructure operations.\nAbility to troubleshoot confidently across Linux, compute, GPU, storage, orchestration and network layers.\nAbility to work effectively with internal engineering teams, operational support tiers, facility providers and vendors.","description_format":"text","description_chars":7868,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Internet Service Providers","IT Consulting & Digital Transformation"],"lifecycle":[{"event":"open","at":"2026-10-09T15:59:24Z"},{"event":"close","at":"2026-10-19T14:27:27Z"}],"visa":[],"liveness":null,"pay":null,"html_url":"https://alion.io/job/liquid-intelligent-technologies-site-reliability-engineer-ai-factory","json_url":"https://alion.io/job/liquid-intelligent-technologies-site-reliability-engineer-ai-factory.json","meta":{"generated_at":"2026-10-22T04:31:25Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":4930,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-23T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":57361},"rest":"https://alion.io/mcp/rest/get_company?id=57361"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fliquid-intelligent-technologies-site-reliability-engineer-ai-factory"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fliquid-intelligent-technologies-site-reliability-engineer-ai-factory"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fliquid-intelligent-technologies-site-reliability-engineer-ai-factory"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/liquid-intelligent-technologies-site-reliability-engineer-ai-factory\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fliquid-intelligent-technologies-site-reliability-engineer-ai-factory"}]}