{"id":199700,"url":"https://alion.io/job/firmus-technologies-senior-ai-infrastructure-engineer-observability","title":"Senior AI Infrastructure Engineer, Observability","company":{"id":63306,"name":"Firmus Technologies","domain":"firmus.ai","url":"https://alion.io/company/firmus-technologies","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"A","score":87,"open_postings":3,"ghost_share":0,"stale_share":0.333,"repost_share":0,"time_to_fill_p50_days":81,"computed_at":"2026-10-26T05:57:25Z"}},"role":"DevOps","role_family":"DevOps","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"explicit","locations":["Singapore"],"countries":["SG"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":59000,"max_usd":105000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":78},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Grafana","optional":false},{"name":"HPC","optional":false},{"name":"Linux","optional":false},{"name":"NCCL","optional":false},{"name":"NVLink","optional":false},{"name":"Python","optional":false},{"name":"Platform Engineering","optional":true}],"status":"live","first_seen_at":"2026-09-04T05:23:56Z","employer_posted_date":"2026-09-22","last_verified_at":"2026-10-26T18:15:00Z","board_verified":true,"closed_at":null,"days_open":52,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":52},"description":"Firmus Technologies\nFirmus Technologies is a global leaderpioneering the development and operation of efficient AI infrastructure across Asia Pacific. \nFounded in Australia in 2019, our mission is to create the most efficient AI infrastructure by combining cutting-edge technology with a steadfast commitment to sustainability. \nAt Firmus, we are unique in our approach. We design, build, and operatea new class of digital infrastructure - the AI Factory. Through our model-to-grid technology approach, we have pushed the boundaries of multi-generational liquid cooling systems, energy management, AI software orchestration, and construction. For our customers, this approach allows us to make every watt count and deliver low-cost AI tokens globally. \nFirmus AI Cloud\nOur large-scale GPU cloud platform, Firmus AI Cloud, is purpose-built to deliver energy-efficient AI compute at scale to customers. \nIt empowers developers, enterprises, educational institutions, and government users to train and deploy AI models with unmatched efficiency and cost savings. With an ever-growing suite of services and applications, we are committed to delivering a cloud experience that is market-leading, proprietary, and built to scale. \nROLE SUMMARY \nFirmus Technologies is seeking a Senior AI Infrastructure Engineer, Observability, to join our Engineering and Technology team. You will establishhow we measure, validateand communicate the health of GPU infrastructure used for customer and internal workloads. You will define trusted health signals and service-readiness criteria, and turn them into reusable dashboards, alerts, queries, diagnostic checksand operational guidance. Your work will help commissioning, infrastructure and operations teams bring capacity online safely, identifydegradation early and recover from failures quickly. You will also make knowledge self-service by publishing clear reference implementations, runbooksand AI-ready operational knowledge that other teams can use and extend. \nKEYRESPONSIBILITIES \nThe Health Standard Define GPU and host health criteria for customer and internal workloads, and the service-readiness gates that repair and capacity workflows depend on. Cover what stops or corrupts AI jobs: GPUs falling off the bus, XID events, ECC and memory faults, NVLink/NVSwitch degradation, thermal and power capping, NCCL and collective failures, silent data corruption, and stragglers running below fleet baseline. \n\nReference Implementations Publish golden dashboards, alerts, PromQL/LogQL queries and health checks that other teams adopt and extend. Your output is the standard and the examples. The alerts must be specific, low-noise, and with a clear next action. \n\nDiagnostics with Real Pass/Fail Criteria DCGM checks, NCCL and bandwidth tests, stress and burn-in, and validation jobs that confirm a server matches expected performance. Others must be able to run them without you. \n\nFault Isolation Separate a bad GPU from a cooling, host, network or power-limit problem using host and BMC/management-interface telemetry together, including when the same signature appears across many servers. A clean management view means nothing if the host is throwing faults. \n\nDetection of the Failures that Don't Crash \nRising correctable ECC counts, NVLink retries, thermal slowdown, XID patterns, wrong results with no error. Keep the knowledge current: what each signal means, what to do next on the machine, and where the operation team must make the final decision. \nPartnership and EscalationWork with commissioning on bring-up and acceptance, operations on break-fix, infrastructure on what healthy hardware looks like, and the telemetry owners on making your signals production-grade. Run technical sessions with customer teams on the telemetry they need. Join GPU and host incidents, including debugging on the server and convert every finding into a reusable check, alert or runbook. Give engineering leadership a straight read on fleet health and risk. \n\nSKILLS AND EXPERIENCE \nBachelor's degree in computer science or a related technical field, or equivalent practical experience. \n7+ years in GPU, HPC, AI infrastructure or closely related large-scale systems engineering environments, including ownership of health monitoring or diagnostics used by customers or internal teams. \nExperience with production GPU fault diagnosis. You'vefound genuine faults through XID events, ECC/memory errors, NVLinkissues, power or thermal limits and caught at least one before the job died. \nStrong Linux and server fundamentals. You can debug on the machine and reason across GPU, CPU, memory, PCIe, power and cooling. You automate in Python or similar. \nHands-on with GPU diagnostics and validation such as DCGM, NCCL/collective tests, stress testing and you've turned them into checks other people run. \nExperience analysing infrastructure telemetry, building trusted dashboards and alerts. Savvy with PromQL, LogQL, Grafana or comparable tooling. Able to follow an unexpected signal until thereis a cause and can turn that into an alert or check others will trust. \nUses AI tools as a normal part of analysis and build work. Can structure health knowledge and build AI skills so operation team withAI assistants can use iteffectively to recover from incidents. \nWilling to take part in the incident-response on-call rotation. \nWilling to travel overseas occasionally when the role requires it. \nClear and effective written and verbal communication in English. \nHighly Desirable Experiences\nProduction experience on large GPU systems with tight GPU-to-GPU interconnect. \nMultiple sites or GPU generations, especially preparing health monitoring before a new GPU class entered production. \nBackground at an AI cloud, GPU cloud or HPC centre running customer workloads. \nExperience working with data or ML engineers on GPU/host failure prediction from health signals. \nBuilt a structured operational knowledge so AI assistants can use it effectively during incident recovery. \nLocation & Reporting\nLocation: Singapore\nReport to: Senior Manager, Platform Engineering\nEmployment Basis\nFull-time\nDiversity\nAt Firmus, we are committed to building a diverse and inclusive workplace. We encourage applications from candidates of all backgrounds who are passionate about creating a more sustainable future through innovative engineering solutions.\nJoin us in our mission to revolutionize the AI industry through sustainable practices and cutting-edge engineering. Apply now to be part of shaping the future of sustainable AI infrastructure.","description_format":"text","description_chars":6563,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":true},"security_clearance":false,"languages":[{"language":"English","level":"Upper-Intermediate (B2)","optional":false}]},"benefits":[],"hiring_locations":[{"name":"Singapore","iso":"SG","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Data Centers & Colocation","Cloud Platforms (IaaS & PaaS)","AI Compute & Inference"],"lifecycle":[{"event":"open","at":"2026-09-04T07:54:25Z"}],"visa":[],"liveness":{"score":75,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.838,"p_room":0.9,"age_days":52,"expected_fill_days":81,"reasons":["conf:1","velocity","win:mid"],"computed_at":"2026-10-26T05:57:25Z"},"pay":null,"html_url":"https://alion.io/job/firmus-technologies-senior-ai-infrastructure-engineer-observability","json_url":"https://alion.io/job/firmus-technologies-senior-ai-infrastructure-engineer-observability.json","meta":{"generated_at":"2026-10-26T21:43:10Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":29294,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-27T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":63306},"rest":"https://alion.io/mcp/rest/get_company?id=63306"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Ffirmus-technologies-senior-ai-infrastructure-engineer-observability"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Ffirmus-technologies-senior-ai-infrastructure-engineer-observability"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Ffirmus-technologies-senior-ai-infrastructure-engineer-observability"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/firmus-technologies-senior-ai-infrastructure-engineer-observability\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Ffirmus-technologies-senior-ai-infrastructure-engineer-observability"}]}