{"id":2311825,"url":"https://alion.io/job/firmus-technologies-principal-engineer-observability","title":"Principal Engineer, Observability","company":{"id":63306,"name":"Firmus Technologies","domain":"firmus.ai","url":"https://alion.io/company/firmus-technologies","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":null},"role":"Backend","role_family":"Backend","seniority":"lead","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"explicit","locations":["Singapore"],"countries":["SG"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":65000,"max_usd":124000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":25},"experience_years_min":10,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Apache Kafka","optional":false},{"name":"Digital Twin","optional":false},{"name":"ISO 27001","optional":false},{"name":"Kubernetes","optional":false},{"name":"Mimir","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Platform Engineering","optional":false},{"name":"Prometheus","optional":false},{"name":"Python","optional":false},{"name":"Redpanda","optional":false},{"name":"SOC 2","optional":false},{"name":"Thanos","optional":false},{"name":"VictoriaMetrics","optional":false},{"name":"Grafana","optional":true},{"name":"Least Privilege","optional":true},{"name":"OPC UA","optional":true}],"status":"live","first_seen_at":"2026-10-11T23:12:13Z","employer_posted_date":"2026-10-11","last_verified_at":"2026-10-13T03:50:49Z","board_verified":true,"closed_at":null,"days_open":1,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":1},"description":"Firmus Technologies\nFirmus Technologies is a global leaderpioneering the development and operation of efficient AI infrastructure across Asia Pacific. \nFounded in Australia in 2019, our mission is to create the most efficient AI infrastructure by combining cutting-edge technology with a steadfast commitment to sustainability. \nAt Firmus, we are unique in our approach. We design, build, and operatea new class of digital infrastructure - the AI Factory. Through our model-to-grid technology approach, we have pushed the boundaries of multi-generational liquid cooling systems, energy management, AI software orchestration, and construction. For our customers, this approach allows us to make every watt count and deliver low-cost AI tokens globally. \nFirmus AI Cloud\nOur large-scale GPU cloud platform, Firmus AI Cloud, is purpose-built to deliver energy-efficient AI compute at scale to customers. \nIt empowers developers, enterprises, educational institutions, and government users to train and deploy AI models with unmatched efficiency and cost savings. With an ever-growing suite of services and applications, we are committed to delivering a cloud experience that is market-leading, proprietary, and built to scale. \nROLE SUMMARY \nFirmus Technologies is seeking a Principal Engineer, Observability to own how Firmus operations teams and customers see the AI factory as one system, from site power and cooling to the GPU and the workload running on it.You will own the telemetry standard every team emits to, the platform that collects, stores, and queries it, and the applications that operations teams and customers use. Every team should be able to build its own dashboards and alerts without waiting on yours. Every new site adds millions of samples per second, and the platform must keep up as Firmus grows to hundreds of thousands of GPUs in gigawatt-scale AI factories, without losing critical data or slowing queries.\nKEYRESPONSIBILITIES \nArchitecture and decisions \nOwn the architecture of the observability platform across every data centre and cloud region, from collection and transport to storage, query, alerting, and the applications on top. Set a target architecture two to three years ahead and plan the steps from today's system to it. \nDesign the platform to grow tenfold without a redesign, through sharding, retention tiers, cardinality budgets, and capacity plans. Prove the headroom with load tests before each new site goes live. \nDesign around the same repeatable unit as the estate. Each site keeps observing itself when its link to the rest of Firmus is down, one global view spans every site, and new sites come on through the site deployment kit with no local fork. \nOwn the telemetry standard: what each layer emits and how signals are named, labelled, and versioned, enforced by checks in every team's build pipeline. \nLead design reviews for significant changes across the teams, sign off architecture decision records, and settle technical disputes escalated by engineers. Decide at pace with incomplete information, write down the assumptions, and revisit a decision when the evidence changes. \nBuild and evolve \nWrite production code for the hardest parts, including collection at fleet scale, the telemetry pipeline, query performance, and the observability API and applications. \nMake observability self-service for every team. Teams onboard a service and build their own dashboards, alerts, and SLOs as code from templates, within the telemetry and alerting standards and their own budgets for cardinality and retention, without filing a ticket. \nSet delivery guarantees for each class of telemetry. Critical operational and customer-billable signals are never lost, and any loss elsewhere is measured and recoverable. \nBuild and own the live topology model (digital twin) of every site, hall, rack, tray, GPU, network link, and cooling loop. Keep it in step with the inventory source of truth and map every signal to it. \nCover all layer customers depend on, from GPU and interconnect health and network fabric events to storage performance per job and Kubernetes control-plane metrics. Run passive and active health checks that signal failing GPUs, links, and nodes for draining before customers notice. \nBring facilities telemetry into the same model as compute. Operations teams see power, cooling, and leak events immediately, together with the workloads they affect. \nServe telemetry to internal consumers, including the data platform and the intelligence platform, through stable, versioned interfaces with agreed freshness and completeness, and make it usable by AI agents for diagnosis, with scoped access and an audit trail. \nDeliver telemetry to customers and partner platforms on open standards such as OpenTelemetry, within seconds of collection, with ready-made dashboards, query access, alerts to their own channels, and forwarding to their own tools. Each customer sees only their own tenancy, enforced at every layer.\nReliability, security, and cost \nSet SLOs and error budgets for observability and use the budget to decide when reliability work comes first. \nLead the technical response to major incidents in observability and make sure every post-mortem ends with a fix that stops it happening again. \nSet the alerting standard. Every alert is actionable, reaches the right responder, and links to a runbook. Prove detection works by injecting faults. \nMake service-level measurement reproducible. Firmus and a customer should get the same availability from the same data. \nDesign the telemetry stores to upgrade in rolling waves with no gap in data, and keep dependencies and images patched within policy. \nKeep Firmus' SOC 2 Type 2 and ISO 27001 controls passing at every new site. \nOwn cost at the design level, including retention, cardinality, and data volume. \nTechnical leadership \nGrow engineers through mentorship, design review, and code review. Hiring and performance review sit with the Engineering Manager. \nSet how the team uses AI to build software effectively and securely, with review gates and automated architecture checks that keep AI-generated code within the agreed design. \nOwn the technical interviews and assessments for engineers and hold every candidate to that bar. \nPartnership and communication \nShape the observability roadmap with operations, infrastructure, facilities, and product, and have telemetry in place before each new site or product launches. \nAgree ownership in writing at each boundary, including which team emits each signal, which team consumes it, and which team acts on it, and settle disagreements directly. \nWork with the operations centre and the intelligence platform team to make first-level incident response and root-cause analysis faster and turn the problems they hit into product improvements. \nRepresent observability in customer briefings on telemetry, health, and how service levels are measured. \nSKILLS AND EXPERIENCE \nBachelor's degree in computer science or a related technical field, or equivalent practical experience. \n10+ years in software, infrastructure, or platform engineering, including at least 3 years at principal level or equivalent, setting the technical direction that multiple teams delivered against. You are still hands-on and intend to stay that way. \nHas designed a metrics, logs, and traces platform handling millions of samples per second and run it in-house on Prometheus-compatible stores such as VictoriaMetrics, Thanos, or Mimir, with high cardinality, long retention, and a global query view across more than one site. Scaled it several times over as load grew, without a redesign. \nHas run a streaming ingestion layer for telemetry (such as Kafka or Redpanda, with Vector or the OpenTelemetry Collector) and handled consumer lag, backpressure, and broker failures without losing critical data. \nHas defined a telemetry schema on open standards such as OpenTelemetry or OpenMetrics that other teams relied on as a contract and given those teams self-service to build their own dashboards and alerts on it. \nHas collected telemetry from hardware or networks, not only applications, through at least one of Redfish, gNMI with OpenConfig models, or DCGM. \nProduction Kubernetes as the runtime for observability: running and upgrading stateful, high-throughput services such as Kafka or a metrics store with no data loss. \nStrong in Go or Python, with production services, APIs, or applications that other teams or customers use. \nHas built security into production services: access control, secrets handling, and patching, under SOC 2 Type 2, ISO 27001, or an equivalent control set. \nHas set an alerting standard across teams that cut noise and measurably reduced time to detect and time to restore, backed by SLOs and incident reviews that changed the system. \nClear written and verbal communication in English, able to explain problems, decisions, and trade-offs to engineers, executives, and customers. \nWilling to join the on-call rotation for the services you own and to travel overseas occasionally. \nBonus Points \nBackground at an AI cloud or high-performance compute provider, ideally with GPU health checks, burn-in testing, or failure prediction across a fleet. \nIntegrated facilities or operational technology telemetry, such as OPC UA, Modbus, or data from SCADA and building management systems. \nBuilt a digital twin or live topology model of a physical estate from an inventory source of truth. \nDelivered telemetry to external customers with tenant isolation and access controls. \nServed telemetry to a data platform or to analytics and machine-learning pipelines used for incident response or root-cause analysis. \nWrote or extended Kubernetes operators for stateful services. \nLed a team's use of AI-assisted and agentic engineering, measured by delivery outcomes and token cost, and kept it secure with sandboxed agents, least-privilege access, and review gates. \nCertifications such as CKA or CKAD, Prometheus or OpenTelemetry Certified Associate, or Confluent Certified Administrator for Apache Kafka. \nLocation & Reporting\nLocation: Singapore\nHybrid: 4 days in office, 1 day work from home\nReport to: Head of Platform Engineering\nEmployment Basis\nFull-time\nDiversity\nAt Firmus, we are committed to building a diverse and inclusive workplace. We encourage applications from candidates of all backgrounds who are passionate about creating a more sustainable future through innovative engineering solutions.\nJoin us in our mission to revolutionize the AI industry through sustainable practices and cutting-edge engineering. Apply now to be part of shaping the future of sustainable AI infrastructure.","description_format":"text","description_chars":10674,"description_truncated":false,"requirements":{"experience_years_min":10,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[{"language":"English","level":"Upper-Intermediate (B2)","optional":false}]},"benefits":[],"hiring_locations":[{"name":"Singapore","iso":"SG","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Data Centers & Colocation","Cloud Platforms (IaaS & PaaS)","AI Compute & Inference"],"lifecycle":[{"event":"open","at":"2026-10-12T01:40:43Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":91,"reasons":["conf:2","win:early"],"computed_at":"2026-10-12T06:55:09Z"},"pay":null,"html_url":"https://alion.io/job/firmus-technologies-principal-engineer-observability","json_url":"https://alion.io/job/firmus-technologies-principal-engineer-observability.json","meta":{"generated_at":"2026-10-13T04:18:02Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":6717,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-14T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":63306},"rest":"https://alion.io/mcp/rest/get_company?id=63306"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Ffirmus-technologies-principal-engineer-observability"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Ffirmus-technologies-principal-engineer-observability"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Ffirmus-technologies-principal-engineer-observability"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/firmus-technologies-principal-engineer-observability\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Ffirmus-technologies-principal-engineer-observability"}]}