{"id":2168987,"url":"https://alion.io/job/bitdeer-technologies-group-sr-inference-service-sre-automation-engineer","title":"Sr. Inference Service SRE & Automation Engineer","company":{"id":174995,"name":"Bitdeer Technologies Group","domain":"bitdeer.com","url":"https://alion.io/company/bitdeer","size_band":"501-1000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Breezy","truth_index":{"grade":"B","score":81,"open_postings":13,"ghost_share":0,"stale_share":0.769,"repost_share":0,"time_to_fill_p50_days":39,"computed_at":"2026-10-25T05:49:26Z"}},"role":"DevOps","role_family":"DevOps","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Singapore"],"countries":["SG"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":58000,"max_usd":104000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":70},"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AIOps","optional":false},{"name":"API Gateway","optional":false},{"name":"ArgoCD","optional":false},{"name":"Bitcoin","optional":false},{"name":"ClickHouse","optional":false},{"name":"DNS","optional":false},{"name":"etcd","optional":false},{"name":"GitOps","optional":false},{"name":"Go","optional":false},{"name":"Grafana","optional":false},{"name":"Helm","optional":false},{"name":"HPC","optional":false},{"name":"Incident Management","optional":false},{"name":"Kubernetes","optional":false},{"name":"KV Cache","optional":false},{"name":"LLM","optional":false},{"name":"NCCL","optional":false},{"name":"NVLink","optional":false},{"name":"PostgreSQL","optional":false},{"name":"Prometheus","optional":false},{"name":"Pyroscope","optional":false},{"name":"Python","optional":false},{"name":"Redis","optional":false},{"name":"Self-Healing","optional":false},{"name":"SGLang","optional":false},{"name":"SLI/SLO/SLA","optional":false},{"name":"TensorRT-LLM","optional":false},{"name":"Terraform","optional":false},{"name":"TPOT","optional":false},{"name":"Triton","optional":false},{"name":"VictoriaMetrics","optional":false},{"name":"vLLM","optional":false},{"name":"TensorRT","optional":true}],"status":"live","first_seen_at":"2026-10-09T08:34:26Z","employer_posted_date":"2026-10-09","last_verified_at":"2026-10-25T06:45:11Z","board_verified":true,"closed_at":null,"days_open":16,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":16},"description":"About Bitdeer:\nBitdeer is a world-leading technology company for Bitcoin mining and AI cloud.\nBitdeer is committed to providing comprehensive Bitcoin mining solutions for its customers. Apart from designing industry-leading ASIC chips and manufacturing mining rigs, the Group handles complex processes involved in computing across the value chain. This includes equipment procurement, transport logistics, datacenter design and construction, equipment management, and network and facility operations. Bitdeer also offers advanced cloud capabilities to customers with a high demand for artificial intelligence.\nHeadquartered in Singapore, Bitdeer operates globally with a diversified 3 GW energy portfolio, and deploys Bitcoin mining and HPC datacenters in the United States, Bhutan, Norway, Canada, Malaysia, and Ethiopia.\nAbout the role\nNeoCloud runs Infermesh, our managed Inference / Model-as-a-Service platform: customers call model endpoints over a public API and pay per token. For a service like this, reliability is the product - a failed request is not an internal incident, it is a customer-visible SLA breach. In this role you own the reliability of the inference serving path end to end: public API gateway, inference router, request scheduler and queue, prefill/decode and KV cache, inference engines, and the GPU model replicas behind them - plus the control plane and shared dependencies they rely on. Your mandate is to aim a 99.95% SLA, by closing the architecture, automation, and operational gaps that stand in the way. You are the SRE who stands between a fleet of H200/B300/GB300/VR NVL72 serving nodes and the customer's experience of a simple, dependable model endpoint.\nWhat you will be responsible for:\nEnd-to-end availability of customer inference endpoints - API availability, request success rate, TTFT/TPOT latency, and the published SLA.\nThe serving path: API gateway, inference router, health-based routing, request scheduler, queueing and admission control, prefill/decode serving, and the KV cache service.\nInference runtime operations: engine health (vLLM / SGLang / TensorRT-LLM class runtimes), model load and readiness validation, model revision rollout and rollback.\nModel replica topology and failure isolation: tensor-parallel group containment, GPU / NVLink / NCCL fault isolation, replica anti-affinity across nodes, racks, and failure domains.\nMulti-cluster serving across MY / SG / JP: cross-cluster traffic steering, tested cluster failover, and regional recovery or evacuation scope.\nGPU serving fleet health (H200 / B300 / GB300 / VR NVL72 bare metal): DCGM / Xid / ECC monitoring, automated drain and cordon, replica replacement with no sustained customer impact.\nThe Kubernetes platform beneath the serving path: HA control plane, etcd, CNI, CoreDNS, public / management / east-west edge tiers, and egress NAT.\nShared control-plane dependencies: PostgreSQL (including Temporal), Redis, Vault secrets and certificates, object storage and model registry, ClickHouse - HA, tested failover, and decoupling live serving from control-plane outages.\nCapacity and saturation engineering: GPU utilization, KV cache occupancy, queue depth and concurrency; N+1 headroom, admission control, load shedding, and graceful degradation instead of collapse.\nThe SLI/SLO/SLA system: define and instrument the SLIs, publish the SLOs, run error budgets and burn-rate alerting, and produce customer-facing SLA reporting.\nObservability stack - VictoriaMetrics, VictoriaLogs, Tempo, Pyroscope, Grafana, ClickHouse - plus external synthetic inference probes that exercise the real customer API path.\nIncident management: severity model, incident commander rotation, primary / secondary / cross-team / vendor escalation, status-page and customer communication, and postmortems with tracked actions.\nChange safety: staged and canary rollout for application, model, and configuration changes; automatic rollback on regression; production change review and release risk classification.\nResilience validation: recurring failover drills, DR tests, game days and fault injection, and backup restore verification - with measured RTO as the evidence.\nReliability ownership\nYou are accountable for the customer's reliability experience - when an endpoint degrades, a replica dies, or a router loses a backend, you own detection, remediation, and the communication loop.\nDrive the platform from to 99.95%, using the SLA gap assessment - architecture, operations, and failure-mode matrices - as your roadmap.\nOwn the failure-mode and resilience matrix: every Critical and High failure mode gets a detection signal, a mitigation path, a target RTO, and a drill that proves it.\nDefine and publish customer-facing SLOs and SLAs, and drive error-budget-based prioritization between feature work and reliability work.\nPartner with support and customer success to close the loop from customer-reported issues to systemic fixes, and build tenant-visible status, usage, and quota observability that reduces support load.\nFeed the AIOps substrat e\nThe remediation actuator and workflow engine land here - you make the serving control plane safe for automated action.\nYour runbooks, CRDs, and workflow definitions are the schema the platform's predictors and remediators write against.\nEvery human intervention you do this quarter becomes an autonomous workflow next quarter - turning customer-impacting incidents into self-healing events.\nWhat success looks like in year 1\nEvery Critical / High failure mode in the resilience matrix has tested detection, an automated or documented mitigation, and a measured RTO.\nCross-cluster inference failover drilled and completing inside the SLA target; no single-cluster or shared-component SPOF left unidentified.\nAutomatic rollback in place for bad application, model, and configuration deployments - a known-bad canary is reverted without human action.\nExternal synthetic inference probing and SLO dashboards live; error-budget burn thresholds trigger documented action.\nMTTD and MTTR for customer-impacting incidents measurably reduced, with the top recurring interventions converted into automated workflows.\nOn-call rotation with no single-person dependency, and P1/P2 runbooks an on-call engineer can execute without the original author.\n\nHow you will stand out:\n5+ years in SRE / production platform operations, including 2+ years on a customer-facing API service operating against a published SLA.\nDeep Kubernetes operations experience: HA control plane, etcd, CNI, service discovery, ingress and edge tiers, and multi-cluster topologies.\nHands-on production experience serving LLM or other model inference - vLLM, SGLang, TensorRT-LLM, Triton, or equivalent: model loading, batching, KV cache behaviour, concurrency limits, TTFT/TPOT latency.\nGPU fleet operations: DCGM / Xid / ECC signal triage, NVLink and NCCL failure diagnosis, node drain and replica recovery.\nTraffic engineering for an availability-critical API: L4/L7 load balancing, gateways, active health checking, DNS/GSLB, and cross-cluster or cross-region failover.\nStrong SRE fundamentals: SLI/SLO/SLA design, error budgets, incident command, postmortems, and capacity planning.\nOperating stateful dependencies in production: PostgreSQL replication and failover, Redis, message queues, object storage, and secrets / certificate lifecycle.\nObservability at scale: Prometheus- or VictoriaMetrics-class TSDB, Grafana, log and trace pipelines, synthetic probing, and alert quality management.\nOverload protection in practice: admission control, concurrency and queue limits, load shedding, and graceful degradation.\nProficiency in Terraform, Helm, and GitOps workflows (ArgoCD/Flux).\nStrong programming skills in Go or Python for automation, controllers, and operator development.\nAIOps aptitude - you view the control plane as an execution surface for automated remediation, not just a scheduler.\nRunbook-as-code mindset - every SRE playbook you write should be executable by the platform.\nComfortable with customer-facing incident communication and SLA reporting.\nWhat you will experience working with us:\nA culture that values authenticity and diversity of thoughts and backgrounds;\nAn inclusive and respectable environment with open workspaces and exciting start-up spirit;\nFast-growing company with the chance to network with industrial pioneers and enthusiasts;\nAbility to contribute directly and make an impact on the future of the digital asset industry;\nInvolvement in new projects, developing processes/systems;\nPersonal accountability, autonomy, fast growth, and learning opportunities;\nAttractive welfare benefits and developmental opportunities such as training and mentoring.\n--------------------------------------------------------------------\nBitdeer is committed to providing equal employment opportunities in accordance with country, state, and local laws. Bitdeer does not discriminate against employees or applicants based on conditions such as race, colour, gender identity and/or expression, sexual orientation, marital and/or parental status, religion, political opinion, nationality, ethnic background or social origin, social status, disability, age, indigenous status, and union.","description_format":"text","description_chars":9218,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Cryptocurrencies","Mining & Staking","AI Compute & Inference"],"lifecycle":[{"event":"open","at":"2026-10-09T11:17:28Z"}],"visa":[],"liveness":{"score":56,"band":"ok","label":"Likely open","p_open":1,"p_active":0.626,"p_room":0.9,"age_days":15,"expected_fill_days":39,"reasons":["conf:2","stale_co","velocity","win:mid","comp:brand"],"computed_at":"2026-10-25T05:49:26Z"},"pay":null,"html_url":"https://alion.io/job/bitdeer-technologies-group-sr-inference-service-sre-automation-engineer","json_url":"https://alion.io/job/bitdeer-technologies-group-sr-inference-service-sre-automation-engineer.json","meta":{"generated_at":"2026-10-25T10:42:04Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":15296,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-26T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":174995},"rest":"https://alion.io/mcp/rest/get_company?id=174995"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fbitdeer-technologies-group-sr-inference-service-sre-automation-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fbitdeer-technologies-group-sr-inference-service-sre-automation-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fbitdeer-technologies-group-sr-inference-service-sre-automation-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/bitdeer-technologies-group-sr-inference-service-sre-automation-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fbitdeer-technologies-group-sr-inference-service-sre-automation-engineer"}]}