Sr. Inference Service SRE & Automation Engineer

About Bitdeer:

Bitdeer is a world-leading technology company for Bitcoin mining and AI cloud.

Bitdeer is committed to providing comprehensive Bitcoin mining solutions for its customers. Apart from designing industry-leading ASIC chips and manufacturing mining rigs, the Group handles complex processes involved in computing across the value chain. This includes equipment procurement, transport logistics, datacenter design and construction, equipment management, and network and facility operations. Bitdeer also offers advanced cloud capabilities to customers with a high demand for artificial intelligence.

Headquartered in Singapore, Bitdeer operates globally with a diversified 3 GW energy portfolio, and deploys Bitcoin mining and HPC datacenters in the United States, Bhutan, Norway, Canada, Malaysia, and Ethiopia.

About the role

NeoCloud runs Infermesh, our managed Inference / Model-as-a-Service platform: customers call model endpoints over a public API and pay per token. For a service like this, reliability is the product - a failed request is not an internal incident, it is a customer-visible SLA breach. In this role you own the reliability of the inference serving path end to end: public API gateway, inference router, request scheduler and queue, prefill/decode and KV cache, inference engines, and the GPU model replicas behind them - plus the control plane and shared dependencies they rely on. Your mandate is to aim a 99.95% SLA, by closing the architecture, automation, and operational gaps that stand in the way. You are the SRE who stands between a fleet of H200/B300/GB300/VR NVL72 serving nodes and the customer's experience of a simple, dependable model endpoint.

What you will be responsible for:

• End-to-end availability of customer inference endpoints - API availability, request success rate, TTFT/TPOT latency, and the published SLA.
• The serving path: API gateway, inference router, health-based routing, request scheduler, queueing and admission control, prefill/decode serving, and the KV cache service.
• Inference runtime operations: engine health (vLLM / SGLang / TensorRT-LLM class runtimes), model load and readiness validation, model revision rollout and rollback.
• Model replica topology and failure isolation: tensor-parallel group containment, GPU / NVLink / NCCL fault isolation, replica anti-affinity across nodes, racks, and failure domains.
• Multi-cluster serving across MY / SG / JP: cross-cluster traffic steering, tested cluster failover, and regional recovery or evacuation scope.
• GPU serving fleet health (H200 / B300 / GB300 / VR NVL72 bare metal): DCGM / Xid / ECC monitoring, automated drain and cordon, replica replacement with no sustained customer impact.
• The Kubernetes platform beneath the serving path: HA control plane, etcd, CNI, CoreDNS, public / management / east-west edge tiers, and egress NAT.
• Shared control-plane dependencies: PostgreSQL (including Temporal), Redis, Vault secrets and certificates, object storage and model registry, ClickHouse - HA, tested failover, and decoupling live serving from control-plane outages.
• Capacity and saturation engineering: GPU utilization, KV cache occupancy, queue depth and concurrency; N+1 headroom, admission control, load shedding, and graceful degradation instead of collapse.
• The SLI/SLO/SLA system: define and instrument the SLIs, publish the SLOs, run error budgets and burn-rate alerting, and produce customer-facing SLA reporting.
• Observability stack - VictoriaMetrics, VictoriaLogs, Tempo, Pyroscope, Grafana, ClickHouse - plus external synthetic inference probes that exercise the real customer API path.
• Incident management: severity model, incident commander rotation, primary / secondary / cross-team / vendor escalation, status-page and customer communication, and postmortems with tracked actions.
• Change safety: staged and canary rollout for application, model, and configuration changes; automatic rollback on regression; production change review and release risk classification.
• Resilience validation: recurring failover drills, DR tests, game days and fault injection, and backup restore verification - with measured RTO as the evidence.

Reliability ownership

• You are accountable for the customer's reliability experience - when an endpoint degrades, a replica dies, or a router loses a backend, you own detection, remediation, and the communication loop.
• Drive the platform from to 99.95%, using the SLA gap assessment - architecture, operations, and failure-mode matrices - as your roadmap.
• Own the failure-mode and resilience matrix: every Critical and High failure mode gets a detection signal, a mitigation path, a target RTO, and a drill that proves it.
• Define and publish customer-facing SLOs and SLAs, and drive error-budget-based prioritization between feature work and reliability work.
• Partner with support and customer success to close the loop from customer-reported issues to systemic fixes, and build tenant-visible status, usage, and quota observability that reduces support load.

Feed the AIOps substrat e

• The remediation actuator and workflow engine land here - you make the serving control plane safe for automated action.
• Your runbooks, CRDs, and workflow definitions are the schema the platform's predictors and remediators write against.
• Every human intervention you do this quarter becomes an autonomous workflow next quarter - turning customer-impacting incidents into self-healing events.

What success looks like in year 1

• Every Critical / High failure mode in the resilience matrix has tested detection, an automated or documented mitigation, and a measured RTO.
• Cross-cluster inference failover drilled and completing inside the SLA target; no single-cluster or shared-component SPOF left unidentified.
• Automatic rollback in place for bad application, model, and configuration deployments - a known-bad canary is reverted without human action.
• External synthetic inference probing and SLO dashboards live; error-budget burn thresholds trigger documented action.
• MTTD and MTTR for customer-impacting incidents measurably reduced, with the top recurring interventions converted into automated workflows.
• On-call rotation with no single-person dependency, and P1/P2 runbooks an on-call engineer can execute without the original author.

How you will stand out:

• 5+ years in SRE / production platform operations, including 2+ years on a customer-facing API service operating against a published SLA.
• Deep Kubernetes operations experience: HA control plane, etcd, CNI, service discovery, ingress and edge tiers, and multi-cluster topologies.
• Hands-on production experience serving LLM or other model inference - vLLM, SGLang, TensorRT-LLM, Triton, or equivalent: model loading, batching, KV cache behaviour, concurrency limits, TTFT/TPOT latency.
• GPU fleet operations: DCGM / Xid / ECC signal triage, NVLink and NCCL failure diagnosis, node drain and replica recovery.
• Traffic engineering for an availability-critical API: L4/L7 load balancing, gateways, active health checking, DNS/GSLB, and cross-cluster or cross-region failover.
• Strong SRE fundamentals: SLI/SLO/SLA design, error budgets, incident command, postmortems, and capacity planning.
• Operating stateful dependencies in production: PostgreSQL replication and failover, Redis, message queues, object storage, and secrets / certificate lifecycle.
• Observability at scale: Prometheus- or VictoriaMetrics-class TSDB, Grafana, log and trace pipelines, synthetic probing, and alert quality management.
• Overload protection in practice: admission control, concurrency and queue limits, load shedding, and graceful degradation.
• Proficiency in Terraform, Helm, and GitOps workflows (ArgoCD/Flux).
• Strong programming skills in Go or Python for automation, controllers, and operator development.
• AIOps aptitude - you view the control plane as an execution surface for automated remediation, not just a scheduler.
• Runbook-as-code mindset - every SRE playbook you write should be executable by the platform.
• Comfortable with customer-facing incident communication and SLA reporting.

What you will experience working with us:

• A culture that values authenticity and diversity of thoughts and backgrounds;
• An inclusive and respectable environment with open workspaces and exciting start-up spirit;
• Fast-growing company with the chance to network with industrial pioneers and enthusiasts;
• Ability to contribute directly and make an impact on the future of the digital asset industry;
• Involvement in new projects, developing processes/systems;
• Personal accountability, autonomy, fast growth, and learning opportunities;
• Attractive welfare benefits and developmental opportunities such as training and mentoring.

--------------------------------------------------------------------

Bitdeer is committed to providing equal employment opportunities in accordance with country, state, and local laws. Bitdeer does not discriminate against employees or applicants based on conditions such as race, colour, gender identity and/or expression, sexual orientation, marital and/or parental status, religion, political opinion, nationality, ethnic background or social origin, social status, disability, age, indigenous status, and union.