{"id":525575,"url":"https://alion.io/job/firmus-technologies-senior-ai-engineer-kubernetes-customised-scheduler","title":"Senior AI Engineer (Kubernetes & Customised Scheduler)","company":{"id":63306,"name":"Firmus Technologies","domain":"firmus.ai","url":"https://alion.io/company/firmus-technologies","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Singapore"],"countries":["SG"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":62000,"max_usd":135000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":28},"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"ArgoCD","optional":false},{"name":"CI/CD","optional":false},{"name":"Fine-tuning","optional":false},{"name":"GitHub Actions","optional":false},{"name":"GitLab CI","optional":false},{"name":"GitOps","optional":false},{"name":"Go","optional":false},{"name":"Grafana","optional":false},{"name":"gRPC","optional":false},{"name":"Helm","optional":false},{"name":"HPC","optional":false},{"name":"Kubernetes","optional":false},{"name":"Kustomize","optional":false},{"name":"Kyverno","optional":false},{"name":"NVLink","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Platform Engineering","optional":false},{"name":"Prometheus","optional":false},{"name":"Python","optional":false},{"name":"Rest API","optional":false},{"name":"SLI/SLO/SLA","optional":false},{"name":"SLURM","optional":false},{"name":"SRE","optional":false},{"name":"Terraform","optional":false}],"status":"live","first_seen_at":"2026-09-08T02:26:07Z","employer_posted_date":"2026-09-22","last_verified_at":"2026-10-30T06:31:16Z","board_verified":true,"closed_at":null,"days_open":52,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":52},"description":"Firmus Technologies\nFirmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific.\nFounded in Australia in 2019, our mission is to create the most efficient AI infrastructure by combining cutting-edge technology with a steadfast commitment to sustainability. \nAt Firmus, we are unique in our approach. We design, build, and operate a new class of digital infrastructure - the AI Factory. Through our model-to-grid technology approach, we have pushed the boundaries of multi-generational liquid cooling systems, energy management, AI software orchestration, and construction. For our customers, this approach allows us to make every watt count and deliver low-cost AI tokens globally.\nFirmus AI Cloud\nOur large-scale GPU cloud platform, Firmus AI Cloud, is purpose-built to deliver energy-efficient AI compute at scale to customers.\nIt empowers developers, enterprises, educational institutions, and government users to train and deploy AI models with unmatched efficiency and cost savings. With an ever-growing suite of services and applications, we are committed to delivering a cloud experience that is market-leading, proprietary, and built to scale.\nWhy Firmus?\nAs an NVIDIA Cloud and Engineering partner in Asia Pacific, you will gain skills, experience, and exposure across the AI industry and be part of shaping what this industry looks like for decades to come.\nWe are founder-led, not a big corporate. Decisions happen fast, our leaders are accessible, and there's minimum bureaucracy between you and the work. Ownership comes early. Whatever your role, you will have a direct line to outcomes, helping shape how the business grows as we scale nationally across a long-term, large-scale roadmap. \nWork alongside founders and experts in AI infrastructure, energy systems and next-generation compute.\nWhat we build here has impact beyond the business. Our AI Factories are designed to operate as assets to the energy grid to actively strengthen the communities and regions they operate in rather than drawing from them.\nConsidering applying? You don't need a perfect background to join our team. If you're driven and curious, there's a path for you. We back our people to grow into new domains and take on challenges beyond their previous experience.\nROLE SUMMARY\nThe Senior AI Engineer will be a core builder of the AI & Applications team’s Model-to-Grid product, designing, developing, and operating the Kubernetes-based workload orchestration layer for massive-scale AI factories. The role has a strong software and platform engineering focus: building a proprietary job scheduler that enables efficient, reliable, and policy-driven execution of training, fine-tuning, inference, batch, benchmarking, and agentic workloads across cutting-edge GPU systems, including NVIDIA NVL72 GB300-scale deployments and future-generation platforms such as VR200.\nThe proprietary scheduler is a central product differentiator. It must be network-topology aware and AI-factory-resource aware: making placement, queueing, prioritization, admission, and execution decisions based not only on nominal GPU availability, but also on GPU and NVLink/NVSwitch topology, node and fault-domain boundaries, RDMA and fabric health, storage locality and throughput, workload characteristics, capacity, power, thermal state, maintenance activity, and other operational constraints.\nThe resulting capability improves outcomes in two directions. For AI users, it provides better workload placement, lower queue times, higher GPU utilization, stronger job-success rates, improved end-to-end throughput, and faster time-to-results. For AI-factory and grid operators, it serves as a technical shock absorber by making demand more observable, controllable, schedulable, and responsive to infrastructure availability, system health, power, thermal, capacity, and operational conditions.\nThe role will work closely with other AI engineers, inference and optimization engineers, the Model-to-Grid product and program lead, Platform, Infrastructure, Security, and operations teams. It will turn scheduler product requirements into robust production capabilities, integrating Kubernetes, custom controllers, APIs, observability, automation, workload recipes, benchmark signals, and supporting platform services. The work will align with the architectural direction of NVIDIA DSX OS and AI Factory Blueprint concepts - co-designed, resilient, multi-tenant AI-factory operations, while delivering the organization’s proprietary scheduler intelligence and differentiated Model-to-Grid capabilities.\nKEY RESPONSIBILITIES\nDesign, build, operate, and continuously improve the proprietary Kubernetes-native job scheduling platform for AI workloads.\nDevelop scheduler architecture, custom resource definitions, Kubernetes controllers, admission webhooks, scheduling plugins, APIs, CLI tools, and automation required to support workload submission, placement, execution, monitoring, and recovery.\nDefine and implement scheduling policies for training, fine-tuning, inference, benchmarking, batch, data-processing, and agentic workloads.\nBuild topology-aware placement mechanisms that account for GPU locality, NVLink/NVSwitch domains, node topology, NUMA affinity, NIC placement, RDMA paths, network-fabric topology, storage locality, and fault-domain boundaries.\nBuild AI-factory resource-aware scheduling mechanisms that account for cluster capacity, node health, fabric condition, storage performance, GPU availability, maintenance windows, software compatibility, capacity reservations, power limits, thermal conditions, and operational constraints.\nImplement workload-control capabilities including admission control, queueing, priorities, quotas, fair sharing, reservations, preemption, gang scheduling, co-scheduling, backfilling, workload aging, retry policies, checkpoint-aware scheduling, deferred execution, and failure recovery.\nDevelop workload demand-shaping and grid-aware controls that allow eligible workloads to be delayed, paced, prioritized, rescheduled, right-sized, or placed differently in response to capacity, health, power, thermal, maintenance, or other AI-factory operating signals.\nIntegrate the scheduler with Kubernetes control-plane capabilities, GPU device plugins, node-feature discovery, network and storage services, observability systems, and policy engines.\nIntegrate, where appropriate, with Slurm, Slinky, vCluster, Kueue, KAI, Volcano, YuniKorn, or related systems to deliver a coherent workload-management experience across AI and HPC use cases.\nBuild a stable developer and user experience for workload submission and management, including APIs, SDKs, CLI workflows, templates, workload definitions, job-status visibility, event streams, scheduling explanations, and self-service troubleshooting capabilities.\nWork with AI and inference engineers to encode validated model and workload recipes into scheduler-aware templates, including requirements for model size, precision, distributed parallelism, GPU count, topology, network, storage, runtime, benchmark target, and expected resource profile.\nEnable Model-to-Grid benchmarking by exposing scheduler, placement, resource, and workload-lifecycle data for end-to-end analysis across models, runtimes, GPUs, network, storage, power, thermals, and application performance.\nBuild observability into the scheduling platform, including queue depth, scheduling latency, admission outcomes, placement decisions, resource fragmentation, topology quality, GPU utilization, job lifecycle, failure reasons, preemption events, retry behavior, power and thermal signals, and workload performance.\nDevelop actionable scheduler explanations and operator views that show why a workload was queued, admitted, placed, deferred, preempted, or rescheduled and what changes could improve execution outcomes.\nEstablish performance testing, load testing, scale testing, chaos testing, fault recovery testing, and regression testing for scheduler components and Kubernetes platform integrations.\nOperate production Kubernetes clusters and associated scheduling services with strong reliability, security, change-management, backup, recovery, and incident-response practices.\nBuild CI/CD and GitOps workflows for scheduler code, configuration, CRDs, controllers, policy definitions, workload templates, and cluster-service releases.\nImplement secure multi-tenancy and resource-governance controls, including namespace and tenant isolation, RBAC, quotas, workload identity, policy enforcement, audit trails, and controlled access to GPU, data, and infrastructure resources.\nPartner with platform, infrastructure, networking, storage, and operations teams to ensure scheduler decisions reflect actual infrastructure capabilities, constraints, health, performance, maintenance, and capacity conditions.\nPartner with the Model-to-Grid product and program lead to define scheduler feature scope, milestone plans, acceptance criteria, release gates, benchmark targets, documentation, and adoption measures.\nContribute to technical roadmaps, architecture decisions, design reviews, release readiness, incident postmortems, and continuous improvement of Model-to-Grid and AI-factory operations.\nSKILLS AND EXPERIENCE\n5+ years of experience in DevOps, site reliability engineering, platform engineering, distributed systems, cloud infrastructure, or comparable software and systems-engineering roles.\nDeep practical expertise in Kubernetes architecture and operations, including control planes, scheduling, controllers, operators, CRDs, admission controllers, APIs, networking, storage, security, and multi-tenancy.\nDemonstrated experience building, extending, or deeply integrating workload schedulers, resource managers, or orchestration systems, such as Kubernetes Scheduler Framework, Kueue, KAI, Volcano, YuniKorn, Slurm, Slinky, Run:ai, or proprietary implementations.\nStrong programming skills in Go, with Python proficiency for automation, integration, tooling, and performance analysis.\nExperience building production Kubernetes controllers, operators, scheduling plugins, admission webhooks, REST or gRPC APIs, CLI tools, and event-driven distributed services.\nStrong understanding of GPU-accelerated AI/ML workloads, including distributed training, fine-tuning, model serving, inference, benchmarking, and batch processing.\nPractical experience with GPU scheduling and resource allocation, including GPU partitioning, MIG where applicable, GPU affinity, multi-GPU workloads, gang scheduling, and topology-aware placement.\nUnderstanding of GPU system topology and high-performance AI networking, including NVLink, NVSwitch, PCIe, NUMA, RDMA, RoCEv2, NIC affinity, collective communication, and network contention.\nFamiliarity with large-scale NVIDIA GPU architectures, including NVL72 GB300-class systems and future-generation high-density GPU platforms, and their implications for cluster scheduling, workload placement, network topology, operations, and fault handling.\nExperience working with storage and data-path considerations for AI workloads, including high-throughput shared file systems, object storage, caching, data locality, checkpointing, and storage-performance constraints.\nUnderstanding of AI-factory or data-center operations, including capacity planning, workload demand forecasting, power-aware scheduling, energy efficiency, thermal constraints, maintenance coordination, system health, resiliency, and infrastructure telemetry.\nExperience with observability and performance analysis using tools and concepts such as Prometheus, Grafana, OpenTelemetry, logs, traces, metrics, DCGM, GPU telemetry, network telemetry, workload profiling, and SLOs.\nExperience with CI/CD, GitOps, infrastructure as code, policy as code, and safe production rollout practices using tools such as GitHub Actions, GitLab CI, Argo CD, Flux, Terraform, Helm, Kustomize, OPA, or Kyverno.\nFamiliarity with...","description_format":"text","description_chars":17188,"description_truncated":true,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Data Centers & Colocation","Cloud Platforms (IaaS & PaaS)","AI Compute & Inference"],"lifecycle":[{"event":"open","at":"2026-09-08T05:21:22Z"}],"visa":[],"liveness":{"score":76,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.844,"p_room":0.9,"age_days":52,"expected_fill_days":81,"reasons":["conf:3","velocity","win:mid"],"computed_at":"2026-10-30T05:48:23Z"},"pay":null,"html_url":"https://alion.io/job/firmus-technologies-senior-ai-engineer-kubernetes-customised-scheduler","json_url":"https://alion.io/job/firmus-technologies-senior-ai-engineer-kubernetes-customised-scheduler.json","meta":{"generated_at":"2026-10-30T08:17:21Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":11722,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-31T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":63306},"rest":"https://alion.io/mcp/rest/get_company?id=63306"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Ffirmus-technologies-senior-ai-engineer-kubernetes-customised-scheduler"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Ffirmus-technologies-senior-ai-engineer-kubernetes-customised-scheduler"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Ffirmus-technologies-senior-ai-engineer-kubernetes-customised-scheduler"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/firmus-technologies-senior-ai-engineer-kubernetes-customised-scheduler\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Ffirmus-technologies-senior-ai-engineer-kubernetes-customised-scheduler"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":525575}},{"tool":"search_jobs","arguments":{"company":"firmus.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":525575},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}