{"id":2081663,"url":"https://alion.io/job/gnani-llmops-engineer","title":"LLMOps Engineer","company":{"id":27998,"name":"Gnani","domain":"gnani.ai","url":"https://alion.io/company/gnani","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":26000,"max_usd":53000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":27},"experience_years_min":6,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"CUDA","optional":false},{"name":"CUDA Toolkit","optional":false},{"name":"CUTLASS","optional":false},{"name":"Knowledge Distillation","optional":false},{"name":"Kubernetes","optional":false},{"name":"KV Cache","optional":false},{"name":"LLM","optional":false},{"name":"LLMOps","optional":false},{"name":"LoRA","optional":false},{"name":"Mixture of Experts","optional":false},{"name":"Model Distillation","optional":false},{"name":"Post-training","optional":false},{"name":"PyTorch","optional":false},{"name":"Quantization","optional":false},{"name":"SGLang","optional":false},{"name":"Speculative Decoding","optional":false},{"name":"TensorRT-LLM","optional":false},{"name":"Triton","optional":false},{"name":"vLLM","optional":false},{"name":"DeepSpeed","optional":true},{"name":"FSDP","optional":true},{"name":"KServe","optional":true},{"name":"Mamba","optional":true},{"name":"Megatron-LM","optional":true},{"name":"NVIDIA NeMo","optional":true},{"name":"NVIDIA NIM","optional":true},{"name":"PEFT","optional":true},{"name":"Ray","optional":true},{"name":"Ray Serve","optional":true},{"name":"Speech Recognition","optional":true},{"name":"TensorRT","optional":true},{"name":"Text-to-Speech","optional":true},{"name":"Triton Inference Server","optional":true}],"status":"live","first_seen_at":"2026-10-08T09:46:31Z","employer_posted_date":null,"last_verified_at":"2026-10-08T09:46:31Z","board_verified":false,"closed_at":null,"days_open":5,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":5},"description":"Role Overview:\n\nWe are hiring an LLMOps Engineer to own how our large language models actually run in production - the serving stack, the optimization work that makes it affordable, and the low-level performance engineering that makes it fast. This is a deep infrastructure and performance role, not a wrapper-and-API role.\n\nYou will work on self-hosted foundational models served on large-scale NVIDIA GPU clusters, under real-time latency budgets set by live spoken conversations. Voice inference is unforgiving: time-to-first-token is a product feature, not a number on a dashboard, and every millisecond of tail latency is audible to a caller. Your mandate is to hold latency and quality while driving cost per million tokens down.\n\nThe work spans three layers: the serving engine (vLLM, SGLang, NVIDIA Dynamo), the model itself (quantization, distillation, speculative decoding, pruning), and the kernels underneath (CUDA/Triton, attention and MoE kernels, profiling and bottleneck analysis). We expect real depth in at least two of the three and genuine curiosity about the third.\n\nKey Responsibilities:\n\nInference Serving and Platform:\n\n- Own the production LLM serving stack across vLLM, SGLang, and NVIDIA Dynamo - including engine selection per workload, with benchmark evidence to justify it\n\n- Tune the serving path end to end: continuous batching, chunked prefill, paged and radix attention, prefix and KV caching, cache-aware and sticky routing, speculative decode integration\n\n- Design and operate disaggregated prefill/decode and multi-node deployments; configure tensor, pipeline, and expert parallelism for both dense and Mixture-of-Experts models\n\n- Run inference on Kubernetes with autoscaling, rolling model updates, canary releases, and tested rollback paths; enforce staging-to-production promotion gates\n\n- Instrument everything that matters: TTFT, inter-token latency, p50/p95/p99, tokens/sec/GPU, KV cache hit rate, queue depth, GPU utilization and MFU, and cost per million tokens\n\n- Build and maintain internal serving APIs, model registries, and deployment tooling so that model updates are routine rather than events\n\nModel Optimization:\n\n- Quantization: FP8, INT8, and INT4 weight and activation quantization plus KV cache quantization - calibration set design, per-layer sensitivity analysis, and accuracy recovery, with measured deltas per language and task\n\n- Speculative decoding: draft-model, EAGLE/Medusa-style, and n-gram approaches; tuned for real acceptance rate and end-to-end latency gain under production traffic, not theoretical speedup\n\n- Distillation: build smaller task-specific students from larger teachers for latency-critical paths, held to explicit eval-parity targets\n\n- Pruning and adapters: structured pruning, LoRA/adapter serving, and multi-adapter batching for per-deployment specialization\n\n- Compilation: torch.compile, CUDA graphs, and TensorRT-LLM engine builds - including diagnosing recompilation and dynamic-shape stalls\n\nKernel and Low-Level Performance:\n\n- Profile with Nsight Systems/Compute and the PyTorch profiler; classify bottlenecks as memory-bound, compute-bound, or launch/synchronization-bound, and act on the classification\n\n- Write and tune custom kernels in CUDA and Triton - attention variants, fused MoE dispatch, sampling, quantized GEMM\n\n- Eliminate host-device synchronization stalls, dynamic-shape recompilation, and unoverlapped collectives in distributed serving\n\n- Work fluently with attention and MoE kernel libraries (FlashAttention, FlashInfer, CUTLASS/cuBLAS, expert-parallel dispatch libraries) and know when to use them versus write your own\n\n- Reason from first principles about arithmetic intensity, memory bandwidth, and roofline limits before reaching for a tool\n\nEvaluation, Reliability, and Cost:\n\n- Own the optimization regression gate: no optimized build reaches production without an accuracy and behavior evaluation across languages and task types\n\n- Build load-testing harnesses that replay realistic traffic - concurrency, length distributions, burstiness, multi-turn sessions\n\n- Run capacity planning and cost modeling; set and hit targets for cost per million tokens and per concurrent session\n\n- Carry on-call for inference services, write runbooks, and lead blameless postmortems on latency and availability incidents\n\n- Work closely with the training and post-training teams so that serving constraints inform model architecture decisions early, not after the checkpoint lands\n\nMust Have:\n\n- 6 - 10 years total experience, with 3+ years owning large-scale LLM or speech inference in production\n\n- Has owned an inference platform end to end: architecture, SLOs, capacity, cost, deployment safety, and on-call\n\n- Has written or substantially tuned custom CUDA or Triton kernels that shipped to production\n\n- MoE serving at scale: expert parallelism, routing load imbalance, dispatch kernels, and the failure modes specific to sparse models\n\n- Deep profiling ability - has diagnosed and fixed a utilization or MFU collapse in a distributed multi-node system, and can walk through the investigation\n\n- Track record running an accuracy-preserving optimization program (quantization plus speculative decoding plus distillation) behind real evaluation gates\n\n- Sets technical direction, mentors engineers, and can make a defensible build-versus-adopt call on serving infrastructure\n\n- Upstream contributions to open-source serving or kernel projects (vLLM, SGLang, FlashInfer, TensorRT-LLM, or similar) - a strong plus, and close to an expectation at this level\n\nGood to Have:\n\n- Real-time speech inference: streaming ASR/TTS, sub-second time-to-first-audio budgets, barge-in and turn-taking constraints\n\n- Serving hybrid-architecture models (state-space/Mamba blocks combined with attention) and understanding their distinct state and cache management\n\n- TensorRT-LLM engine building, NVIDIA NIM, or Triton Inference Server in production\n\n- Experience with Indic or other multilingual and code-mixed models, and the evaluation discipline that comes with them\n\n- Ray Serve, KServe, LLM gateways, or semantic and prefix cache layers\n\n- Enough training-side exposure (Megatron, NeMo, DeepSpeed, FSDP) to work fluently with the pretraining team\n\n- Public benchmarks, blog posts, or talks on inference optimization\n\nWhat You Will Work On:\n\nYou will work on in-house foundational models - not third-party API endpoints - running on large NVIDIA GPU clusters and serving live enterprise and government-scale voice traffic. The models are ours, so the whole stack is open to you: you can change the serving engine, the quantization recipe, the kernel, or the model itself, and you will often need to change more than one to hit a target.\n\nThe constraints are real and the feedback loop is fast. A latency regression is heard by callers within minutes. A successful optimization shows up directly in infrastructure spend. You will have access to proprietary speech and language data, in-house training and evaluation infrastructure, a close engineering partnership with NVIDIA, and a team that has been building Indian-language AI since 2016.\n\nThis role has a clear path to owning the inference platform and its technical direction, or to a deeper specialization in performance and kernel engineering, depending on where your strength lies.\nSkills\nMLOps, LLM, CUDA, Kubernetes, PyTorch","description_format":"text","description_chars":7382,"description_truncated":false,"requirements":{"experience_years_min":6,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Natural Language Processing","Conversational AI","Speech & Audio AI"],"lifecycle":[{"event":"open","at":"2026-10-08T10:29:15Z"}],"visa":[],"liveness":{"score":89,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.894,"p_room":1,"age_days":4,"expected_fill_days":22,"reasons":["seen:4","velocity","win:early"],"computed_at":"2026-10-13T05:45:00Z"},"pay":null,"html_url":"https://alion.io/job/gnani-llmops-engineer","json_url":"https://alion.io/job/gnani-llmops-engineer.json","meta":{"generated_at":"2026-10-13T13:23:11Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":20682,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-14T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":27998},"rest":"https://alion.io/mcp/rest/get_company?id=27998"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fgnani-llmops-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fgnani-llmops-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fgnani-llmops-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/gnani-llmops-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fgnani-llmops-engineer"}]}