ML Engineer, Inference and Serving

About the Role

We are looking for an experienced ML Engineer to help build, deploy, and optimize machine learning systems in production. In this role, you will focus on model serving, inference performance, and scalable deployment of large language models, working closely with platform and infrastructure teams to deliver reliable and efficient AI solutions.

What You'll Do

• Design, deploy, and optimize large language model (LLM) inference systems for production environments
• Evaluate, benchmark, and improve serving performance across latency, throughput, memory utilization, and cost-efficiency metrics
• Implement and optimize model-serving solutions using frameworks such as vLLM, TensorRT-LLM, SGLang, Triton Inference Server, or similar technologies
• Develop and maintain model conversion, deployment, and evaluation workflows across the inference stack
• Apply and validate quantization techniques to improve performance while preserving model quality
• Profile GPU workloads, identify bottlenecks, and drive performance improvements using industry-standard observability and profiling tools
• Collaborate with Platform, Infrastructure, and Product Engineering teams to deploy and operate scalable AI services
• Contribute to architecture decisions related to distributed inference, model serving, caching strategies, and resource utilization
• Improve reliability, monitoring, and operational excellence of production inference systems
• Stay current with emerging advancements in LLM serving, inference optimization, and AI infrastructure technologies, and help evaluate their adoption

What We’re Looking For

• 3+ years ML engineering focused on model deployment and inference, not primarily training or application-layer LLM work.
• Production ownership of at least two of vLLM, SGLang, TensorRT-LLM under real traffic. You have diagnosed and fixed what each one does when it is misconfigured.
• LLM inference internals: attention, KV cache, prefill versus decode, memory-bandwidth-bound versus compute-bound regimes, continuous batching, chunked prefill.
• Speculative decoding in production: EAGLE-3, ngram, suffix or draft-model approaches; calibrating draft depth and acceptance rate against concurrency.
• MoE model serving: expert parallelism, attention data-parallel, the interaction with tensor parallelism and KV capacity.
• Quantization applied and validated on production models, at least one of FP8, INT8, AWQ, GPTQ, MXFP4, including measuring the accuracy impact, not only the throughput gain.
• OpenAI chat-completions conformance at the engine level: chat templates, tool-call and reasoning parsers, streaming semantics and usage blocks, cancellation behaviour.
• Benchmarking methodology: TTFT, ITL, goodput, closed versus open loop, user-centric load; hands-on with AIPerf or GenAI-Perf.
• Accuracy and quality evaluation: has built or operated an evaluation harness (lm-eval or equivalent) to validate quantized or speculative variants against a baseline.
• GPU performance diagnosis with Nsight Systems, DCGM, nvidia-smi and engine metrics; can explain a throughput regression from a Grafana board.
• Strong Python; working PyTorch and Hugging Face model formats, tokenizers and chat templates.

Nice to Have

• GPU hardware characteristics: HBM bandwidth versus compute, NVLink topology, tensor cores; hands-on with H100, L40S, B200-class.
• KV-aware routing from the model side: llm-d, Gateway API Inference Extension (InferencePool/EPP), NVIDIA Dynamo disaggregated serving.
• Prefix caching in operation: block-hash versus prefix-tree, multi-tenant isolation.
• Embeddings and reranking model serving, including multi-model bundles on Triton.
• Triton Inference Server
• Multi-node tensor parallelism: NCCL tuning, RDMA and InfiniBand, GPUDirect.
• Kubernetes and Helm sufficient to read and edit a variant's rendered deployment.
• LoRA multi-adapter serving.
• Model export and compilation workflows (ONNX, TensorRT)