{"id":3264400,"url":"https://alion.io/job/yal-senior-ml-engineer-inference-and-model-serving","title":"Senior ML Engineer - Inference and Model Serving","company":{"id":20378,"name":"YAL","domain":"yal.inc","url":"https://alion.io/company/yal","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Keka","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Hyderabad, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":22000,"max_usd":46000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":41},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Apache TVM","optional":false},{"name":"C++","optional":false},{"name":"CI/CD","optional":false},{"name":"Hugging Face","optional":false},{"name":"Kubernetes","optional":false},{"name":"KV Cache","optional":false},{"name":"LoRA","optional":false},{"name":"MLFlow","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Prometheus","optional":false},{"name":"Python","optional":false},{"name":"PyTorch","optional":false},{"name":"SGLang","optional":false},{"name":"Speculative Decoding","optional":false},{"name":"Speech Recognition","optional":false},{"name":"TensorRT","optional":false},{"name":"TensorRT-LLM","optional":false},{"name":"Triton","optional":false},{"name":"Triton Inference Server","optional":false},{"name":"vLLM","optional":false},{"name":"Apache Kafka","optional":true},{"name":"CUDA","optional":true},{"name":"Envoy","optional":true},{"name":"Kong","optional":true},{"name":"Multimodal AI","optional":true},{"name":"PEFT","optional":true},{"name":"Redpanda","optional":true}],"status":"live","first_seen_at":"2026-10-26T06:46:59Z","employer_posted_date":"2026-10-26","last_verified_at":"2026-10-28T16:29:46Z","board_verified":true,"closed_at":null,"days_open":2,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":2},"description":"Role overview\nWe are hiring a senior ML engineer to own how our language and speech models run in production. Our\nagents depend on fast, reliable inference: every spoken reply passes through speech recognition, a\nlanguage model and speech synthesis within a tight latency budget. You will serve these models on\nGPUs at the lowest cost per call without losing quality, build the security and metering layers around\nthem, and automate how models move from registry to production. You will work with the architect on\nmodel-serving interfaces and with the AI engineers who build agents on top.\nKey responsibilities\nLanguage model serving\nServe open-weight language models with engines such as vLLM, TensorRT-LLM or SGLang, tuned\nfor both time to first token and throughput.\nApply continuous batching, prefix caching, speculative decoding and multi-LoRA serving to cut\nlatency and cost per request.\nManage GPU memory with PagedAttention, FlashAttention, KV-cache quantisation and tensor\nor pipeline parallelism, to maximise batch size and context length without out-of-memory\nfailures.\nQuantise models (INT8, INT4, FP8) and choose precision per model based on measured quality\nand speed.\nSpeech model serving\nServe streaming speech recognition and speech synthesis models on GPUs, with low time to first\npartial transcript and time to first audio.\nBatch and schedule many concurrent audio streams on shared GPUs without letting one stream\ndelay another.\nWork with AI engineers to optimise end-to-end latency of the speech pipeline, from audio\nchunking to model execution to streaming output.\nPlace speech and language models on GPUs to balance latency, utilisation and cost when they\nrun side by side.\nPerformance and quality\nProfile models and serving stacks with tools such as NVIDIA Nsight Systems and Nsight Compute,\nthen find and fix bottlenecks.\nUse compilers and runtimes such as TensorRT, TVM or PyTorch Inductor to fuse operations and\nreduce memory traffic.\nCheck every optimisation against quality evals - task accuracy, word error rate and voice\nquality - and block any change that makes a model faster but worse.\nBenchmark candidate models and hardware on latency, throughput, quality and cost, and\nrecommend what to serve.\nScale, cost and deployment\nAutoscale GPU capacity with traffic, plan capacity for peaks, and track cost per request, per call\nand per tenant.\nDeploy to cloud, on-premises and restricted-network environments, including air-gapped installs\nfor enterprise clients.\nBuild fallback paths across model sizes, replicas and providers, so traffic keeps flowing when a\nmodel or node fails.\nAccess, metering and MLOps\nDesign secure model endpoints with OAuth2, JWT and role-based access control, and enforce\nstrict multi-tenant isolation.\nBuild high-throughput metering for input and output tokens, audio seconds and GPU time,\nfeeding rate limits, billing and cost allocation.\nBuild CI/CD pipelines that profile, regression-test and deploy models from registries such as\nMLflow or Hugging Face Hub into Kubernetes.\nMonitor latency, throughput, GPU utilisation and errors with OpenTelemetry and Prometheus,\nand respond to production incidents.\nRequired qualifications and skills\nStrong Python and working C++ skills.\nHands-on experience serving language models in production with engines such as vLLM,\nTensorRT-LLM, SGLang or Triton Inference Server.\nExperience serving or optimising speech or other streaming models under strict real-time\nlatency budgets.\nSolid understanding of GPU architecture - streaming multiprocessors, memory hierarchy,\nasynchronous copies - and how it drives inference performance.\nPractical experience with quantisation, batching strategies, KV-cache management and model\nparallelism.\nExperience profiling and fixing performance problems with tools such as Nsight Systems or\nPyTorch Profiler.\nExperience running GPU workloads on Kubernetes, with CI/CD, monitoring and autoscaling.\nAbility to measure quality alongside speed, and to explain trade-offs between latency,\nthroughput, cost and accuracy.\nA degree in computer science, computer or electrical engineering, or a related field, or\nequivalent practical experience, with clear examples of delivered work.\nPreferred experience\nWriting or modifying custom GPU kernels in CUDA or Triton, including kernel fusion.\nDeep familiarity with recent NVIDIA architectures such as Ampere, Hopper and Blackwell.\nServing large models (70B+ parameters) across multiple GPUs or nodes under strict latency\nbudgets.\nAPI gateways and rate limiting (Envoy, Kong), and streaming systems (Kafka, Redpanda).\nAir-gapped or on-premises GPU deployments for enterprise clients.\nServing vision or multimodal models.\nA masters degree or PhD in a relevant field.\nSuccess in the role\nSuccess means our language and speech models respond fast enough for natural conversation, at a cost\nper call that scales with the business. In your first six months, that looks like:\nLatency: speech and language models meet their per-hop budgets at p95 under peak traffic.\nCost: cost per call and GPU utilisation are tracked, and both improve measurably.\nQuality: no optimisation ships without passing quality evals, and quality never drops unnoticed.\nReliability: model endpoints stay available through node and model failures, with fallbacks that\nwork.\nDeployment: new models reach production through an automated, tested pipeline, in the cloud\nor on-premises.\nMetering: every tenants usage is measured accurately for rate limits and billing.","description_format":"text","description_chars":5497,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"master","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Conversational AI","Speech & Audio AI","AI Agents"],"lifecycle":[{"event":"open","at":"2026-10-26T08:07:24Z"}],"visa":[],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":1,"expected_fill_days":42,"reasons":["conf:3","velocity","win:early"],"computed_at":"2026-10-28T05:59:10Z"},"pay":null,"html_url":"https://alion.io/job/yal-senior-ml-engineer-inference-and-model-serving","json_url":"https://alion.io/job/yal-senior-ml-engineer-inference-and-model-serving.json","meta":{"generated_at":"2026-10-28T18:58:27Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":25913,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-29T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":20378},"rest":"https://alion.io/mcp/rest/get_company?id=20378"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fyal-senior-ml-engineer-inference-and-model-serving"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fyal-senior-ml-engineer-inference-and-model-serving"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fyal-senior-ml-engineer-inference-and-model-serving"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/yal-senior-ml-engineer-inference-and-model-serving\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fyal-senior-ml-engineer-inference-and-model-serving"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3264400}},{"tool":"search_jobs","arguments":{"company":"yal.inc","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3264400},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}