{"id":2626585,"url":"https://alion.io/job/recho-senior-ai-inference-engineer-llm-speech-models","title":"Senior AI Inference Engineer (LLM & Speech Models)","company":{"id":2675564,"name":"Recho","domain":"recho-ai.com","url":"https://alion.io/company/recho-ai","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"HERP","truth_index":{"grade":"C","score":64,"open_postings":15,"ghost_share":0.6,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-10-21T05:46:15Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":null,"work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Chiyoda, Japan"],"countries":["JP"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":10000000,"max":20000000,"currency":"JPY","period":"year","gross":null,"usd_annual":126480},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":true,"relocation_package":false,"has_equity":true,"technologies":[{"name":"CUDA","optional":false},{"name":"Kubernetes","optional":false},{"name":"KV Cache","optional":false},{"name":"LLM","optional":false},{"name":"Multimodal AI","optional":false},{"name":"PyTorch","optional":false},{"name":"Ray","optional":false},{"name":"SGLang","optional":false},{"name":"SLURM","optional":false},{"name":"Speech Recognition","optional":false},{"name":"Text-to-Speech","optional":false},{"name":"Triton","optional":false},{"name":"vLLM","optional":false},{"name":"AI Agents","optional":true},{"name":"C++","optional":true},{"name":"Claude Code","optional":true},{"name":"Cursor","optional":true},{"name":"Devin","optional":true},{"name":"NCCL","optional":true},{"name":"Python","optional":true},{"name":"TensorRT","optional":true},{"name":"TensorRT-LLM","optional":true},{"name":"Triton Inference Server","optional":true}],"status":"live","first_seen_at":"2026-10-15T12:49:47Z","employer_posted_date":"2026-10-15","last_verified_at":"2026-10-22T05:07:18Z","board_verified":true,"closed_at":null,"days_open":6,"trust":{"level":"ok","repost_count":0,"flags":["company_stale"],"days_open":5},"description":"Senior AI Inference Engineer (LLM & Speech Models)\nDescription\n## In short\nYou'll design and build Recho's inference framework for speech models and LLMs, and you'll own its throughput and GPU cost. This is a hands-on senior role: you'll architect the system and write much of the code yourself.\n- **Team language:** English (Japanese not required)\n- **Salary:** ¥10M-¥20M per year, plus stock options\n- **Location:** Tokyo (Hanzomon), hybrid. Candidates based outside Japan are welcome. We offer visa sponsorship and relocation support, and we're open to discussing fully remote arrangements.\n## You might be a fit if you are…\n- An MLOps / ML platform engineer who has built and tuned LLM inference stacks (vLLM, SGLang, etc.)\n- A speech recognition or speech synthesis engineer/researcher who wants to shift toward inference systems\n- An engineer who has optimized throughput and cost for inference workloads on large GPU clusters\n## About the role\nAt Recho, we run training, evaluation, data generation and large-scale batch inference across multiple speech models and LLMs. Your core mission is to independently design and implement a high-efficiency inference framework that serves all of them.\nThis isn't about wrapping a model in an API or tweaking config files. You'll need to understand each model's architecture, inputs and outputs, sequence lengths, pre- and post-processing, and GPU characteristics. From there, you'll separate and integrate model execution, batching, data processing and scheduling into a reusable inference architecture.\nOn the LLM side, you'll build on mature engines such as vLLM and SGLang. You'll tune model adaptation, inference parameters, batching, prefill/decode behavior and request priority. Across both speech and LLMs, your north star is throughput and GPU cost.\n## What this role is *not*\n- Not an on-call or SRE role. SLA management, incident response and infrastructure maintenance for online services are not the focus.\n- GPU cluster operations and general DevOps are handled by a separate team.\n- Writing CUDA kernels or building an inference engine from scratch is not the main job.\n- RL training infrastructure belongs to our separate RL Infrastructure Engineer role. You'll partner with that team on the inference side.\n## What you'll do\n### 1. Build a common inference framework for speech models\n- Design a shared inference architecture for ASR, TTS, speech-to-speech and other audio generation models\n- Draw clear boundaries between shared and model-specific modules, with adapter/plugin interfaces for onboarding new models\n- Build the inference APIs, SDKs and tooling our model teams use every day\n### 2. Optimize large-scale batch inference for audio\n- Apply dynamic batching and bucketing for variable-length audio, and run data-parallel inference across GPUs and nodes\n- Build asynchronous pipelines that overlap preprocessing, inference, postprocessing and storage I/O\n- Measure and remove bottlenecks based on end-to-end processing time and cost\n### 3. Adapt and optimize LLM inference\n- Stand up LLM inference on mature engines such as vLLM and SGLang\n- Tune tensor/pipeline parallelism, continuous batching, KV cache and prefix cache\n- Design scheduling that accounts for prefill/decode characteristics and task priority\n### 4. Plan GPU usage for inference workloads\n- Handle GPU allocation and capacity planning for batch inference and evaluation\n- Orchestrate GPU inference jobs with Kubernetes, Slurm and Ray\n- Track and reduce GPU usage and inference cost per model and per task\nWe're happy to go into more detail during the intro call and interviews.\n## Why this role\n- **Set the playbook in an area that doesn't have one yet.** Inference for variable-length audio, codecs, vocoders, speech-to-speech and multimodal models has no established best practices. What you design becomes the foundation for every future model and team.\n- **Own inference end to end.** You'll cover both speech and LLMs, from framework design to batch inference and scheduling. That means you optimize the whole system, not just one piece of it.\n- **Directly move the business.** Inference efficiency determines GPU cost per unit of generated speech and response latency. Those decide how competitive our product is for the major enterprises running it in production.\n## Tech stack\n- **Languages & frameworks:** Python, PyTorch\n- **LLM inference:** vLLM, SGLang, TensorRT-LLM, Triton Inference Server\n- **Speech models:** ASR, TTS, speech-to-speech, audio codecs, vocoders\n- **Orchestration:** Kubernetes, Slurm, Ray\n- **Low-level optimization (nice to have):** C++, CUDA, Triton, NCCL\n## Our product\nWe build the Recho AI Voice Platform, which delivers voice conversations natural enough to be hard to tell apart from a human. We also deploy and operate Voice AI Agents for large enterprises.\nOur edge is a platform for building and improving high-quality agents quickly. It captures industry know-how and turns user feedback into a fast, continuous improvement loop. Our agents now handle complex, large-scale contact center operations in finance, manufacturing and the public sector. In some deployments, they've resolved every call without handing off to a human agent.\n## Hiring process\nIntro call → Case assignment & first interview → Paid work trial* → Final interview → Offer meeting\n*The work trial is 1-2 consecutive days at our Tokyo office, working on real tasks alongside the team, and it's paid. It's a chance for both sides to check fit and get a feel for the work.\n- **If you're based outside Japan,** we'll agree on the format together. We won't ask you to travel to Japan just for this step.\n- If your schedule or your current employer's side-work policy makes the trial difficult, just let us know.\n- We may also ask for references.\n## About Recho\nRecho develops foundation models for customer-support AI agents and also deploys them for clients, all within one organization. Our core strength is voice AI technology we built entirely in-house, which resolves the industry's long-standing trade-off between quality and cost.\nWe've closed a Series A of approximately ¥700M and are rapidly rolling out to demanding production environments at major companies such as SBI Shinsei Bank and KDDI.\n## Learn more\n- [Corporate website](https://recho-ai.com/)\n- [Team interviews (note, in Japanese)](https://note.com/recho_ai)\n- [Company deck](https://recruit.recho-ai.com/)\n- [Case study: Partnership with KDDI and Altius Link (Japanese)](https://prtimes.jp/main/html/rd/p/000000021.000159800.html)\n- [Case study: SBI Shinsei Bank (Japanese)](https://mainichi.jp/articles/20260130/k00/00m/020/275000c)\n- [2-minute voice demo](https://recruit.recho-ai.com/#2f276fb799f480d5bba7d5784d6ad31e) - give it a listen!\nRequirements\n## In short\nYou'll design and build Recho's inference framework for speech models and LLMs, and you'll own its throughput and GPU cost. This is a hands-on senior role: you'll architect the system and write much of the code yourself.\n- **Team language:** English (Japanese not required)\n- **Salary:** ¥10M-¥20M per year, plus stock options\n- **Location:** Tokyo (Hanzomon), hybrid. Candidates based outside Japan are welcome. We offer visa sponsorship and relocation support, and we're open to discussing fully remote arrangements.\n## You might be a fit if you are…\n- An MLOps / ML platform engineer who has built and tuned LLM inference stacks (vLLM, SGLang, etc.)\n- A speech recognition or speech synthesis engineer/researcher who wants to shift toward inference systems\n- An engineer who has optimized throughput and cost for inference workloads on large GPU clusters\n## About the role\nAt Recho, we run training, evaluation, data generation and large-scale batch inference across multiple speech models and LLMs. Your core mission is to independently design and implement a high-efficiency inference framework that serves all of them.\nThis isn't about wrapping a model in an API or tweaking config files. You'll need to understand each model's architecture, inputs and outputs, sequence lengths, pre- and post-processing, and GPU characteristics. From there, you'll separate and integrate model execution, batching, data processing and scheduling into a reusable inference architecture.\nOn the LLM side, you'll build on mature engines such as vLLM and SGLang. You'll tune model adaptation, inference parameters, batching, prefill/decode behavior and request priority. Across both speech and LLMs, your north star is throughput and GPU cost.\n## What this role is *not*\n- Not an on-call or SRE role. SLA management, incident response and infrastructure maintenance for online services are not the focus.\n- GPU cluster operations and general DevOps are handled by a separate team.\n- Writing CUDA kernels or building an inference engine from scratch is not the main job.\n- RL training infrastructure belongs to our separate RL Infrastructure Engineer role. You'll partner with that team on the inference side.\n## What you'll do\n### 1. Build a common inference framework for speech models\n- Design a shared inference architecture for ASR, TTS, speech-to-speech and other audio generation models\n- Draw clear boundaries between shared and model-specific modules, with adapter/plugin interfaces for onboarding new models\n- Build the inference APIs, SDKs and tooling our model teams use every day\n### 2. Optimize large-scale batch inference for audio\n- Apply dynamic batching and bucketing for variable-length audio, and run data-parallel inference across GPUs and nodes\n- Build asynchronous pipelines that overlap preprocessing, inference, postprocessing and storage I/O\n- Measure and remove bottlenecks based on end-to-end processing time and cost\n### 3. Adapt and optimize LLM inference\n- Stand up LLM inference on mature engines such as vLLM and SGLang\n- Tune tensor/pipeline parallelism, continuous batching, KV cache and prefix cache\n- Design scheduling that accounts for prefill/decode characteristics and task priority\n### 4. Plan GPU usage for inference workloads\n- Handle GPU allocation and capacity planning for batch inference and evaluation\n- Orchestrate GPU inference jobs with Kubernetes, Slurm and Ray\n- Track and reduce GPU usage and inference cost per model and per task\nWe're happy to go into more detail during the intro call and interviews.\n## Why this role\n- **Set the playbook in an area that doesn't have one yet.** Inference for variable-length audio, codecs, vocoders, speech-to-speech and multimodal models has no established best practices. What you design becomes the foundation for every future model and team.\n- **Own inference end to end.** You'll cover both speech and LLMs, from framework design to batch inference and scheduling. That means you optimize the whole system, not just one piece of it.\n- **Directly move the business.** Inference efficiency determines GPU cost per unit of generated speech and response latency. Those decide how competitive our product is for the major enterprises running it in production.\n## Tech stack\n- **Languages & frameworks:** Python, PyTorch\n- **LLM inference:** vLLM, SGLang, TensorRT-LLM, Triton Inference Server\n- **Speech models:** ASR, TTS, speech-to-speech, audio codecs, vocoders\n- **Orchestration:** Kubernetes, Slurm, Ray\n- **Low-level optimization (nice to have):** C++, CUDA, Triton, NCCL\n## Our product\nWe build the Recho AI Voice Platform, which delivers voice conversations natural enough to be hard to tell apart from a human. We also deploy and operate Voice AI Agents for large enterprises.\nOur edge is a platform for building and improving high-quality agents quickly. It captures industry know-how and turns user feedback into a fast, continuous improvement loop. Our agents now handle complex, large-scale contact center operations in finance, manufacturing and the public sector. In some deployments, they've resolved every call without handing off to a human agent.\n## Hiring process\nIntro call → Case assignment & first interview → P...","description_format":"text","description_chars":18098,"description_truncated":true,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[{"language":"Japanese","level":"All levels","optional":false},{"language":"Chinese","level":"All levels","optional":true},{"language":"English","level":"All levels","optional":false}]},"benefits":["Annual leave","Apple Macbook","Stock options"],"hiring_locations":[{"name":"Japan","iso":"JP","kind":"country"}],"hiring_excludes":[],"relocation_offered":true,"industries":["Speech & Audio AI","AI Agents","Call Center Services"],"lifecycle":[{"event":"open","at":"2026-10-15T15:19:30Z"}],"visa":[],"liveness":{"score":63,"band":"ok","label":"Likely open","p_open":1,"p_active":0.632,"p_room":1,"age_days":5,"expected_fill_days":37,"reasons":["conf:2","stale_co","velocity","win:early","comp:attention"],"computed_at":"2026-10-21T05:46:15Z"},"pay":{"stated_usd_annual":126480,"is_top_pay":false},"html_url":"https://alion.io/job/recho-senior-ai-inference-engineer-llm-speech-models","json_url":"https://alion.io/job/recho-senior-ai-inference-engineer-llm-speech-models.json","meta":{"generated_at":"2026-10-22T05:46:35Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":320,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-23T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2675564},"rest":"https://alion.io/mcp/rest/get_company?id=2675564"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Frecho-senior-ai-inference-engineer-llm-speech-models"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Frecho-senior-ai-inference-engineer-llm-speech-models"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Frecho-senior-ai-inference-engineer-llm-speech-models"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/recho-senior-ai-inference-engineer-llm-speech-models\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Frecho-senior-ai-inference-engineer-llm-speech-models"}]}