Senior AI Inference Engineer (LLM & Speech Models)
Confirmed on the employer's own hiring board on Oct 20, 2026. First seen by Alion on Oct 15, 2026. Recho scores C on the Alion truth index.
Senior AI Inference Engineer (LLM & Speech Models)
Description
## In short
You'll design and build Recho's inference framework for speech models and LLMs, and you'll own its throughput and GPU cost. This is a hands-on senior role: you'll architect the system and write much of the code yourself.
- **Team language:** English (Japanese not required)
- **Salary:** ¥10M-¥20M per year, plus stock options
- **Location:** Tokyo (Hanzomon), hybrid. Candidates based outside Japan are welcome. We offer visa sponsorship and relocation support, and we're open to discussing fully remote arrangements.
## You might be a fit if you are…
- An MLOps / ML platform engineer who has built and tuned LLM inference stacks (vLLM, SGLang, etc.)
- A speech recognition or speech synthesis engineer/researcher who wants to shift toward inference systems
- An engineer who has optimized throughput and cost for inference workloads on large GPU clusters
## About the role
At Recho, we run training, evaluation, data generation and large-scale batch inference across multiple speech models and LLMs. Your core mission is to independently design and implement a high-efficiency inference framework that serves all of them.
This isn't about wrapping a model in an API or tweaking config files. You'll need to understand each model's architecture, inputs and outputs, sequence lengths, pre- and post-processing, and GPU characteristics. From there, you'll separate and integrate model execution, batching, data processing and scheduling into a reusable inference architecture.
On the LLM side, you'll build on mature engines such as vLLM and SGLang. You'll tune model adaptation, inference parameters, batching, prefill/decode behavior and request priority. Across both speech and LLMs, your north star is throughput and GPU cost.
## What this role is *not*
- Not an on-call or SRE role. SLA management, incident response and infrastructure maintenance for online services are not the focus.
- GPU cluster operations and general DevOps are handled by a separate team.
- Writing CUDA kernels or building an inference engine from scratch is not the main job.
- RL training infrastructure belongs to our separate RL Infrastructure Engineer role. You'll partner with that team on the inference side.
## What you'll do
### 1. Build a common inference framework for speech models
- Design a shared inference architecture for ASR, TTS, speech-to-speech and other audio generation models
- Draw clear boundaries between shared and model-specific modules, with adapter/plugin interfaces for onboarding new models
- Build the inference APIs, SDKs and tooling our model teams use every day
### 2. Optimize large-scale batch inference for audio
- Apply dynamic batching and bucketing for variable-length audio, and run data-parallel inference across GPUs and nodes
- Build asynchronous pipelines that overlap preprocessing, inference, postprocessing and storage I/O
- Measure and remove bottlenecks based on end-to-end processing time and cost
### 3. Adapt and optimize LLM inference
- Stand up LLM inference on mature engines such as vLLM and SGLang
- Tune tensor/pipeline parallelism, continuous batching, KV cache and prefix cache
- Design scheduling that accounts for prefill/decode characteristics and task priority
### 4. Plan GPU usage for inference workloads
- Handle GPU allocation and capacity planning for batch inference and evaluation
- Orchestrate GPU inference jobs with Kubernetes, Slurm and Ray
- Track and reduce GPU usage and inference cost per model and per task
We're happy to go into more detail during the intro call and interviews.
## Why this role
- **Set the playbook in an area that doesn't have one yet.** Inference for variable-length audio, codecs, vocoders, speech-to-speech and multimodal models has no established best practices. What you design becomes the foundation for every future model and team.
- **Own inference end to end.** You'll cover both speech and LLMs, from framework design to batch inference and scheduling. That means you optimize the whole system, not just one piece of it.
- **Directly move the business.** Inference efficiency determines GPU cost per unit of generated speech and response latency. Those decide how competitive our product is for the major enterprises running it in production.
## Tech stack
- **Languages & frameworks:** Python, PyTorch
- **LLM inference:** vLLM, SGLang, TensorRT-LLM, Triton Inference Server
- **Speech models:** ASR, TTS, speech-to-speech, audio codecs, vocoders
- **Orchestration:** Kubernetes, Slurm, Ray
- **Low-level optimization (nice to have):** C++, CUDA, Triton, NCCL
## Our product
We build the Recho AI Voice Platform, which delivers voice conversations natural enough to be hard to tell apart from a human. We also deploy and operate Voice AI Agents for large enterprises.
Our edge is a platform for building and improving high-quality agents quickly. It captures industry know-how and turns user feedback into a fast, continuous improvement loop. Our agents now handle complex, large-scale contact center operations in finance, manufacturing and the public sector. In some deployments, they've resolved every call without handing off to a human agent.
## Hiring process
Intro call → Case assignment & first interview → Paid work trial* → Final interview → Offer meeting
*The work trial is 1-2 consecutive days at our Tokyo office, working on real tasks alongside the team, and it's paid. It's a chance for both sides to check fit and get a feel for the work.
- **If you're based outside Japan,** we'll agree on the format together. We won't ask you to travel to Japan just for this step.
- If your schedule or your current employer's side-work policy makes the trial difficult, just let us know.
- We may also ask for references.
## About Recho
Recho develops foundation models for customer-support AI agents and also deploys them for clients, all within one organization. Our core strength is voice AI technology we built entirely in-house, which resolves the industry's long-standing trade-off between quality and cost.
We've closed a Series A of approximately ¥700M and are rapidly rolling out to demanding production environments at major companies such as SBI Shinsei Bank and KDDI.
## Learn more
- [Corporate website](https://recho-ai.com/)
- [Team interviews (note, in Japanese)](https://note.com/recho_ai)
- [Company deck](https://recruit.recho-ai.com/)
- [Case study: Partnership with KDDI and Altius Link (Japanese)](https://prtimes.jp/main/html/rd/p/000000021.000159800.html)
- [Case study: SBI Shinsei Bank (Japanese)](https://mainichi.jp/articles/20260130/k00/00m/020/275000c)
- [2-minute voice demo](https://recruit.recho-ai.com/#2f276fb799f480d5bba7d5784d6ad31e) - give it a listen!
Requirements
## In short
You'll design and build Recho's inference framework for speech models and LLMs, and you'll own its throughput and GPU cost. This is a hands-on senior role: you'll architect the system and write much of the code yourself.
- **Team language:** English (Japanese not required)
- **Salary:** ¥10M-¥20M per year, plus stock options
- **Location:** Tokyo (Hanzomon), hybrid. Candidates based outside Japan are welcome. We offer visa sponsorship and relocation support, and we're open to discussing fully remote arrangements.
## You might be a fit if you are…
- An MLOps / ML platform engineer who has built and tuned LLM inference stacks (vLLM, SGLang, etc.)
- A speech recognition or speech synthesis engineer/researcher who wants to shift toward inference systems
- An engineer who has optimized throughput and cost for inference workloads on large GPU clusters
## About the role
At Recho, we run training, evaluation, data generation and large-scale batch inference across multiple speech models and LLMs. Your core mission is to independently design and implement a high-efficiency inference framework that serves all of them.
This isn't about wrapping a model in an API or tweaking config files. You'll need to understand each model's architecture, inputs and outputs, sequence lengths, pre- and post-processing, and GPU characteristics. From there, you'll separate and integrate model execution, batching, data processing and scheduling into a reusable inference architecture.
On the LLM side, you'll build on mature engines such as vLLM and SGLang. You'll tune model adaptation, inference parameters, batching, prefill/decode behavior and request priority. Across both speech and LLMs, your north star is throughput and GPU cost.
## What this role is *not*
- Not an on-call or SRE role. SLA management, incident response and infrastructure maintenance for online services are not the focus.
- GPU cluster operations and general DevOps are handled by a separate team.
- Writing CUDA kernels or building an inference engine from scratch is not the main job.
- RL training infrastructure belongs to our separate RL Infrastructure Engineer role. You'll partner with that team on the inference side.
## What you'll do
### 1. Build a common inference framework for speech models
- Design a shared inference architecture for ASR, TTS, speech-to-speech and other audio generation models
- Draw clear boundaries between shared and model-specific modules, with adapter/plugin interfaces for onboarding new models
- Build the inference APIs, SDKs and tooling our model teams use every day
### 2. Optimize large-scale batch inference for audio
- Apply dynamic batching and bucketing for variable-length audio, and run data-parallel inference across GPUs and nodes
- Build asynchronous pipelines that overlap preprocessing, inference, postprocessing and storage I/O
- Measure and remove bottlenecks based on end-to-end processing time and cost
### 3. Adapt and optimize LLM inference
- Stand up LLM inference on mature engines such as vLLM and SGLang
- Tune tensor/pipeline parallelism, continuous batching, KV cache and prefix cache
- Design scheduling that accounts for prefill/decode characteristics and task priority
### 4. Plan GPU usage for inference workloads
- Handle GPU allocation and capacity planning for batch inference and evaluation
- Orchestrate GPU inference jobs with Kubernetes, Slurm and Ray
- Track and reduce GPU usage and inference cost per model and per task
We're happy to go into more detail during the intro call and interviews.
## Why this role
- **Set the playbook in an area that doesn't have one yet.** Inference for variable-length audio, codecs, vocoders, speech-to-speech and multimodal models has no established best practices. What you design becomes the foundation for every future model and team.
- **Own inference end to end.** You'll cover both speech and LLMs, from framework design to batch inference and scheduling. That means you optimize the whole system, not just one piece of it.
- **Directly move the business.** Inference efficiency determines GPU cost per unit of generated speech and response latency. Those decide how competitive our product is for the major enterprises running it in production.
## Tech stack
- **Languages & frameworks:** Python, PyTorch
- **LLM inference:** vLLM, SGLang, TensorRT-LLM, Triton Inference Server
- **Speech models:** ASR, TTS, speech-to-speech, audio codecs, vocoders
- **Orchestration:** Kubernetes, Slurm, Ray
- **Low-level optimization (nice to have):** C++, CUDA, Triton, NCCL
## Our product
We build the Recho AI Voice Platform, which delivers voice conversations natural enough to be hard to tell apart from a human. We also deploy and operate Voice AI Agents for large enterprises.
Our edge is a platform for building and improving high-quality agents quickly. It captures industry know-how and turns user feedback into a fast, continuous improvement loop. Our agents now handle complex, large-scale contact center operations in finance, manufacturing and the public sector. In some deployments, they've resolved every call without handing off to a human agent.
## Hiring process
Intro call → Case assignment & first interview → Paid work trial* → Final interview → Offer meeting
*The work trial is 1-2 consecutive days at our Tokyo office, working on real tasks alongside the team, and it's paid. It's a chance for both sides to check fit and get a feel for the work.
- **If you're based outside Japan,** we'll agree on the format together. We won't ask you to travel to Japan just for this step.
- If your schedule or your current employer's side-work policy makes the trial difficult, just let us know.
- We may also ask for references.
## About Recho
Recho develops foundation models for customer-support AI agents and also deploys them for clients, all within one organization. Our core strength is voice AI technology we built entirely in-house, which resolves the industry's long-standing trade-off between quality and cost.
We've closed a Series A of approximately ¥700M and are rapidly rolling out to demanding production environments at major companies such as SBI Shinsei Bank and KDDI.
## Learn more
- [Corporate website](https://recho-ai.com/)
- [Team interviews (note, in Japanese)](https://note.com/recho_ai)
- [Company deck](https://recruit.recho-ai.com/)
- [Case study: Partnership with KDDI and Altius Link (Japanese)](https://prtimes.jp/main/html/rd/p/000000021.000159800.html)
- [Case study: SBI Shinsei Bank (Japanese)](https://mainichi.jp/articles/20260130/k00/00m/020/275000c)
- [2-minute voice demo](https://recruit.recho-ai.com/#2f276fb799f480d5bba7d5784d6ad31e) - give it a listen!
Preferred Experiences
## Nice-to-haves
**Advanced models & speech**
- Inference experience with audio generation models (diffusion, flow matching) or audio codecs
- Hands-on experience across multiple model types, such as ASR, TTS and speech-to-speech
- Interest or experience in building and optimizing RL rollout inference environments
**Performance & low-level optimization**
- Designing dynamic batching for variable-length audio, or distributed/parallel inference across multiple GPUs and nodes
- Working knowledge of C++, CUDA, Triton and NCCL, and using profilers to pinpoint bottlenecks
- Managing and optimizing GPU cost and compute budgets
**Platform design & technical leadership**
- Designing internal AI frameworks, inference libraries or model SDKs
- Open-source contributions to vLLM, SGLang, PyTorch, Ray or similar
- Setting technical roadmaps, design standards or code review guidelines
**Languages**
- Japanese or Chinese. Our team works in English, but Japanese (the company's working language) or Chinese (spoken by many of our colleagues) makes cross-team collaboration even smoother.
We are looking for
## What we look for
- **Hands-on:** You don't stop at architecture. You're on the front line writing the code and care about getting the implementation right.
- **Data-driven:** You tackle performance and cost problems with profilers and benchmarks, not gut feel, and you optimize for the whole system.
- **Pragmatic:** You don't reinvent the wheel. You pick the right open-source tools and frameworks and extend only what we actually need.
- **Platform-minded:** You enjoy building tools and libraries that make other teams, especially model developers, more productive. That's Team-First in practice.
## Our values
These are the values our team relies on to deliver Recho's value as fast as possible.
- **Exceed Expectations:** We keep exceeding the expectations of customers and teammates by delivering both speed and quality.
- **Team-First:** Each of us takes professional ownership, and we maximize the team's results, not individual ones.
- **Continuous Feedback:** We engage seriously and speak candidly for the sake of results. The harder something is to say, the more we say it with honesty, respect and care.
- **Earn Trust:** Trust is the foundation of every collaboration and every challenge. We act ethically and keep personal and company matters separate. Trust takes time to build and a moment to lose.
Working Conditions
Salary
**Salary**
Compensation is based on your current compensation, skills and experience.
- Annual salary: ¥10M-¥20M
- Salary review: Once a year, based on your contribution and company performance
- Stock option plan
Location
WeWork Hanzomon (Hanzomon PREX North, 2-3-2 Kojimachi, Chiyoda-ku, Tokyo)
Hybrid: mostly in the office, with some remote work. For candidates based outside Japan, remote arrangements are open to discussion.
Job Type
Employment type: Full-time, permanent. Starting with a paid contract engagement first is also an option.
Work hours
**Working hours**
- Discretionary work system (professional type under Japanese labor law)
- Deemed working hours: 9 hours/day
- Start and end times are up to you. As a general guideline, the team's core working window is 10:00-19:00, adjusted as work requires.
**Scope of duties and work location**
This disclosure is required under Japanese labor law.
- Duties: AI inference infrastructure development at hire; subject to change as determined by the company
- Work location: Hanzomon WeWork at hire (some remote work possible); subject to change as determined by the company
**Holidays & leave**
Weekends, Japanese public holidays, year-end/New Year holidays (Dec 29-Jan 3), special leave, and paid annual leave (10 days in the first year)
Probation period
Probation period: 3 months, with no change in terms
Benefits
## Benefits
**Insurance**
- Full Japanese social insurance (health, pension, employment)
- Commuting allowance (up to ¥25,000/month)
**Office**
- WeWork Hanzomon, 2 minutes from Hanzomon Station, with a dedicated floor, focus booths and meeting rooms
- Free drinks, including coffee, soft drinks and alcohol
- High-quality office chairs
- MacBook Air and external monitor provided
**Productivity**
- Dev tools such as Claude Code, Cursor and Devin
- AI tool stipend (up to ¥30,000/month)
- Amazon Business account for work-related books and equipment
**Team**
- Daily lunch meeting covering work updates and casual conversation
- Monthly team social on the first Friday evening, at the office with food
**Smoking**
- Indoor smoking prohibited; outdoor smoking area available