Forward Deployed AI / ML Engineer

Forward Deployed AI / ML Engineer — Evaluations (Product + Engineering)

Website: ai.joshtalks.com and voicearena.com

Location: Gurugram

About us

Josh Talks AI builds and licenses audio datasets to leading AI labs. We've been deliberately quiet for the last year and just built. The product has earned the right to be expanded; we are now hiring the people who will own what gets built next.

About the role

This is an Evaluations PM + Engineer role. You will own how we measure the quality of our speech, voice, and image data, and of our customers models: ASR, TTS, speech-to-speech (S2S), and voice agents. You will define what good means for each dataset and model, build and run the evaluations yourself, and turn the results into clear decisions for the product, research, and business teams.

You don't hand evaluations off. You plan them, run them end to end, and own the outcome.

What you'll do

Evaluation product ownership

• Own the evaluation roadmap across ASR, TTS, S2S, and voice-agent models.
• Work with research, product, and business teams to understand use cases and define what quality means for each one.
• Turn quality goals into clear evaluation plans: test sets, metrics, pass/fail thresholds, and timelines.
• Prioritise which models, languages, and use cases to evaluate first, based on impact.
• Share results through clear reports and dashboards, and make go/no-go recommendations on model releases.
• Track model quality over time and flag regressions early.

Hands-on evaluation (done by you)

• Run end-to-end evaluations of ASR, TTS, S2S, and voice-agent models yourself.
• Set up and run open-source and in-house models locally and on cloud GPUs for inference.
• Integrate and run models from different vendors and customers through their APIs (batch and streaming), and compare them fairly.
• Manage cloud GPU instances (GCP/AWS/Azure) to run large-scale evaluation and inference jobs efficiently.
• Build and curate evaluation datasets, including multilingual, Indian-language, noisy, and real-world audio.
• Design and run human evaluations: listening tests, rating guidelines, annotator training, and agreement checks.
• Write reproducible evaluation pipelines in Python and automate recurring benchmarks.
• Analyse errors to find failure patterns and feed them back to the research and data teams.
• Benchmark in-house models against third-party and open-source models.

What we're looking for

Skills

• Hands-on evaluation of ASR, TTS, S2S, or voice/LLM agent models.
• Strong Python, with Pandas/NumPy, Hugging Face, and audio libraries (librosa, torchaudio).
• Running ML model inference (PyTorch / Hugging Face) on GPUs.
• Working with third-party model APIs (REST, WebSocket/streaming).
• Cloud platforms (GCP, AWS, or Azure), including GPU instances.
• Evaluation metrics, test-set design, and statistical significance.
• Designing human evaluation studies and rating guidelines.
• Clear written and verbal communication, and explaining results to technical and non-technical people.
• Strong ownership, and the ability to work independently in a fast-moving team.

Nice to have

• Experience fine-tuning speech models (ASR, TTS, or S2S).
• Experience with Indian languages, code-mixed, or low-resource speech data.
• Familiarity with Whisper, NeMo, pyannote, ESPnet, or commercial speech APIs.
• Experience evaluating LLMs or voice agents (ASR → LLM → TTS pipelines).
• Experience building evaluation dashboards or automated regression testing.

Why join us

• Own how quality is defined and measured for speech AI used at scale.
• Work closely with research and product, and directly shape which models ship.
• High ownership and real impact from day one.