Backend / ML-Ops Engineer — Speech Model Deployment & Inference Optimization

Own the infrastructure and pipelines for integrating trained ASR/TTS/Speech-LLM models into production. Focus on scalable serving, GPU optimization, monitoring, and continuous improvement of inference latency and reliability.

What You’ll Do

• Containerize and deploy speech models using Triton Inference Server with TensorRT/FP16 optimizations.
• Develop and manage CI/CD pipelines for model promotion (staging → production).
• Configure autoscaling on Kubernetes (GPU pools) based on active calls or streaming sessions.
• Build health and observability dashboards: latency, token delay, WER drift, SNR/packet loss monitors.
• Integrate LM bias APIs, failover logic, and model switchers for fallback to larger/cloud models.
• Implement on-device or edge inference paths for low-latency scenarios.
• Collaborate with AI team to expose APIs for context biasing, rescoring, and diagnostics.
• Optimize GPU/CPU utilization, cost optimization, and memory footprint for concurrent ASR/TTS/Speech LLM workloads.
• Maintain data and model versioning pipelines with MLflow, DVC, or internal registries.

Desired Skills

• Experience with Triton, TensorRT, Docker, Kubernetes, and GPU scheduling.
• Familiarity with speech inference (streaming ASR, TTS pipelines).
• Proficient in Python, Bash, and cloud services (AWS/GCP/Azure).
• Understanding of observability stacks (Prometheus, Grafana, ELK).
• Knowledge of DevSecOps, access policies, and PHI-safe environments.
• Interest in inference optimization, mixed precision, and quantization.

Qualifications

• B.Tech / M.Tech in Computer Science or related field.
• 4-6 years of experience in backend or ML-ops; at least 1-2 years with GPU inference pipelines.
• Proven experience deploying models to production environments with measurable latency gains.