AI Engineer

Responsibilities:

• Design and build evaluation systems for conversational AI across chat and voice.
• Define and track core metrics: task success, correctness, latency, conversation quality, etc.
• Analyse conversation logs to identify failure patterns and edge cases.
• Build failure taxonomies (intent errors, hallucinations, tool misuse, conversation breakdowns).
• Create automated eval pipelines (LLM-based + rule-based + dataset-driven).
• Set up regression test suites for prompts, workflows, and agent behaviour.
• Run experiments and A/B tests across prompts, models, and agent configurations.
• Work closely with Applied AI and product teams to translate insights into system improvements.
• Build internal tooling and dashboards to monitor AI performance at scale.

Requirements:

• 5+ years of experience in backend engineering, data systems, or ML-adjacent roles.
• Experience working with data pipelines, logs, or analytics systems.
• Strong understanding of metrics, experimentation, and system behaviour.
• Ability to debug complex systems and trace issues across multiple components.
• Comfort working with ambiguity and defining structure where none exists.

Additional experience (not mandatory):

• Experience with LLMs, NLP systems, or conversational AI.
• Familiarity with prompt design or agent workflows.
• Experience with evaluation frameworks, test systems, or QA automation.
• Exposure to speech systems (ASR/TTS) or voice products.
• Experience in high-scale or fast-moving startup environments.

Good to Have:

• You think in terms of systems and feedback loops, not just outputs.
• You care about measuring quality, not guessing it.
• You naturally break down failures into patterns and root causes.
• You are comfortable working with messy real-world data (especially conversations).
• You've built systems that helped teams improve continuously, not just once.