Senior QA Engineer (AI)

We are looking for a Senior AI QA Engineer to lead quality assurance, evaluation, and validation of AI-driven systems, including LLM-based applications, agentic workflows, and intelligent platforms. This role focuses on ensuring reliability, accuracy, safety, and performance of AI solutions in production environments. This role goes beyond traditional QA; it requires expertise in AI testing strategies, evaluation frameworks, and continuous validation of probabilistic systems, ensuring robust and trustworthy AI deployments.

Responsibilities

• Owned end-to-end quality and evaluation strategy for AI-driven systems.
• Design and implement AI evaluation frameworks, test harnesses, and validation pipelines.
• Ensure accuracy, reliability, and safety of LLM outputs and agentic workflows.
• Collaborate closely with AI engineers, product teams, and full-stack developers.
• Work within POD-based delivery with clear ownership of QA and evaluation outcomes.
• Drive continuous improvement of model performance, test coverage, and quality metrics.
• Balances speed of delivery with robustness and governance requirements.
• Design and implement evaluation frameworks for LLMs, RAG systems, and AI agents.
• Develop and maintain golden datasets, benchmarks, and test corpora.
• Build automated test harnesses for functional, semantic, and performance validation.
• Evaluates search relevance, embeddings' quality, and retrieval accuracy.
• Validate AI outputs for correctness, consistency, hallucination, and bias.
• Define and track metrics (precision, recall, relevance, latency, and pass/fail criteria).
• Work on LLM assurance pipelines and evaluation coverage expansion.
• Collaborate with AI engineers on prompt tuning and model optimisation.
• Implement A/B testing frameworks and experimentation strategies.
• Test multi-agent workflows, orchestration logic, and integration layers.
• Ensure compliance with security, governance, and guardrails for AI systems.
• Build and maintain automation for regression testing of AI systems.
• Contribute to observability by defining monitoring, logging, and alerting for AI behaviour.
• Participate in design reviews to ensure testability and quality-by-design.
• Support production validation, issue triage, and continuous improvement cycles.

Requirements:

• 6-10 years in QA / testing / validation roles, with 2+ years in AI/ML or LLM testing.
• Experience validating enterprise applications or AI-driven platforms.

AI Evaluation and Testing:

• Strong understanding of LLM evaluation techniques and metrics.
• Experience with RAG validation, embedding evaluation, and search relevance testing.
• Ability to design golden datasets, synthetic data, and benchmarking strategies.
• Familiarity with prompt evaluation, hallucination detection, and bias analysis.

Automation and Engineering:

• Experience in test automation frameworks and scripting (Python preferred).
• Ability to build evaluation pipelines and automated validation workflows.
• Knowledge of API testing, integration testing, and system testing.

Data and Systems:

• Experience withdatabases, vector stores, and data pipelines.
• Understanding of logs, telemetry, and observability for AI systems.

Cloud and DevOps:

• Experience with AWS / Azure / GCP environments.
• Familiarity with CI/CD pipelines and automated test integration.
• Exposure to containerised environments (Docker/Kubernetes).

AI Assurance and Governance:

• Understanding of AI safety, guardrails, and governance frameworks.
• Ability to define and enforce quality standards for AI-assisted outputs.
• Experience in evaluation-driven development and continuous testing loops.

Additional Capabilities:

• Strong analytical and problem-solving skills.
• Ability to work in agile POD-based delivery models.
• Strong collaboration with AI, engineering, and product teams.
• Attention to detail with a focus on accuracy and reliability in AI systems.