AI - QA Engineer

Overview

Mission

Own release readiness for agents whose outputs go into client-facing RFPs, data grids and pitch decks. Test that answers are grounded and correctly cited, that users never see content they are not entitled to, that multi-agent handoffs behave, and that generated files are correct. Then turn this into automated gates that every agent must pass.

Responsibilities

Key Responsibilities

• Define quality standards and release gates per agent (groundedness, citation accuracy, hallucination, task completion, latency, cost) aligned to the PRD's decision-grade vs good-enough split.
• Build golden datasets with Global BD SMEs from real RFPs, grids and prior submissions; run UAT cycles with named business testers.
• Test citations by checking every claim links to a real, permission-appropriate source, and that gaps and contradictions are flagged rather than invented.
• Test permissions using personas with different access to prove security trimming holds across retrieval and live tool calls.
• Test agent behaviour including tool selection, execution correctness, failure handling and multi-agent handoffs / downstream triggers.
• Test document outputs for grid cell accuracy, template fidelity, and tracked-change style proofing suggestions rather than silent edits.
• Run adversarial tests for prompt injection (including via uploaded RFP files), data leakage and safety boundaries.
• Automate regression for model, prompt, embedding and retrieval changes inside the CI/CD pipeline.

Qualifications

Experience & Working Style

• 5-8 years in QA / test engineering, with 2+ years testing LLM, RAG or agent systems.
• Has built an automated LLM evaluation suite that gated real releases.
• Can sit with business reviewers, capture why an output is wrong and turn it into a repeatable test.

Not required (don't screen out for these): Traditional manual-only QA backgrounds are not sufficient. Deep ML modelling knowledge is not required.