QA Automation Engineer
First seen by Alion on Sep 21, 2026.
QA Automation Engineer - GenAI / Agentic AI
We are looking for a QA Automation Engineer with 2+ years of relevant experience in test automation, including hands-on experience testing Generative AI and Agentic AI applications.
Key Responsibilities :
- Design, develop, and maintain automated test frameworks using Playwright or similar tools.
- Build automated UI, API, integration, and end-to-end test suites for GenAI and Agentic AI applications.
- Validate LLM responses, RAG workflows, agent/tool interactions, multi-step agentic workflows, and failure scenarios.
- Test AI applications for accuracy, consistency, hallucinations, response quality, latency, reliability, and regression.
- Automate testing of RAG pipelines, including retrieval quality, context relevance, citations, and response grounding.
- Validate multi-agent workflows, tool calling, retries, state transitions, guardrails, and human-in-the-loop scenarios.
- Integrate automated tests into CI/CD pipelines and support continuous quality monitoring.
Required Skills :
- 2+ years of relevant QA automation experience with exposure to GenAI / Agentic AI applications.
- Strong hands-on experience with Playwright and modern automation frameworks.
- Experience with API testing, UI automation, integration testing, and end-to-end testing.
- Good understanding of Generative AI, RAG, LLMs, Agentic AI, and multi-agent workflows.
- Experience testing non-deterministic AI outputs using appropriate evaluation and validation approaches.
- Strong programming skills in Python, JavaScript, or TypeScript.
- Experience with CI/CD, Git, test reporting, defect tracking, and debugging.
- Understanding of LLM evaluation, prompt testing, guardrails, performance testing, and AI application observability is preferred.
Skills
Python, TypeScript, Automation Testing, Artificial Intelligence, API Testing, Playwright Testing, Testing, Software Quality Assurance, AI/ML Testing