GenAI Test Lead

Job description:

We are seeking an experienced AI Test Lead to lead end-to-end testing and quality assurance of Generative AI applications. This role bridges AI quality engineering, test automation, and customer engagement. You will collaborate with QA Managers, AI/ML Engineers, Prompt Engineers, Full Stack Developers, and DevOps Engineers to validate production-ready, scalable AI applications against industry AI testing standards, while serving as the primary testing point of contact for customers.

Key Responsibilities:

• Define and execute AI testing strategy, planning, execution, defect management, and release sign-off.
• Lead functional, automation, API, integration, performance, and AI-specific testing for GenAI applications.
• Design test approaches for LLM evaluation, prompt regression, hallucination detection, bias and fairness testing, robustness validation, and output quality assessment.
• Manage sprint-level test planning, estimation, quality metrics, and release readiness in Agile environments.
• Act as the primary QA contact for customers, driving requirement discussions, status reviews, defect triage, and UAT support.
• Build and enhance Python-based automation frameworks using Pytest, Playwright, Selenium, and API automation tools.
• Implement automated AI evaluation pipelines using tools such as DeepEval, Ragas, LangSmith, or promptfoo.
• Integrate test automation into CI/CD pipelines and establish quality governance, documentation, and reporting standards.
• Mentor QA engineers and promote best practices in AI testing and automation.

Required skills:

• 8–15+ years in software testing and QA, with experience leading testing teams.
• Hands-on experience in AI/GenAI testing, LLM validation, or intelligent application testing.
• Strong Python programming and test automation expertise.
• Experience with API testing, automation frameworks, and CI/CD pipelines.
• Strong customer communication, stakeholder management, and delivery leadership skills.
• Experience with Agile/Scrum methodologies and test management tools.

Preferred qualifications:

• Experience with RAG systems, conversational AI, AI agents, or LLM-based applications.
• Familiarity with AI governance frameworks such as ISO/IEC 42001, NIST AI RMF, or EU AI Act.
• Knowledge of AI quality metrics including accuracy, relevance, groundedness, consistency, and toxicity evaluation.

Skills

API, Gen AI, Generative AI, LLM, Playwright, Python, QA, Quality Assurance, Selenium, Testing, pytest