AI QA Engineer

Hiring: Senior Generative AI QA Engineer

Job Title: Senior Generative AI QA Engineer

Experience: 8+ Years

Job Summary

We are looking for an experienced Senior Generative AI QA Engineer with 8+ years of overall experience in software testing and quality engineering, including hands-on experience in testing Generative AI applications, Large Language Models (LLMs), RAG-based systems, and AI-powered solutions.

The ideal candidate should have strong expertise in test automation, Python, API testing, and AI evaluation methodologies. The candidate will be responsible for ensuring the accuracy, reliability, performance, security, and scalability of enterprise GenAI applications.

Key Responsibilities

• Lead QA strategy, test planning, and quality engineering activities for Generative AI and LLM-based applications.
• Design and implement automated testing frameworks using Python.
• Perform functional, integration, regression, API, performance, and end-to-end testing.
• Validate LLM-generated responses for accuracy, relevance, consistency, and contextual understanding.
• Test RAG pipelines, including document ingestion, chunking, embeddings, vector search, retrieval accuracy, and response generation.
• Identify and evaluate AI hallucinations, bias, toxicity, and inconsistent model responses.
• Develop evaluation datasets, test benchmarks, and automated validation pipelines for LLM applications.
• Perform prompt testing, prompt regression testing, and adversarial testing across multiple LLM models.
• Validate AI agents, tool calling, multi-agent workflows, and API integrations.
• Evaluate LLM performance using metrics such as faithfulness, answer relevance, context precision, and context recall.
• Conduct AI security testing, including prompt injection, jailbreak attacks, data leakage, and sensitive information exposure.
• Perform load, stress, latency, and scalability testing for AI applications.
• Integrate automated test suites into CI/CD pipelines.
• Collaborate with AI/ML engineers, developers, architects, and product teams to improve application quality.
• Mentor QA engineers and drive best practices in AI testing and automation.

Mandatory Technical Skills

• 8+ years of overall experience in Software Testing / Quality Engineering.
• Strong hands-on experience in Python-based test automation.
• Experience testing Generative AI applications and LLM-based solutions.
• Strong knowledge of LLM concepts, Prompt Engineering, and AI response evaluation.
• Hands-on experience with API testing using Postman, REST Assured, or Python Requests.
• Experience with RAG architecture, embeddings, vector databases, and document retrieval.
• Understanding of hallucination detection, response validation, and contextual accuracy.
• Experience in functional, integration, regression, and performance testing.
• Experience designing and maintaining scalable automation frameworks.

Preferred Technical Skills

• Playwright, Selenium, Pytest, or Robot Framework.
• RAGAS, DeepEval, TruLens, LangSmith, or Promptfoo.
• OpenAI, Azure OpenAI, Anthropic Claude, Gemini, and Llama.
• LangChain, LangGraph, CrewAI, or other Agentic AI frameworks.
• AI security testing and OWASP Top 10 for LLM Applications.
• Jenkins, GitHub Actions, Azure DevOps, and CI/CD.
• AWS, Azure, or GCP.