AI Quality Engineer

TECHJAYS

THE AI REIMAGINATION COMPANY

AI Quality Engineer

Position : AI Quality Engineer

Experience : 3–10 Years

Location : Coimbatore / Hybrid

Joining : Immediate / Within 30 Days

Primary SkillsManual & Functional Testing, AI/ML Application Testing, LLM Testing, Prompt Testing, Test Case Design, API Testing, Regression Testing, Defect Management

Preferred SkillsRAG Testing, Chatbot Testing, AI Response Evaluation, Selenium, Playwright, Cypress, Python, JavaScript, SQL, Postman, JIRA, BDD

About Techjays

At Techjays, we're helping businesses reimagine how they build, operate, and scale with AI at the core. We focus on developing intelligent, scalable, and production-grade AI solutions that solve real-world business challenges.

Our global team brings together experienced technology professionals from organizations such as Google, NetApp, ADP, Cognizant Consulting, and Capgemini. We work with global clients to build AI-powered applications, intelligent automation platforms, and enterprise software solutions.

We're looking for an AI Quality Engineer with strong functional testing expertise and prior hands-on experience testing AI/ML-powered applications.

This role goes beyond traditional software testing. You will be responsible for validating the quality, accuracy, reliability, and behaviour of AI-powered features, ensuring that intelligent applications perform consistently across different scenarios and real-world use cases.

The ideal candidate should understand both traditional QA methodologies and the unique testing challenges associated with AI systems, including non-deterministic outputs, prompt behaviour, hallucinations, response relevance, and edge-case validation.

Prior hands-on experience testing AI/ML-based products or applications is mandatory. Candidates with only traditional manual testing experience and no AI project exposure will not be a fit.

Key Responsibilities

1. AI Application & LLM Testing

• Perform functional and quality testing of AI-powered applications, LLM-based products, chatbots, and intelligent automation systems.
• Design test scenarios to validate AI-generated responses for accuracy, relevance, consistency, completeness, and contextual understanding.
• Test prompts, prompt variations, and conversational flows to identify unexpected outputs and behavioural inconsistencies.
• Evaluate AI responses for hallucinations, incorrect information, irrelevant answers, and context retention issues.
• Validate AI application behaviour across different user inputs, edge cases, and complex scenarios.
• Test RAG-based applications by validating document retrieval, response relevance, contextual accuracy, and source grounding.
• Evaluate AI-powered search, recommendation engines, and other intelligent product features.
• Collaborate with AI engineers and data science teams to define acceptance criteria and quality benchmarks.

2. Functional & Manual Testing

• Analyse business and technical requirements and convert them into comprehensive test scenarios and test cases.
• Execute functional, regression, integration, system, and user acceptance testing across web and mobile applications.
• Validate end-to-end application workflows, business logic, user interactions, and data processing.
• Identify functional gaps, unexpected behaviour, and usability issues.
• Perform exploratory testing to uncover defects that may not be covered by predefined test cases.
• Ensure application functionality remains stable across different releases and environments.
• Review requirements early in the development lifecycle to identify potential quality risks.

3. AI Quality Evaluation & Validation

• Develop structured testing approaches for non-deterministic AI outputs using behaviour-based, range-based, and scenario-based validation.
• Evaluate AI response quality against expected outcomes and predefined acceptance criteria.
• Validate prompt sensitivity, context handling, conversation memory, and multi-turn interactions.
• Identify failure patterns involving hallucinations, prompt injection, incorrect retrieval, and inconsistent responses.
• Test AI system performance across different input formats, languages, and user scenarios where applicable.
• Maintain test datasets and evaluation scenarios for repeatable AI quality assessment.
• Work with engineering teams to identify root causes of AI-related defects and verify fixes.

4. API & Integration Testing

• Perform manual API testing using Postman or similar tools.
• Validate API request and response structures, status codes, authentication, authorization, and error handling.
• Test integrations between AI services, backend systems, databases, and external APIs.
• Validate data flow between application components and identify integration issues.
• Test API behaviour under different input conditions, including invalid inputs and boundary cases.
• Verify API responses and ensure consistency between frontend and backend functionality.

5. Defect Management & Quality Reporting

• Identify, document, track, and manage defects throughout the testing lifecycle.
• Maintain detailed test cases, test execution reports, defect logs, and test documentation.
• Collaborate with developers to reproduce issues, analyse root causes, and validate fixes.
• Track test coverage, defect trends, and application quality metrics.
• Communicate release risks, unresolved defects, and testing status to relevant stakeholders.
• Participate in release readiness reviews and provide quality sign-off for assigned modules.

6. Automation & Continuous Improvement

• Identify repetitive test scenarios that can be automated to improve testing efficiency.
• Support automation initiatives using Selenium, Playwright, or Cypress.
• Work with developers to improve testability and integrate testing practices into CI/CD workflows.
• Explore AI-assisted testing tools and approaches to improve test coverage and efficiency.
• Continuously improve testing methodologies for both conventional software applications and AI-powered systems.

Required Skills & Experience

1. Functional & Manual QA

• 3–5 years of hands-on experience in manual and functional software testing.
• Strong understanding of software testing methodologies, SDLC, STLC, and defect management.
• Experience designing and executing test cases for functional, regression, integration, and end-to-end testing.
• Strong knowledge of Agile/Scrum methodologies and sprint-based QA processes.
• Experience testing web applications, APIs, and mobile applications.
• Ability to analyse requirements and identify functional gaps, edge cases, and potential risks.

2. Mandatory AI/ML Testing Experience

• Prior hands-on experience testing AI/ML-powered applications or products.
• Practical experience testing LLM-based applications, AI chatbots, conversational systems, or intelligent automation features.
• Understanding of prompt testing and evaluating AI-generated responses.
• Ability to validate AI outputs for accuracy, relevance, consistency, and contextual correctness.
• Experience identifying hallucinations, unexpected AI behaviour, and response-quality issues.
• Understanding of AI application workflows, model outputs, and the challenges associated with non-deterministic systems.
• Experience collaborating with AI/ML engineers or data science teams during testing and validation.

3. API & Integration Testing

• Hands-on experience with manual API testing using Postman.
• Understanding of REST APIs, JSON, request/response validation, authentication, and error handling.
• Experience validating integrations between frontend, backend, and external services.
• Basic understanding of database validation and SQL queries.

4. Testing Tools & Technologies

• Experience with JIRA and Confluence for defect tracking and test documentation.
• Familiarity with Postman, SQL, and browser developer tools.
• Understanding of test management and reporting practices.
• Basic knowledge of BDD frameworks such as Cucumber/Gherkin is an advantage.

Preferred Skills

• Experience testing RAG-based applications, including document retrieval and contextual response validation.
• Exposure to vector databases and semantic search testing.
• Familiarity with LangChain, LangGraph, LlamaIndex, or similar AI frameworks.
• Understanding of LLM evaluation frameworks and AI quality assessment methodologies.
• Experience with AI evaluation tools such as Ragas, DeepEval, or similar frameworks.
• Exposure to prompt injection, adversarial testing, and responsible AI validation.
• Working knowledge of Selenium, Playwright, or Cypress.
• Basic scripting knowledge in Python or JavaScript.
• Familiarity with CI/CD pipelines and automated testing practices.
• Exposure to cross-browser testing tools such as BrowserStack or Sauce Labs.

What We're Looking For Beyond Skills

• AI Testing Mindset: Understands that testing AI applications requires more than checking whether a feature works. Can evaluate response quality, accuracy, relevance, and behaviour.
• Analytical Thinking: Ability to identify unusual scenarios, inconsistent outputs, and hidden quality issues.
• Strong QA Fundamentals: Applies structured testing methodologies while adapting them to AI-powered applications.
• Attention to Detail: Documents defects clearly and identifies patterns in application and AI behaviour.
• Ownership: Takes responsibility for testing quality from requirement analysis through release validation.
• Collaboration: Works effectively with developers, AI engineers, product managers, and data science teams.
• Continuous Learning: Interested in exploring emerging AI technologies and evolving testing methodologies.
• Communication: Clearly communicates test results, defect impact, quality risks, and release readiness.

What We Offer

• Competitive compensation aligned with experience and expertise.
• Flexible work environment (Coimbatore office / Hybrid).
• Paid holidays and flexible time off.
• Medical insurance covering self and family, up to ₹4 lakh per person.
• Opportunity to work on real-world AI-powered products and enterprise applications.
• Hands-on exposure to LLMs, RAG systems, agentic AI, and modern AI engineering practices.
• Opportunities to develop expertise in AI quality engineering and automation testing.
• A collaborative, multicultural environment with opportunities for professional growth.
• A culture that values clarity, integrity, ownership, and continuous learning.

Join Techjays

If you're passionate about software quality and have hands-on experience validating AI-powered applications, we'd love to hear from you.

Join us in building reliable, intelligent, and production-ready AI systems.

Skills

Software Testing (QA), Artificial Intelligence (AI), Anthropic Claude, Prompt engineering, Machine Learning (ML), Selenium, Retrieval Augmented Generation (RAG)