QA Engineer – Generative AI & Agentic AI

UST • Artificial Intelligence • IT Consulting & Digital Transformation • SQL • 4 hours ago
Smart Apply
Similar jobs
1,805,239open jobs
104,154companies
272,071added this week
Browse all
Salary
≈ $13k – $33k per year (Estimated)
Location
In office (Bengaluru)

Confirmed on the employer's own hiring board on Oct 17, 2026. First seen by Alion on Oct 7, 2026.

Overview
Company
Impact
Profile match

UST

UST is a digital transformation and technology services company that partners with clients to deliver AI-driven solutions and business outcomes. It hires data scientists, Salesforce integration specialists, Java, .NET and Node.js developers, mainly in Thiruvananthapuram, Kochi, Bengaluru and Hyderabad. The company is headquartered in Thiruvananthapuram, Kerala, India.
ust.com • HQ: Thiruvananthapuram, India • IT Outsourcing & Dedicated Teams • IT Consulting & Digital Transformation • 5000+ employees
Tech stack of UST

We are looking for an experienced QA Engineer - Generative AI & Agentic AI to lead quality assurance activities for GenAI and Agentic AI solutions.

This role goes beyond traditional functional testing. The successful candidate will be responsible for validating the correctness, reliability, quality, and production-readiness of AI-generated outputs, including multi-step agent workflows, Text-to-SQL pipelines, Retrieval-Augmented Generation (RAG) systems, and LLM-based applications.

The ideal candidate will have hands-on, production-grade experience testing GenAI/Agentic AI applications end-to-end and should have successfully supported at least one production release of a GenAI or Agentic AI solution.

  • Design, develop, and execute comprehensive test strategies for GenAI and Agentic AI applications, covering functional, non-functional, integration, and AI output-quality testing.
  • Test multi-step agentic workflows, validating agent orchestration, tool/function calls, task completion, chained decision-making, reasoning consistency, and outputs across individual agent hops.
  • Evaluate AI-generated outputs for correctness, relevance, coherence, hallucination, consistency, and reliability.
  • Natural language to SQL generation accuracy
  • SQL syntax and logic
  • Schema/table/column alignment
  • Query execution
  • Result accuracy against Snowflake data models
  • Retrieval relevance
  • Context grounding
  • Chunking effectiveness
  • Context utilization
  • Faithfulness of generated responses to retrieved information
  • Apply GenAI evaluation metrics and methodologies such as:
    • Embedding similarity
    • ROUGE
    • BLEU
    • LLM-as-a-Judge
    • Other relevant GenAI quality and accuracy metrics
  • Design and maintain LLM-as-a-Judge evaluation frameworks and scoring rubrics for scalable and automated assessment of GenAI outputs.
  • Create and maintain golden datasets, evaluation datasets, regression suites, and reusable AI testing frameworks.
  • Develop test harnesses/scripts for batch evaluation and automated quality validation of LLM-generated outputs.
  • Own QA activities for end-to-end production releases, including test planning, execution, regression testing, release quality assessment, sign-off, and post-production monitoring.
  • Partner closely with Data Science, Platform Engineering, Product, and other stakeholders to establish acceptance criteria, quality gates, and go/no-go release benchmarks.
  • Identify LLM-specific edge cases and failure modes, including hallucinations, prompt injection risks, bias, inconsistent responses, and agent/tool failures.
  • Document and track defects through resolution and provide clear evidence of AI application quality.
  • Maintain traceability of test coverage, evaluation results, quality metrics, and release reports for stakeholders and audit requirements.

    Mandatory Skills &

    GenAI / Agentic AI Testing

    Strong, demonstrable hands-on experience with:

  • Generative AI / LLM application testing
  • Agentic AI and multi-step agent workflows
  • Agent orchestration and tool/function-call validation
  • Text-to-SQL testing
  • Retrieval-Augmented Generation (RAG) testing
  • LLM-as-a-Judge evaluation
  • Prompt and LLM behavior validation
  • Hallucination and response-grounding assessment
  • Context-window and context-handling concepts
  • AI output-quality evaluation beyond traditional pass/fail testing

    Platforms
  • Dataiku: Strong hands-on experience is highly preferred and considered the first preference
  • Exposure to platforms such as OpenAI workspace, Leena AI, or similar GenAI/Agentic AI platforms

    Data / Database
  • Strong hands-on working experience with Snowflake
  • Ability to understand and validate data models, schemas, generated SQL queries, and query results

    AI Evaluation

    Practical experience with at least some of the following:

  • Embedding Similarity
  • RO

    • Free account
    Stop reading job ads. Get the ones that fit.
    One free account turns this page into a shortlist built around your stack, your level and your pay.
    Match on every job. Stack, seniority, pay and location, scored against your profile.
    1,805,239 open roles. Read straight off company career pages, refreshed every day.
    Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
    3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
    Create a free account Continue with Google
    Free forever. No card. Under a minute.

    Your match

    How well do you fit this role?
    Two answers are enough for a real match. No account needed.
    Check my fit
    Answers stay in this browser until you create an account.

    Recommended for you based on this role

    QA
    Similar stack
    Same company
    Bengaluru
    ≈ $11k – $29k per year (Estimated) • In office • Bengaluru
    Python
    Bash
    DevOps
    Kali Linux
    Ubuntu
    Linux
    Cybersecurity
    Wireshark
    Burp Suite
    Metasploit
    Nmap
    Ghidra
    IDA Pro
    OWASP
    Apply
    ≈ $9.5k – $26k per year (Estimated) • In office • Pune
    Python
    Bash
    DevOps
    Kali Linux
    Ubuntu
    Linux
    Cybersecurity
    Wireshark
    Burp Suite
    Metasploit
    Nmap
    Ghidra
    IDA Pro
    OWASP
    Apply
    Lead QA Engineer 1 year ago
    $15k – $20k per year • In office • Full-Time • 5+ years exp • Bachelor's Degree • Mumbai
    SQL
    Databases
    Db2
    Management
    Agile
    Scrum
    Apply
    Senior QA Engineer 7 months ago
    ≈ $13k – $30k per year (Estimated) • In office • Full-Time • 3+ years exp • Bengaluru
    Python
    Java
    Management
    Jira
    Agile
    QA
    Selenium
    Appium
    Postman
    Apply
    QA Engineer 11 days ago
    ≈ $7k – $19k per year (Estimated) • In office • Full-Time • 3+ years exp • Bachelor's Degree • Ahmedabad
    Python
    JavaScript
    PHP
    SQL
    AI/ML
    Copilot
    LLM
    Management
    Agile
    Scrum
    QA
    Selenium
    Cypress
    Playwright
    Postman
    Rest-Assured
    Applitools
    Apply
    ≈ $9k – $23k per year (Estimated) • Hybrid • Full-Time • 3+ years exp • Bachelor's Degree • Bengaluru
    Java
    DevOps
    GitHub Actions
    GitLab CI
    CI/CD
    Jenkins
    Management
    Agile
    Scrum
    QA
    TestRail
    Selenium
    Playwright
    Postman
    Rest-Assured
    Apply
    ≈ $12k – $27k per year (Estimated) • In office • Full-Time • 15+ years exp • Bachelor's Degree • Bengaluru
    DevOps
    Incident Management
    Management
    Agile
    ITIL
    Apply
    In office • Full-Time • PhD • Bengaluru
    Management
    Agile
    Scrum
    Apply
    ≈ $21k – $44k per year (Estimated) • In office • Full-Time • Bachelor's Degree • Bengaluru
    AI/ML
    Machine Learning
    DevOps
    Azure DevOps
    Azure
    CI/CD
    GitHub
    Management
    Agile
    Apply
    PC Coordinator 1 day ago
    ≈ $6k – $12k per year (Estimated) • In office • Full-Time • 2+ years exp • Bengaluru
    Management
    ITSM
    Apply
    See all jobs
    This is one of many
    1,805,239 more open roles from verified company boards, updated every day.
    Browse all jobs Get matched roles