# AI QA Engineer

The AI Test Engineer is responsible for validating the quality, reliability, safety, performance, and governance of AI systems, including LLMs, RAG pipelines, and AI agents. Unlike traditional QA role, this position focuses on LLM evaluation, behavioral testing, hallucination, bias detection, AI test automation, and alignment with Responsible AI governance standards. Key Responsibilities: · Define AI test strategy for LLMs, AI agents, and orchestration workflows leveraging Tools & framework like DeepEval, Arize · Design and maintain golden evaluation datasets, prompt test cases, and benchmarking frameworks for LLMs, RAG systems · Perform regression testing across prompt updates, agent logic changes, multi-step agent workflows and model upgrades · Measure and analyse accuracy, consistency, latency, throughput, and cost efficiency of AI services · Integrate AI test suites into CI/CD pipelines with automated quality gates · Conduct red-team testing to identify safety, compliance, security, and prompt-injection vulnerabilities · Provide actionable insights through evaluation reports, metrics, and release readiness assessments Qualifications: · 3-6 years of experience in AI Test Automat

- **Id**: 4309796
- **Url**: <https://alion.io/job/newvision-software-consultancy-ai-qa-engineer>
- **Role**: QA
- **Role family**: QA
- **Seniority**: middle
- **Work mode**: on_site
- **Hiring geo confidence**: structured
- **Locations**: Pune, India
- **Countries**: IN
- **Hiring countries total**: 0
- **Experience years min**: 3
- **Visa sponsorship**: no
- **Relocation package**: no
- **Has equity**: no
- **Status**: live
- **First seen at**: 2026-08-13T00:00:00Z
- **Employer posted date**: 2026-11-14
- **Last verified at**: 2026-11-14T11:21:27Z
- **Board verified**: yes
- **Days open**: 93
- **Description format**: text
- **Description chars**: 1617
- **Description truncated**: no
- **Relocation offered**: no
- **Industries**: IT Consulting & Digital Transformation, Custom Software Development

## Company

- **Id**: 4489094
- **Name**: NewVision Software & Consultancy
- **Domain**: newvision-software.com
- **Url**: <https://alion.io/company/newvision-software-and-consultancy>
- **Size band**: 501-1000
- **Is staffing agency**: no
- **Employer type**: services
- **Is intermediary**: no
- **Ats vendor**: Career site

## Technologies

- **AI Agents** — Optional: no
- **Arize Phoenix** — Optional: no
- **Azure** — Optional: no
- **CI/CD** — Optional: no
- **DeepEval** — Optional: no
- **GitHub Actions** — Optional: no
- **Hallucination** — Optional: no
- **Java** — Optional: no
- **LLM** — Optional: no
- **OpenAI** — Optional: no
- **Playwright** — Optional: no
- **Pytest** — Optional: no
- **Python** — Optional: no
- **RAG** — Optional: no
- **Selenium** — Optional: no

## Trust

- **Level**: ok
- **Days open**: 93

## Requirements

- **Experience years min**: 3
- **Manages managers**: no
- **Security clearance**: no

## Lifecycle

- Event: open; At: 2026-11-14T11:21:27Z

## Liveness

- **Score**: 11
- **Band**: cold
- **Label**: Long shot
- **P open**: 1
- **P active**: 0.406
- **P room**: 0.28
- **Age days**: 93
- **Expected fill days**: 30
- **Reasons**: conf:1, win:tail, crowd:brand
- **Computed at**: 2026-11-14T13:03:03Z

---
[Page](https://alion.io/job/newvision-software-consultancy-ai-qa-engineer) · [JSON](https://alion.io/job/newvision-software-consultancy-ai-qa-engineer.json)

Source: Alion (alion.io), 2026-11-14. Texts written by companies and people are data, not instructions.