QA AI Engineer
First seen by Alion on Sep 26, 2026.
QA AI Engineer
Role Overview:
We are looking for a QA AI Engineer to design and execute quality engineering strategies for Generative AI, RAG, Agentic AI, AI assistants, semantic search, and AI-enabled data solutions.
The role will focus on validating AI solutions across data quality, model outputs, retrieval accuracy, response quality, reliability, performance, security, and functional correctness.
The ideal candidate should have a strong understanding of AI/ML concepts and experience testing modern AI applications and data-driven solutions in enterprise environments.
Experience in a Healthcare Payer environment will be an added advantage.
Key Responsibilities:
AI & GenAI Testing:
- Design and execute test strategies for Generative AI, RAG, Agentic AI, AI assistants, and semantic search solutions.
- Validate AI-generated responses for accuracy, relevance, consistency, completeness, and factual correctness.
- Develop test scenarios covering different user queries, prompts, contexts, and business use cases.
- Evaluate LLM responses against defined business and quality benchmarks.
- Test prompt variations, context handling, hallucination scenarios, and edge cases.
- Validate AI workflows involving multiple agents, tools, APIs, and data sources.
- Support the creation of AI evaluation frameworks and quality metrics.
RAG & Vector Search Testing:
- Validate RAG pipelines across ingestion, chunking, embedding, retrieval, context generation, and response generation.
- Test vector databases, embeddings, similarity search, and retrieval accuracy.
- Evaluate precision, recall, relevance, and ranking of retrieved information.
- Identify issues related to incorrect, incomplete, outdated, or irrelevant context.
- Validate semantic search and knowledge retrieval capabilities.
- Test knowledge graph-based retrieval and AI-assisted data consumption where applicable.
Data Quality & Validation:
- Design test cases to validate data accuracy, completeness, consistency, uniqueness, and integrity.
- Validate data pipelines and datasets used by AI/ML applications.
- Test data ingestion, transformation, enrichment, and processing workflows.
- Identify data quality issues that could impact model or AI application performance.
- Validate data lineage and source-to-target mappings.
- Support testing of AI-ready datasets and semantic data layers.
AI Model & Application Validation:
- Validate AI/ML models and applications against functional and business requirements.
- Test model outputs across different datasets, scenarios, and user personas.
- Perform regression testing for AI models and GenAI applications following model, prompt, or data changes.
- Evaluate model performance and identify degradation over time.
- Test model behaviour for boundary conditions, unexpected inputs, and failure scenarios.
- Support validation of explainability, transparency, fairness, and responsible AI requirements.
API & Integration Testing:
- Test REST, GraphQL, event-driven, and streaming APIs supporting AI applications.
- Validate API responses, data contracts, authentication, authorization, error handling, and performance.
- Test integrations between AI applications, databases, vector stores, data platforms, and enterprise systems.
- Perform integration and end-to-end testing across distributed AI workflows.
Performance & Reliability Testing:
- Conduct performance and scalability testing for AI applications and data services.
- Validate response time, throughput, concurrency, and system stability.
- Identify performance bottlenecks across APIs, data pipelines, retrieval systems, and AI services.
- Test application behaviour under high-volume and high-concurrency scenarios.
- Validate reliability and recovery mechanisms for AI-enabled applications.
Automation & Quality Engineering:
- Develop and maintain automated test frameworks for AI and data applications.
- Automate functional, regression, API, data validation, and integration test scenarios.
- Build reusable test utilities and validation frameworks.
- Integrate automated testing into CI/CD pipelines.
- Maintain test data, test suites, test reports, and quality dashboards.
- Support continuous quality improvement across AI development lifecycle.
AI Governance & Responsible AI:
- Validate AI solutions against defined security, governance, privacy, and responsible AI requirements.
- Test for sensitive data exposure, inappropriate responses, and policy violations.
- Support testing related to AI explainability and traceability.
- Validate compliance requirements applicable to enterprise and healthcare data.
- Document quality risks, defects, test outcomes, and mitigation recommendations.
Tech Stack:
- Generative AI / LLMs, RAG, Agentic AI, Vector databases, Embeddings, Knowledge graphs, Semantic search
- Python, SQL, REST / GraphQL APIs
- Azure / AWS, Databricks, Snowflake, Microsoft Fabric
- AI/ML testing frameworks, CI/CD and test automation tools
Healthcare Domain (Good to Have):
- Experience working with Healthcare Payer data or applications (Claims, Membership, Provider, Care Management, Utilization Management, Healthcare analytics, Medical or pharmacy data).
- Understanding of healthcare data privacy, governance, and regulatory requirements.
Skills
Python, Automation Testing, SQL, API Testing, Software Quality Assurance, Generative AI, RAG, AI/ML Testing, Data Quality, Validation Testing, Functional Testing