Generative AI Quality Assurance Engineer
First seen by Alion on Jul 28, 2026.
Role Overview:
We are seeking a seasoned Gen AI QA Engineer to join our Bangalore-based engineering team to spearhead the quality assurance strategy for our next-generation AI products. In this role, you will be responsible for defining testing frameworks for Large Language Models (LLMs), ensuring that our generative outputs are accurate, safe, and aligned with user intent. You will collaborate closely with Data Scientists, Product Managers, and Backend Engineers to bridge the gap between traditional software testing and the non-deterministic nature of AI. Your work will directly influence the reliability of our AI-driven features, ensuring that we deliver high-quality, scalable solutions that provide tangible value to our global customer base.
Key Responsibilities:
- Design and implement robust testing strategies for LLM-based applications to evaluate model performance, hallucinations, and response latency.
- Develop and maintain automated test suites using Java, Selenium, and Playwright to ensure seamless integration between AI models and existing web interfaces.
- Engineer sophisticated prompts and evaluation datasets to benchmark model behavior against diverse edge cases and user scenarios.
- Partner with cross-functional teams to establish quality gates in the CI/CD pipeline, ensuring that AI model updates do not compromise system stability.
- Analyze complex failure patterns in AI outputs and provide actionable insights to engineering teams to improve model fine-tuning and prompt engineering techniques.
Required Skillset:
- Demonstrated expertise in AI/ML testing methodologies, with a specific focus on validating Generative AI models and LLM architectures.
- Strong proficiency in Java for building scalable automation frameworks, complemented by hands-on experience with Selenium and Playwright for UI and API testing.
- Proven ability to perform prompt engineering to stress-test models and evaluate the quality, safety, and relevance of generated content.
- Exceptional analytical skills to interpret non-deterministic test results and communicate technical findings to both technical stakeholders and product leadership.
- Ability to thrive in a high-growth, hybrid work environment in Bangalore, demonstrating the adaptability to pivot between manual exploratory testing and complex automation development.
- A Bachelor's or Master's degree in Computer Science or a related field, with 5 to 8 years of professional experience in software quality assurance and AI/ML testing.
Skills
AI/ML Testing, Java, Selenium, Playwright Testing, Software Quality Assurance, Automation Testing, Testing, Generative AI, Product Testing