Automation Test Engineer
Smart Apply
Similar jobs
Location
In office (Bengaluru)
Employment
Full-Time
First seen by Alion on Oct 10, 2026.
Overview
Company
Impact
Profile match
Sonata Software is an Indian IT services and technology company headquartered in Bengaluru that provides modernization engineering across cloud, data, AI, Microsoft Dynamics, test automation and managed services. Founded in 1986 and listed in India, it runs the Harmoni.AI enterprise platform and works with Fortune 500 clients in banking, retail, healthcare, manufacturing and technology, with a presence in the US, UK, Malaysia, Mexico, Australia, the DACH region and the Nordics. It hires data and cloud architects, MLOps and DevOps engineers, .NET and full-stack developers, identity and security specialists, SharePoint administrators and JD Edwards consultants.
Job Title- Gen AI Tester
Location- Bangalore/Hyderabad/ Chennai
Roles And Responsibilities
Key Responsibilities:AI QA / LLM Automation Engineer
Job Overview
We are seeking a highly skilled and innovative AI QA / LLM Automation Engineer to join our team. In this pivotal role, you will be responsible for designing, developing, and executing sophisticated testing strategies tailored specifically for generative AI systems. You will play a critical role in evaluating Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) pipelines, and autonomous AI agents to ensure high accuracy, contextual relevance, and robust logical reasoning.
Key Roles & Responsibilities
AI Model Validation: Design and execute comprehensive test strategies specifically for evaluating LLM responses. Focus heavily on measuring accuracy, ensuring long-term context retention, and drastically reducing or mitigating hallucinations.
AI Agent Testing: Rigorously test the underlying logic, complex state management, and tool-calling capabilities of autonomous AI agents across various edge cases.
Python Automation: Develop, maintain, and scale custom automated testing frameworks from scratch using Python.
CI/CD Integration: Continuously evaluate AI models by seamlessly integrating custom testing and evaluation frameworks into existing CI/CD pipelines.
Evaluation Metrics: Define and implement novel evaluation metrics for non-deterministic AI features where standard deterministic software testing (exact-match) falls short.
Cross-Functional Collaboration: Work closely with AI developers, data scientists, and product managers to establish quality benchmarks for AI-driven features.
Required Skills & Qualifications
Programming: Strong programming proficiency in Python, with an emphasis on writing clean, scalable, and maintainable automation scripts.
AI/ML Knowledge: Deep understanding of Large Language Model (LLM) architectures, prompt engineering, and model behaviors.
Framework Experience: Hands-on experience building or testing RAG (Retrieval-Augmented Generation) pipelines and agentic frameworks (e.g., LangChain, LlamaIndex).
Non-Deterministic Testing: Proven experience in validating non-deterministic outputs where traditional exact-match assertions do not apply (e.g., semantic similarity, LLM-as-a-judge, or custom evaluation heuristics).
Automation & DevOps: Solid understanding of CI/CD tools (e.g., GitHub Actions, GitLab CI, Jenkins) and version control (Git).
Problem Solving: Strong analytical mindset with the ability to foresee edge cases in generative AI applications.
Experience in AI evals, guardrails, and testing/automating AI agents, which are critical skills for the MAP team.
Key Responsibilities
AI Model Validation: Design and execute test strategies specifically for evaluating LLM responses, focusing on accuracy, context retention, and reducing hallucinations.
AI Agent Testing: Test the logic, state management, and tool-calling capabilities of autonomous AI agents.
Python Automation: Develop custom automated testing frameworks using Python to continuously evaluate AI models in CI/CD pipelines.
Required Skills
Strong programming proficiency in Python.
Deep understanding of LLM architectures, RAG (Retrieval-Augmented Generation), and agentic frameworks (e.g., LangChain, LlamaIndex).
Experience validating non-deterministic outputs where traditional exact-match assertions do not apply.
Qualifications & Required Skills- BE/ B.Tech/ Any Engineering
Mandatory skills-Python, DeepEval ,AI Evals ,AIGuardrails , LLM Testing ,RAG , AI Agents , Agentic AI ,LangChain , LlamaIndex
About Sonata Software
Sonata Software is an AI-first modernization engineering company that helps enterprises transform legacy systems into intelligent, scalable business platforms. Powered by its Platformation™ framework and Harmoni.AI platform, Sonata delivers AI-led modernization across cloud, data, AI, Dynamics, test automation, and managed services. Headquartered in Bengaluru, India, Sonata has more than $1.2 billion in revenue and 6,400+ AI engineers supporting global delivery across regions including the US, UK, India, Malaysia, Mexico, Australia, DACH, and the Nordics. With deep partnerships across Microsoft, AWS, Salesforce, and Snowflake, Sonata helps Fortune 500 enterprises accelerate innovation, improve efficiency, and drive sustainable growth.
For more information, please visit www.sonata-software.com
Location- Bangalore/Hyderabad/ Chennai
Roles And Responsibilities
Key Responsibilities:AI QA / LLM Automation Engineer
Job Overview
We are seeking a highly skilled and innovative AI QA / LLM Automation Engineer to join our team. In this pivotal role, you will be responsible for designing, developing, and executing sophisticated testing strategies tailored specifically for generative AI systems. You will play a critical role in evaluating Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) pipelines, and autonomous AI agents to ensure high accuracy, contextual relevance, and robust logical reasoning.
Key Roles & Responsibilities
AI Model Validation: Design and execute comprehensive test strategies specifically for evaluating LLM responses. Focus heavily on measuring accuracy, ensuring long-term context retention, and drastically reducing or mitigating hallucinations.
AI Agent Testing: Rigorously test the underlying logic, complex state management, and tool-calling capabilities of autonomous AI agents across various edge cases.
Python Automation: Develop, maintain, and scale custom automated testing frameworks from scratch using Python.
CI/CD Integration: Continuously evaluate AI models by seamlessly integrating custom testing and evaluation frameworks into existing CI/CD pipelines.
Evaluation Metrics: Define and implement novel evaluation metrics for non-deterministic AI features where standard deterministic software testing (exact-match) falls short.
Cross-Functional Collaboration: Work closely with AI developers, data scientists, and product managers to establish quality benchmarks for AI-driven features.
Required Skills & Qualifications
Programming: Strong programming proficiency in Python, with an emphasis on writing clean, scalable, and maintainable automation scripts.
AI/ML Knowledge: Deep understanding of Large Language Model (LLM) architectures, prompt engineering, and model behaviors.
Framework Experience: Hands-on experience building or testing RAG (Retrieval-Augmented Generation) pipelines and agentic frameworks (e.g., LangChain, LlamaIndex).
Non-Deterministic Testing: Proven experience in validating non-deterministic outputs where traditional exact-match assertions do not apply (e.g., semantic similarity, LLM-as-a-judge, or custom evaluation heuristics).
Automation & DevOps: Solid understanding of CI/CD tools (e.g., GitHub Actions, GitLab CI, Jenkins) and version control (Git).
Problem Solving: Strong analytical mindset with the ability to foresee edge cases in generative AI applications.
Experience in AI evals, guardrails, and testing/automating AI agents, which are critical skills for the MAP team.
Key Responsibilities
AI Model Validation: Design and execute test strategies specifically for evaluating LLM responses, focusing on accuracy, context retention, and reducing hallucinations.
AI Agent Testing: Test the logic, state management, and tool-calling capabilities of autonomous AI agents.
Python Automation: Develop custom automated testing frameworks using Python to continuously evaluate AI models in CI/CD pipelines.
Required Skills
Strong programming proficiency in Python.
Deep understanding of LLM architectures, RAG (Retrieval-Augmented Generation), and agentic frameworks (e.g., LangChain, LlamaIndex).
Experience validating non-deterministic outputs where traditional exact-match assertions do not apply.
Qualifications & Required Skills- BE/ B.Tech/ Any Engineering
Mandatory skills-Python, DeepEval ,AI Evals ,AIGuardrails , LLM Testing ,RAG , AI Agents , Agentic AI ,LangChain , LlamaIndex
About Sonata Software
Sonata Software is an AI-first modernization engineering company that helps enterprises transform legacy systems into intelligent, scalable business platforms. Powered by its Platformation™ framework and Harmoni.AI platform, Sonata delivers AI-led modernization across cloud, data, AI, Dynamics, test automation, and managed services. Headquartered in Bengaluru, India, Sonata has more than $1.2 billion in revenue and 6,400+ AI engineers supporting global delivery across regions including the US, UK, India, Malaysia, Mexico, Australia, DACH, and the Nordics. With deep partnerships across Microsoft, AWS, Salesforce, and Snowflake, Sonata helps Fortune 500 enterprises accelerate innovation, improve efficiency, and drive sustainable growth.
For more information, please visit www.sonata-software.com
Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
1,463,766 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Free forever. No card. Under a minute.
Your match
How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.
Recommended for you based on this role
QA
Similar stack
Same company
Bengaluru
Senior QA Engineer
5 hours ago
$52k – $66k per year (gross) • In office • Full-Time • 5+ years exp • Singapore
Cybersecurity
CAPA
Management
Microsoft Office
Apply
QA Engineer
5 hours ago
$38k – $75k per year (gross) • In office • Contractor • 4+ years exp • Singapore
Management
Agile
Scrum
Apply
Apply
Software Engineer - Test Automation
5 hours ago
$38k – $56k per year (gross) • In office • Full-Time • Singapore
Python
Java
C#
C++
DevOps
Unix
Management
Agile
QA
Selenium
Apply
Hybrid • 4+ years exp • Lisbon
DevOps
Azure DevOps
Azure
Management
Jira
QA
TestRail
Apply
Advisor TPM
1 day ago
≈ $24k – $49k per year (Estimated) • Hybrid • Full-Time • 8+ years exp • Bachelor's Degree • Bengaluru
Management
Agile
Apply
Product Software Engineer II
1 day ago
In office • Full-Time • 7+ years exp • Bachelor's Degree • Bengaluru
SQL
Databases
Snowflake
Apache Kafka
DevOps
GCP
Azure
AWS
Incident Management
Management
Agile
Apply
Apply
Apply
Apply
This is one of many
1,463,766 more open roles from verified company boards, updated every day.