Automation Engineer - AI Testing

Job Summary :

We are looking for an Automation Engineer specializing in AI Testing to develop and maintain automated testing frameworks for modern software applications, including AI-enabled systems.

The ideal candidate will have strong experience in test automation, API testing, quality engineering, and test framework development, with an understanding of how to validate AI-driven features for accuracy, reliability, consistency, and performance.

The role involves collaborating with software developers, QA engineers, data teams, and product stakeholders to improve test coverage, accelerate release cycles, and maintain high-quality software delivery.

Key Responsibilities :

- Design, develop, and maintain scalable automated testing frameworks for web applications, backend services, and APIs.

- Build reusable test scripts and automation components using Python, Java, or JavaScript.

- Develop automated functional, regression, integration, and end-to-end test suites.

- Perform API testing using tools such as Postman, REST Assured, or equivalent frameworks.

- Automate UI testing using Selenium, Playwright, Cypress, or similar tools.

- Integrate automated test suites into CI/CD pipelines to enable continuous testing and faster feedback.

- Develop test strategies for AI-enabled applications, including validation of AI-generated outputs and intelligent application workflows.

- Evaluate AI system behavior for accuracy, consistency, robustness, reliability, and expected outcomes.

- Design test cases for edge cases, unexpected inputs, failure scenarios, and non-deterministic AI responses.

- Support testing of LLM-powered applications, including prompt-based workflows, response validation, and regression testing.

- Establish suitable evaluation metrics and test datasets for AI-driven functionality.

- Collaborate with developers and AI/ML teams to identify defects, investigate unexpected behavior, and improve system quality.

- Perform database validation using SQL and verify data integrity across application workflows.

- Execute performance, reliability, and compatibility tests where required.

- Analyze test results, document defects, and work with engineering teams to ensure timely resolution.

- Maintain test documentation, test reports, automation standards, and reusable testing assets.

- Identify opportunities to improve test coverage, reduce manual testing effort, and increase automation efficiency.

Required Skills :

- 4 - 8 years of experience in software testing, QA automation, or quality engineering.

- Strong programming skills in Python, Java, or JavaScript.

- Hands-on experience developing automated testing frameworks.

- Experience with Selenium, Playwright, Cypress, or equivalent automation tools.

- Strong knowledge of API testing, REST services, and JSON-based data validation.

- Experience with functional, regression, integration, and end-to-end testing.

- Familiarity with CI/CD pipelines and tools such as Jenkins, GitHub Actions, or Azure DevOps.

- Working knowledge of SQL and database validation.

- Experience with Git and collaborative software development workflows.

- Strong understanding of test design techniques, defect lifecycle management, and root cause analysis.

- Ability to analyze complex application behavior and create comprehensive test scenarios.

- Strong communication, documentation, and cross-functional collaboration skills.

Preferred Qualifications:

- Experience testing AI/ML-enabled applications or LLM-powered systems.

- Familiarity with AI evaluation frameworks and tools such as DeepEval, Ragas, or equivalent solutions.

- Understanding of prompt injection, hallucination testing, output consistency, and AI safety validation.

- Experience with performance testing tools such as JMeter or Locust.

- Familiarity with Docker, Kubernetes, and cloud-based testing environments.

- Exposure to test reporting and quality metrics dashboards.

- Experience with risk-based testing, test data management, and large-scale automation initiatives.

Skills

Automation Testing, Software Quality Assurance, Python, Java, Javascript, Selenium, Cypress, RESTful API, Artificial Intelligence, Playwright Testing