QA Automation Engineer
First seen by Alion on Sep 10, 2026.
QA Automation Engineer - AI & Application Testing
Experience : 3 + Years
Location : Remote
Employment Type : Full-time
About LegacyLeap :
LegacyLeap is a fast-growing technology startup building AI-powered solutions for enterprise application modernization.
We're working at the intersection of GenAI, Agentic AI, data intelligence, and software engineering to solve complex real-world problems around legacy enterprise systems.
This is an opportunity to join an early-stage, engineering-driven team where you won't just execute predefined tasks - you'll have the opportunity to build, experiment, challenge existing approaches, and directly influence the quality of the product.
If you enjoy working on challenging engineering problems and want to grow with a startup that is building something meaningful with AI, we'd love to hear from you.
About the Role :
We are looking for a QA Automation Engineer with strong core QA fundamentals, hands-on coding skills in Python or Java, and experience in automation, API testing and CI/CD.
The ideal candidate has exposure to AI/GenAI/LLM or Agentic AI application testing and brings a strong QA mindset - someone who naturally looks for gaps, edge cases and opportunities to improve the product and testing process.
What You'll Do :
- Design and develop scalable test automation frameworks for web, mobile, API and modern applications.
- Perform functional, integration, regression and end-to-end testing.
- Build automation using Python or Java and frameworks such as PyTest, Playwright, Selenium, TestNG or similar.
- Integrate automated testing into CI/CD pipelines.
- Test AI/GenAI/LLM-powered applications and Agentic AI workflows.
- Evaluate AI outputs for correctness, relevance, consistency and quality.
- Identify AI-specific risks including hallucinations, prompt injection, jailbreaks and unexpected behaviour.
- Perform API and backend/data validation.
- Work closely with Product and Engineering teams to identify quality risks early.
- Continuously improve test coverage, automation, processes and overall product quality.
- Quickly adapt to new technologies, products and business domains.
Must-Have Skills :
- 3 + years of QA / Software Testing / Automation experience
- Strong understanding of Core QA concepts
- Good programming skills in Python or Java
- Strong problem-solving and logical reasoning skills
- Hands-on automation experience with PyTest, Playwright, Selenium, TestNG or similar
- API testing/automation experience
- CI/CD experience
- SQL/backend validation
- Experience with AI/GenAI/LLM application testing is highly preferred
- Experience with AI evaluation / LLM evaluation / RAG / AI Agent testing is a strong plus
Good to Have :
- Mobile application testing
- Linux / Unix and shell scripting
- RAGAS, DeepEval, LangSmith or similar AI evaluation tools
- Conversational AI / chatbot testing
- Agentic AI testing
- Experience across multiple projects, products or domains
- Product-based company experience
Who Will Thrive Here?
Someone who doesn't simply ask: "What should I test?" - but asks: "What could go wrong? What are we missing? How can we make this better?"
We value strong engineering fundamentals, curiosity, ownership, adaptability and a continuous-improvement mindset.
Why Join LegacyLeap?
- Growing startup - Be part of the journey from an early stage.
- Work with AI - Get hands-on exposure to GenAI, Agentic AI and intelligent enterprise systems.
- Build, don't just maintain - Your work will directly shape products and engineering practices.
- High ownership - Opportunity to take responsibility and make an impact.
- Learn & grow - Work alongside experienced engineers on challenging real-world problems.
- Remote - Work with a distributed, collaborative team.
If you're excited about QA, automation and the future of AI-powered software, come build it with us.
Skills
Automation Testing, API Testing, Python, Java, Selenium, PyTest, AI/ML Testing, TestNG, Application Testing, RAG