Senior Automation QA Engineer — AI Products & Customer Implementations

About the role

Helpware's AI Platform team builds a portfolio of AI-native products for customer support and business operations, all running on a shared multi-tenant platform. The team also implements these products for individual clients with custom configurations, integrations, and SLAs.

We're looking for two Senior QA Engineers to own quality end-to-end - from pre-release testing of the products themselves to validating each customer's live implementation. The two hires will share this work across the portfolio, with each expected to flex between product testing and customer-facing implementation QA as priorities shift.

Operating time zone:

8/9 AM - 5/6 PM EST

14:00/15:00 - 23:00/24:00 CET

What you'll do

• Own test strategy for one or more products in the AI portfolio, from requirements review through release sign-off.

• Design, build, and maintain automated test suites covering API, UI, integration, and regression testing, integrated into CI/CD pipelines.

• Test multi-tenant platform behavior, including RBAC, tenant isolation, data access, and integrations with CRM, ticketing, telephony, and other third-party systems.

• Validate AI-agent behavior against functional requirements, including conversation flows, workflow automation, scoring and coaching logic, and structured-data extraction.

• Lead QA for customer implementations by validating client-specific configurations, connectors, knowledge bases, scoring rubrics, and workflow rules against requirements before go-live.

• Design and execute UAT plans together with client stakeholders and internal delivery and Customer Success teams.

• Perform exploratory and scenario-based testing against real customer workflows and monitor early-production behavior after launch.

• Distinguish between configuration issues, implementation defects, and product-level bugs, and ensure the right teams address them.

• Triage, document, and track defects; work closely with engineers to identify root causes and verify fixes.

• Maintain test plans, implementation checklists, and QA playbooks that can scale across a growing product portfolio and client base.

• Turn recurring implementation issues into automated regression coverage and continuously improve overall QA processes.

• Report on release readiness, product quality, and implementation risks to Product, Engineering, Delivery, and client-facing teams.

What we're looking for

• 5+ years of experience in software QA, including both product-level testing and customer-facing implementation or UAT activities.

• Strong hands-on experience with test automation, using tools such as Playwright, Cypress, Selenium, or similar.

• Solid experience with API testing using Postman, REST-assured, or comparable tools.

• Strong manual and exploratory testing skills, with the ability to create meaningful test scenarios from business and client requirements.

• Experience testing multi-tenant and role-based access systems, including data isolation and configuration-driven deployments.

• Ability to read and understand API and database schemas and work with engineers on root-cause analysis.

• Experience working with CI/CD pipelines and integrating automated tests into development and release processes.

• Comfortable working directly with external stakeholders during UAT and implementation validation.

• Strong understanding of software testing principles, defect lifecycle, regression strategy, and release readiness.

• Clear written and verbal communication skills, with the ability to adapt communication for both technical and client-facing audiences.

Nice to have

• Experience testing AI/LLM-powered products, including accuracy evaluation, hallucination checks, prompt regression, RAG quality, or AI output validation.

• Experience in customer support, contact center, or BPO technology, including call QA, workforce management, agent-assist, or support automation platforms.

• Familiarity with AWS-based environments and cloud-based application testing.

• Experience with observability, logging, tracing, or monitoring tools used for production troubleshooting.

• Exposure to testing in compliance-sensitive environments, such as SOC 2, HIPAA, or PCI-DSS.

• Experience testing integrations with platforms such as Zendesk, Freshdesk, Intercom, telephony systems, or other customer-support technologies.