Senior QA Engineer - Automation
Confirmed on the employer's own hiring board on Oct 15, 2026. First seen by Alion on Sep 9, 2026.
Job Summary:
Convo is looking for QA Automation Engineers with 5+ years of experience in test automation to build robust automated testing frameworks for APIs, agent workflows, tool calls, and critical UI experiences. The ideal candidate should have strong expertise in Python, pytest, API testing, Playwright, CI/CD, contract testing, and asynchronous workflows, with experience creating reusable test fixtures, mocks, and simulators to ensure reliable and scalable product quality.
Technical mission
Automate regression testing for agent tool calls, planner outputs, APIs and the operator-facing agentic console.
Key responsibilities
Create API and workflow regression suites for orchestrator, planner and tool-registry behavior.
Build deterministic assertions around schemas, tool-call sequences, constraints and expected side effects.
Implement pytest-based service tests and Playwright tests for relevant console workflows.
Create mocks, simulators and fixtures for tools, model responses and external dependencies.
Integrate suites into CI, publish actionable results and manage flaky-test remediation.
Maintain traceability from technical requirements and defects to automated coverage.
Required technical capabilities
Minimum experience: 4+ years in test automation, including 2+ years testing APIs, asynchronous workflows or distributed services.
Strong Python, pytest and API-testing experience.
Playwright or comparable browser automation for critical UI workflows.
CI pipeline integration and test-result reporting.
Contract/schema validation for REST and event-driven interfaces.
Ability to test asynchronous, non-deterministic and retrying workflows without masking defects.
Experience designing reusable test fixtures, mocks and service simulators.
Preferred experience
Agent/LLM workflow evaluation and tool-call trace analysis.
Performance, resilience and fault-injection testing.
OpenTelemetry or distributed-trace-based test diagnostics.
Expected deliverables / acceptance evidence
Automated regression suites for planner, tool calls, APIs and critical UI paths.
CI integration with quality thresholds and readable failure evidence.
Reusable mocks, fixtures and simulators.
Coverage and defect-leakage reporting with maintained documentation.
Primary interfaces
Works with Agent Runtime, DevOps, AI Test & Dataset Engineering and product teams consuming the shared framework.
What We Have For You- Great compensation package, medical benefit for you and your family, free lunch, annual performance-tied increments & performance recognition awards and a great lean and agile work culture!
- Convo endorses a culture of diversity in all aspects and aims to build a diverse team of amazing individuals!