AI-First Test Automation Engineer / QA Architec

Job Specification: AI-First Test Automation Engineer / QA Architect

Company: Kalibr8

Project: Next Generation Kalibr8 Platform

Employment Type: Contract / Fixed-Term

Location: Remote

Role Overview

Kalibr8 is developing the next generation of its platform and is seeking an experienced AI-First Test Automation Engineer / QA Architect to design, implement and hand over a comprehensive, AI-driven quality assurance capability.

This is not a traditional manual testing role. The primary objective is to establish a robust test strategy, automation framework and operating model that can progressively be handed over to AI agents, minimising ongoing human testing and maintenance.

The successful candidate will work closely with the development team, ideally leveraging our Microsoft Azure and Azure DevOps ecosystem, to embed testing into the software development lifecycle.

Key Responsibilities

1. Test Strategy & Planning

• Develop a comprehensive testing strategy for the next generation Kalibr8 platform.
• Define functional, integration, API, end-to-end, regression, performance and security testing requirements.
• Establish test coverage, quality gates, acceptance criteria and release readiness standards.
• Create structured test plans and reusable test cases suitable for AI interpretation and execution.
• Align test planning and traceability with development requirements and work items, preferably using Azure DevOps (ADO).

2. Test Automation Framework

• Design and implement a scalable, maintainable automation framework.
• Establish automated testing across UI, APIs, backend services and integrations.
• Integrate testing into CI/CD pipelines, preferably using Azure Pipelines.
• Implement test data management, environment configuration, reporting and automated quality dashboards.
• Develop reusable testing components that can be maintained and extended by AI agents.
• Support test execution and reporting through Azure DevOps Test Plans or equivalent tooling.

3. AI-Driven Testing & Autonomous QA

This is the most important element of the role.

• Evaluate and implement AI-powered testing frameworks and agent-based solutions.
• Use AI to generate test cases, scenarios, scripts and test data.
• Enable AI agents to interpret new requirements and propose or generate tests automatically.
• Establish AI-driven execution, failure analysis, defect reporting and regression testing.
• Enable AI agents to maintain and update tests as the platform evolves.
• Explore self-healing test automation and autonomous exploratory testing.
• Integrate AI testing workflows with development repositories, work items and pipelines.
• Implement appropriate safeguards, approval controls and validation mechanisms for AI-generated changes.

The long-term objective is for AI to become the primary operator of Kalibr8's testing capability, with humans focused on oversight, exceptions and critical decisions.

4. Azure DevOps & Cloud Integration

Preferably, the candidate will be comfortable working within Microsoft's development ecosystem, including:

• Azure DevOps (ADO): Boards, Repos, Test Plans and Pipelines.
• Azure Pipelines: Automated build, test, deployment and quality gates.
• Azure Repos / Git: Source control, pull requests and automated test maintenance.
• Azure Test Plans: Test case management, traceability and defect reporting.
• Azure AI Foundry / Azure OpenAI: AI-assisted test generation, orchestration and analysis.
• Azure Functions / Azure Container Apps: Hosting and executing automated testing workflows or AI agents.
• Azure Monitor / Application Insights: Monitoring, diagnostics and test observability.

Equivalent tooling and experience will also be considered where the candidate can demonstrate the ability to integrate with the Kalibr8 environment.

5. AI Handover & Knowledge Transfer

A central deliverable is the successful transition from a human-designed testing framework to an AI-operated testing capability.

The candidate will:

• Document the test architecture, standards and operating procedures in AI-consumable formats.
• Develop structured prompts, agent instructions, reusable skills and automated workflows.
• Establish how AI agents interact with ADO work items, repositories, pipelines and testing environments.
• Demonstrate AI agents generating, executing, analysing and maintaining tests.
• Define autonomous operation boundaries, human approval requirements and escalation procedures.
• Establish monitoring and controls to assess the reliability of AI testing.
• Complete a structured handover to the development team and AI-operated testing environment.

Required Experience & Skills

Essential

• Strong software QA and test automation architecture experience.
• Proven experience building test strategies and automation frameworks from scratch.
• Experience with Playwright, Cypress, Selenium or equivalent.
• Strong API, integration and end-to-end testing knowledge.
• Experience with CI/CD pipelines and Git-based development workflows.
• Proficiency in Python, TypeScript, JavaScript or equivalent.
• Practical experience using generative AI and LLMs in software engineering or testing.
• Understanding of AI agents, orchestration and automated development workflows.
• Ability to work independently and deliver a complete framework through to handover.

Preferred => Azure & Microsoft Ecosystem

• Hands-on experience with Azure DevOps (ADO), particularly Boards, Repos, Pipelines and Test Plans.
• Experience integrating automated testing into Azure Pipelines.
• Familiarity with Microsoft Azure cloud services and deployment environments.
• Experience with Azure AI Foundry, Azure OpenAI or comparable AI platforms.
• Familiarity with GitHub Copilot, Microsoft tooling and AI-assisted software development.
• Experience integrating AI agents with ADO work items, test management and repositories.
• Knowledge of Azure Monitor, Application Insights and cloud-based observability.

Additional Desirable Experience

• Autonomous or semi-autonomous AI testing agents.
• AI-generated test cases and intelligent regression testing.
• Self-healing test automation.
• SaaS platform testing.
• Cloud-native application architectures.
• Performance, security and reliability testing.

Key Project Deliverables