SDET Senior Test Lead

Job Summary

We are looking for an experienced QA Engineer with a strong background in both functional and automated testing, and a working understanding of AI and machine learning concepts. This role is central to ensuring the quality, reliability, and trustworthiness of enterprise-grade AI solutions - spanning traditional ML models, Generative AI applications, RAG pipelines, agentic workflows, and the data pipelines that power them.

This is not a generic QA role. The ideal candidate understands how AI systems behave differently from traditional software - where outputs are probabilistic rather than deterministic, where correctness is contextual, and where testing strategies must account for model drift, hallucination, retrieval quality, and data integrity. You will work alongside AI Engineers, data engineers, and architects to define test strategies, build validation frameworks, and deliver clear, actionable quality reports to technical and non-technical stakeholders.

Key Responsibilities

Functional Testing

• Design, execute, and maintain comprehensive test plans and test cases for AI/ML solutions, Generative AI applications, RAG pipelines, agentic workflows, and supporting REST APIs.
• Validate end-to-end functional behaviour of AI and its integrated solutions against defined acceptance criteria and business requirements.
• Test ML model outputs for correctness, consistency, and alignment with expected behaviour - including classification results, predictions, anomaly flags, and recommendations.
• Evaluate RAG pipeline quality: validate retrieval relevance, chunking effectiveness, reranking behaviour, hybrid search results, and the accuracy of LLM-generated responses against source documents.
• Test agentic AI workflows for correct tool/function invocation, reasoning logic, multi-step task completion, and appropriate handling of edge cases and failures.
• Conduct API testing to validate request/response behaviour, error handling, authentication, input validation, and performance under load. Official Use Only
• Perform data pipeline testing: validate data ingestion, transformation, cleansing, and output quality across batch and streaming workflows.
• Execute regression testing following model updates, pipeline changes, or infrastructure modifications.
• Test non-functional requirements including latency, throughput, scalability, and resilience.
• Conduct exploratory testing to surface unexpected or emergent behaviours in AI systems - particularly in agentic and LLM-based solutions. Test Automation
• Design and implement automated test frameworks and suites for AI solutions, APIs, and data pipelines.
• Build reusable, maintainable automation scripts in Python, integrating with testing frameworks such as pytest, Robot Framework, or equivalent.
• Develop automated regression suites that can be integrated into CI/CD pipelines to enable continuous quality validation.
• Implement automated evaluation pipelines for Generative AI outputs - including LLM-as-judge scoring, semantic similarity checks, factual grounding validation, and response consistency testing.
• Automate data quality checks within Databricks or equivalent pipeline environments to validate schema, completeness, referential integrity, and statistical distributions.
• Integrate test automation with monitoring and alerting tools to detect quality degradation in deployed AI solutions. AI-Specific Validation
• Evaluate LLM outputs for hallucination, factual accuracy, relevance, coherence, and adherence to safety and content guidelines.
• Validate embedding quality, vector search precision/recall, and metadata filtering behaviour in RAG architectures.
• Test model fairness, consistency, and robustness across diverse inputs, including adversarial and edge-case scenarios.
• Validate that responsible AI principles - including content safety filters, guardrails, and bias mitigations - are functioning as designed. Official Use Only
• Support model drift monitoring by defining baseline benchmarks and alerting thresholds for deployed ML models. Reporting & Stakeholder Communication
• Produce clear, well-structured test reports summarizing test coverage, results, defect trends, quality risks, and recommendations - tailored for both technical and non-technical audiences.
• Maintain dashboards and metrics that provide ongoing visibility into solution quality, test execution progress, and open defects.
• Document defects clearly, with reproducible steps, expected vs. actual behavior, severity assessment, and supporting evidence (logs, screenshots, sample data).

Skill Requirements

Functional Testing

Proven experience in functional, integration, regression, and exploratory testing of complex software systems.

Strong ability to analyse requirements, design test strategies, and write high-quality test cases with clear pass/fail criteria.

Experience testing REST APIs using tools such as Postman, REST Assured, or equivalent.

Familiarity with testing data-intensive applications, including validation of pipeline outputs and structured/unstructured data quality.

Experience working in Agile/Scrum delivery environments, including sprint-based test planning and execution.

Test Automation

Hands-on experience building and maintaining automated test suites using Python-based frameworks (pytest preferred).

Official Use Only

Experience integrating test automation into CI/CD pipelines (GitHub Actions, Azure DevOps, or equivalent).

Ability to write clean, maintainable automation code following good software engineering practices.

Experience with API test automation and data validation automation.

AI & Domain Knowledge

Working knowledge of AI and ML concepts: understanding of how models are trained, evaluated, and deployed - and the implications for testing strategies.

Familiarity with Generative AI concepts: LLMs, RAG pipelines, embeddings, vector search, chunking, reranking, and agentic workflows.

Ability to evaluate LLM output quality, including identifying hallucinations, factual errors, relevance gaps, and unsafe content.

Understanding of AI non-determinism and how to design testing approaches that account for probabilistic outputs (e.g., semantic equivalence checks rather than exact string matching).

Awareness of responsible AI principles: content safety, bias, fairness, and transparency.

Reporting & Communication

Strong written communication skills, with the ability to produce clear, professional test reports and defect documentation.

Ability to distil technical quality findings into concise summaries suitable for non-technical stakeholders.

Experience maintaining test metrics and quality dashboards - using tools such as Azure DevOps, Jira, Confluence, or equivalent.

Comfortable presenting quality status and risks in team meetings and sprint ceremonies.

Tools & Platform Familiarity

Familiarity with Azure cloud services relevant to AI solutions: Azure OpenAI, Azure AI Search, Azure Databricks, and ADLS - sufficient to understand system architecture and design targeted test scenarios.

Experience with version control (Git) and collaborative development workflows.

Familiarity with logging and observability tools (e.g., Azure Application Insights) to support defect investigation and root cause analysis.

Other Requirements

Nice to Have

Hands-on experience with LLM evaluation frameworks such as RAGAS, DeepEval, or LLM-as-judge implementations.

Experience with performance/load testing tools (e.g., Locust, k6, or Azure Load Testing).

Exposure to Databricks notebooks or PySpark for data pipeline validation.

Experience testing ML model fairness and bias using tools such as Fairlearn or equivalent.

ISTQB certification or equivalent formal QA qualification.

Experience in a data engineering or AI engineering adjacent role - providing deeper empathy for the systems being tested.