Senior Software Test Engineer (Customer Insights & Analytics)

Purpose of the role

The role serves as the primary owner of test and quality control for software and AI products, safeguarding product reliability, accuracy, and performance across the development lifecycle and supporting internal and external stakeholders on testing, release readiness, and continuous quality improvement initiatives.

Scope of the role

Design, execute, and refine test frameworks to ensure software and AI products consistently meet functional, performance, and quality expectations. Drive defect root cause analysis and resolution across engineering, data science, and product teams, and support the development and implementation of test automation, reporting, and evaluation practices - with particular focus on the demands of media and advertising platforms.

Key responsibilities

• Design and implement end-to-end test strategies and test plans for software and AI products, covering functional, integration, regression, performance, security, and user acceptance testing.
• Build and maintain automated test frameworks and CI/CD-integrated test pipelines to enable fast, reliable releases at scale.
• Define evaluation methodologies for AI/ML and LLM-based products, including accuracy, precision/recall, hallucination, bias, fairness, latency, and cost benchmarks, as well as human-in-the-loop and offline/online evaluation pipelines.
• Develop test data strategies, including synthetic data generation, golden datasets, and prompt/response evaluation sets relevant to advertising and media use cases (e.g., audience targeting, creative optimisation, bid management, campaign performance analysis).
• Lead root cause analysis and corrective action for production defects, model regressions, and customer-reported issues; partner with engineering and data science teams to drive timely resolution.
• Establish Quality Control KPIs (e.g., defect density, escape rate, model drift, test coverage, MTTR) and translate quality data into meaningful, actionable insights for engineering and business stakeholders.
• Partner with product, engineering, data science, and ad operations teams to embed quality early in the development lifecycle and support new feature and model launches.
• Evaluate and adopt best practices, tools, and frameworks in software QA, ML testing, and LLM evaluation.

Foundational / Leadership Competencies

• Demonstrate clear and thorough analytical thinking.
• Able to multi-task and deliver against timelines.
• Good interpersonal and communication skills.
• Takes ownership of projects that are assigned.
• Able to work as an individual or as a team player to deliver high quality of work.
• Critical thinking and problem-solving skills are essential for this role.

Functional Competencies

• Strong understanding of software testing methodologies (functional, integration, regression, performance, security) and their application to complex AI systems.
• Proficiency in test automation frameworks and tools such as Selenium, Playwright, Cypress, PyTest, JUnit, TestNG, or equivalents, and experience integrating tests into CI/CD pipelines.
• Hands-on experience testing AI/ML and LLM-based systems, including model evaluation, prompt/response validation, A/B testing, and monitoring for drift, bias, and hallucination.
• Proficiency in at least one programming or scripting language (preferably Python, or JavaScript/TypeScript) and comfort working with SQL and APIs (REST/GraphQL).
• Familiarity with observability tooling (e.g., Grafana, Datadog, ELK) for test environment and production quality monitoring.
• Must possess sound engineering judgment to identify core quality risks, synthesise signals from disparate data sources, and translate findings into commercial and product insights for key stakeholders.

Bachelor or Masters Degree in Computer Science, Software Engineering, Information Systems or a related field. Certifications such as ISTQB Advanced Level, Certified Agile Tester, or relevant cloud/ML certifications are a plus.

Minimum 5 years of hands-on experience in software QA/QC or test engineering, including significant experience testing data-driven, AI/ML, or LLM-based products in production. Prior working experience in the media, advertising, or marketing tech industry is strongly preferred.