Senior Manager, QA & Analytic Engineer (Contract till 31st Dec 2028)

[What the role is]

About the Role

You'll be a key member of the Child & Youth Programme & Partnership - Digital Enablement & Analytics team, sitting at the intersection of AI quality assurance and data analytics. Your work will directly shape the quality, safety, and trustworthiness of a Parent Engagement Digital Assistant - an AI-powered tool reaching parents across channels - while also driving the team's broader data and dashboard strategy.

This is a dual-function role: part QA Tester and AI Evaluator, part Data Analyst. You'll own the end-to-end testing lifecycle of a live AI product, evaluate emerging agentic AI tools for potential adoption, and translate conversation data into insights that drive continuous improvement.

[What you will be working on]

What You'll Do

AI Quality Assurance

• Design and execute structured test cases covering accuracy, tone, safety, and data privacy. This includes simulating edge cases, harmful queries, and abuse attempts.
• Perform full regression testing after every prompt update, model change, or deployment. You will own the go/no-go sign-off before every release.

Third-Party AI Evaluation

• Independently research and assess external agentic AI tools using a structured scoring framework across accuracy, safety, cost, and integration feasibility,
• Produce clear recommendation reports for the product and leadership team.

Platform & Cross-Channel Testing

• Verify consistent AI behaviour, formatting, and response quality across various identified channels and test the embeddable chat widget across browsers and devices.

Data Analysis & Insights

• Analyse conversation logs to surface patterns, build and maintain live performance dashboards, and conduct user behaviour analysis by platform.

[What we are looking for]

What an ideal candidate looks like

We are looking for a dynamic and independent individual who is comfortable working in a fast-moving digital product environment. You should be analytical, curious about AI-enabled solutions, and able to balance technical quality with user needs, safety, and public-sector considerations.

Candidates should preferably have at least 5 years of relevant experience in quality assurance, software testing, data analysis, digital product operations, or related fields. Experience with conversational AI, digital assistants, chatbots, or AI-enabled products would be advantageous.

You should also possess:

• Strong analytical and problem-solving skills, with the ability to assess issues systematically and recommend practical improvements.
• Experience in designing test cases, conducting user acceptance testing, regression testing, or quality assurance for digital products.
• Strong awareness of accuracy, bias, safety, and data privacy in AI - particularly in contexts where misinformation can directly affect families and children - is essential.
• Advanced proficiency in at least one analytical tool such as Python, SQL, Power BI, Tableau, or equivalent tools.
• Good written and verbal communication skills, with the ability to explain technical issues clearly to both technical and non-technical stakeholders.
• Strong stakeholder management skills, with the ability to work across teams, vendors, and internal stakeholders.
• Ability to manage multiple workstreams, adapt to changing priorities, and deliver within tight timelines.
• Resourcefulness, sound judgement, and attention to detail, especially when working on products that may impact parents, children, and youth.

Candidates with academic training in data analytics, computer science, information systems, engineering, statistics, public health, business analytics, or related disciplines would be advantageous

Contract till 31st December 2028.