{"id":3329099,"url":"https://alion.io/job/aviva-canada-ai-quality-engineer","title":"AI Quality Engineer","company":{"id":1074183,"name":"Aviva Canada","domain":"aviva.ca","url":"https://alion.io/company/aviva-ca","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":{"grade":"A","score":91,"open_postings":8,"ghost_share":0,"stale_share":0.375,"repost_share":0,"time_to_fill_p50_days":36,"computed_at":"2026-11-02T06:13:15Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":null,"work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Warsaw, Poland"],"countries":["PL"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":63000,"max_usd":118000,"period":"year","method":"role_country_seniority_unknown","sample_n":267},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"LLM","optional":false},{"name":"LLM Guardrails","optional":false},{"name":"Python","optional":false},{"name":"Function Calling","optional":true},{"name":"Git","optional":true}],"status":"live","first_seen_at":"2026-10-26T00:00:00Z","employer_posted_date":"2026-10-26","last_verified_at":"2026-11-02T16:41:54Z","board_verified":true,"closed_at":null,"days_open":7,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":7},"description":"Hi, we're glad you're here! We are hiring to our Aviva Services Excellence Centre! Take a look at our job description - maybe it suits you or one of your friends?\nWe are looking for an AI Quality Engineer to join Aviva Services Excellence Centre (ASEC) and help build confidence in the quality, safety and reliability of AI-powered products.\nThis is a hands-on role for someone who enjoys understanding how AI systems behave, identifying weaknesses before they become customer problems, and creating reliable ways of measuring whether AI solutions are actually delivering the outcomes they were designed for. You will work across conversational AI, autonomous agents and other AI-powered systems, partnering closely with engineers, product teams and business stakeholders to ensure solutions are robust, trustworthy and ready for production use.\nWhat will you be doing?\nTranslating business expectations, user outcomes and operational risks into measurable evaluation criteria. \nDefining what successful, partially successful and unsuccessful system behaviour looks like, including the gradient between clear success and clear failure. \nDesigning evaluation approaches for conversational, multi-turn and non-deterministic AI systems. \nBuilding and maintaining representative evaluation datasets covering common, rare, boundary and high-risk scenarios. \n Assessing dataset quality, including coverage, class balance, duplicates, data leakage, label quality, synthetic-versus-production differences and dataset freshness. \nSelecting appropriate manual, deterministic, model-based and statistical evaluation methods. \nAnalysing errors by category, user journey, risk level and likely root cause. \nDistinguishing model failures from failures in retrieval, chunking, ranking, prompts, tools, guardrails, integrations, source data and evaluators. \nValidating automated evaluators, including LLM-as-a-Judge, against human judgement. \nDetecting blind spots, noise, bias and misleading aggregation in evaluation results. \nCreating baselines, controlled comparisons and regression tests that allow meaningful comparison between system versions. \nPrioritising tests according to the probability and impact of failure rather than coverage alone. \nCommunicating results, uncertainty, limitations and residual risks in a form that supports release and product decisions. \nProactively identifying emerging failure modes based on system changes, production observations and research. \nWhat we're looking for:\nProfessional experience in quality engineering, software testing, test automation, data science, software engineering or a related field (we can consider different seniority levels). \nExperience or strong interest in working with AI-powered solutions, either through developing, testing, evaluating or supporting them. \nGood Python proficiency and experience writing clean, maintainable code for analysis, automation or testing purposes. \nInterest in understanding how AI systems behave, how reliable they are, and how their quality can be measured. \nStrong analytical and problem-solving skills, with the ability to investigate issues and identify root causes. \nAbility to translate business objectives, customer outcomes and risks into practical evaluation approaches. \nStrong communication skills and the ability to explain findings to both technical and non-technical stakeholders. \nA curious, proactive mindset and a genuine interest in AI, emerging technologies and system robustness. \nNice to have:\nExperience evaluating, testing or monitoring AI-powered systems in production environments. \nWorking with structured data, especially JSON, logs and tabular evaluation results. \nUnderstanding of LLM behaviour, tokenisation, context limits, parameters and non-determinism. \nUnderstanding of multi-turn context, RAG, grounding, faithfulness, hallucinations, guardrails, attacks (like prompt injection etc.) and tool/function calling. \nSoftware engineering practices: Git, modular code, configuration management and automated tests. \nIn Aviva Services Excellence Centre, we provide:\nContract of employment \nPerformance Bonus \nPrivate medical care (ENEL-MED) \nCafeteria system of benefits (e.g. MultiSport card) \nAdditional parent privileges: - baby leave - 6 weeks leave for second parent - shortened working time after parental leave \nEmployer-funded group insurance \nEmployee Shares Plan \nExtra day off for birthday \nPaid time off to volunteer \nAdditional leave acquired over the years in the company (even 5 days more in a year) \nWellbeing Leave - additional two weeks paid leave after 10 years of service \nPsychological support consultations \nAccess to the e-learning platforms \nInternal employee referral program \nHybrid working model from Poland with a few days in Warsaw office per month\nAttractive office location in Warsaw - Gdański Business Center \nAdditional entitlements for people with disabilities - Work Without Barriers \nWork environment that promotes diversity - Diversity Charter \nDo you see yourself in Aviva Services Excellence Centre? Be sure to apply!\nAviva is for everyone!\nWe’ve been ranked the #1 Best Workplace in Poland for a second year in a row and a top choice for Millennials by Great Place to Work®!\nAs a signatory to the Diversity Charter, Aviva applies a policy of equal treatment and creates a work environment that is friendly to people with disabilities. We do not require from candidates the date of birth, marital status and attached photos to your CV. We encourage all individuals regardless of gender to apply.\nMost of our people are smart working - spending also time in our office - combining the benefits of flexibility, with time together with colleagues.\nWe’re inclusive - we welcome applications from people with diverse backgrounds and experiences.\nExcited but not sure you tick every box? Research tells us that women, particularly, feel this way. So, regardless of gender, why not apply and try!\nTo find out more about current job offers at Aviva take a look here.","description_format":"text","description_chars":6010,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Cafeteria","Hybrid work","Parental leave"],"hiring_locations":[{"name":"Poland","iso":"PL","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Property & Casualty Insurance"],"lifecycle":[{"event":"open","at":"2026-10-27T11:25:49Z"}],"visa":[],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":7,"expected_fill_days":36,"reasons":["conf:9","velocity","win:early"],"computed_at":"2026-11-02T06:13:15Z"},"pay":null,"html_url":"https://alion.io/job/aviva-canada-ai-quality-engineer","json_url":"https://alion.io/job/aviva-canada-ai-quality-engineer.json","meta":{"generated_at":"2026-11-02T18:17:01Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":25522,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-03T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1074183},"rest":"https://alion.io/mcp/rest/get_company?id=1074183"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Faviva-canada-ai-quality-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Faviva-canada-ai-quality-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Faviva-canada-ai-quality-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/aviva-canada-ai-quality-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Faviva-canada-ai-quality-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3329099}},{"tool":"search_jobs","arguments":{"company":"aviva.ca","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3329099},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}