{"id":1428952,"url":"https://alion.io/job/novara-senior-ai-quality-engineer","title":"Senior AI Quality Engineer","company":{"id":679743,"name":"Novara","domain":"novara.com","url":"https://alion.io/company/novara","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Lever","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Canada"],"countries":["CA"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":79000,"max_usd":145000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":74},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"CI/CD","optional":false},{"name":"Cypress","optional":false},{"name":"GitHub Actions","optional":false},{"name":"LLM","optional":false},{"name":"Playwright","optional":false},{"name":"Postman","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"Rest API","optional":false},{"name":"Scrum","optional":false},{"name":"Shift-Left Security","optional":false},{"name":"TypeScript","optional":false},{"name":"AWS","optional":true},{"name":"Braintrust","optional":true},{"name":"Datadog","optional":true},{"name":"DeepEval","optional":true},{"name":"Jira","optional":true},{"name":"Langfuse","optional":true},{"name":"LangSmith","optional":true},{"name":"RAG","optional":true},{"name":"Ragas","optional":true},{"name":"Synthetic Data","optional":true}],"status":"live","first_seen_at":"2026-09-28T19:33:10Z","employer_posted_date":"2026-09-28","last_verified_at":"2026-10-29T03:23:43Z","board_verified":true,"closed_at":null,"days_open":30,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":30},"description":"Position Description:\nAI is core to where our product is heading, and the quality bar for it has to be as high as anything else we ship. This role owns that bar. You will define what \"working\" means for non-deterministic systems, build the tooling to prove it, and give our teams the confidence to move faster on AI-powered features and agent-based products.\nYou will partner closely with product, engineering, and the broader QA team to bring rigor to how we test LLM-driven and agentic workflows, while also contributing to traditional automation coverage where it matters.\nThis is a hands-on IC role reporting into the QA Manager, with high visibility into AI initiatives across the company.\nResponsibilities:\nDesign and maintain evaluation frameworks for LLM outputs and agentic workflows, including regression suites, golden datasets, and scoring rubrics\nBuild test harnesses that catch hallucinations, tool-calling failures, prompt regressions, and unsafe or off-policy behavior before they reach production\nDefine measurable quality criteria for agent reliability: task completion, factual grounding, latency, cost, and reasoning quality\nIntegrate evaluation runs into CI/CD so model, prompt, and agent changes are gated the same way code changes are\nPartner with engineers on observability and tracing for agent runs, so failures are diagnosable rather than mysterious\nContribute to conventional API and end-to-end automation where AI features sit inside larger product flows\nHelp shape the team's shared playbook for testing AI features, and coach other QA engineers as agentic work spreads across scrum teams\nKnowledge, Experience, Requirements:\n6+ years in QA, SDET, or test automation, with real production automation shipping\nHands-on experience testing LLM-based or agentic systems: building evals, working with LLM-as-judge patterns, prompt regression testing, or agent trajectory analysis\nPrior experience in a shift-left, embedded QA model\nComfort with at least one modern automation stack (Playwright, Cypress, or similar) and a typed language, TypeScript preferred, Python fine\nDeep experience with test frameworks such as vitest, jest, or pytest, and comfort building custom test harnesses rather than only running off-the-shelf suites\nAPI-first testing mindset, including REST and Postman or equivalent\nFluency in HTTP-level API testing, including recording proxies and observing service-to-service traffic\nWorking knowledge of CI/CD pipelines, GitHub Actions a plus, and how to plug evals into them\nFamiliarity with cloud secret managers and disciplined handling of sensitive test data in restore-from-prod environments\nAbility to reason clearly about probabilistic systems: variance, sample sizes, confidence, and when a flaky result is signal rather than noise\nPreferred Qualifications:\nNot required, but they will stand out:\nExperience with eval tooling such as Langfuse, Braintrust, LangSmith, Ragas, or DeepEval\nFamiliarity with RAG systems, vector stores, or tool-calling frameworks\nBackground in test data strategy for AI, including synthetic data generation\nExposure to Datadog or a similar observability platform\nTech Stack:\nTypeScript, Playwright, vitest, Postman, GitHub Actions, Jira/Xray, Datadog, AWS, plus emerging AI evaluation tooling.\nCompensation:\nAnnual Base Salary Range of 100k-130k CAD\nAnnual Bonus Opportunity of 10%\nLocation:\nWe’re a remote-first team that gets together in our North York office in Toronto every Tuesday, and as needed for in-person team events. The person in this role will be willing and able to commute to the office one day a week, and comfortable working remotely the rest of the time.","description_format":"text","description_chars":3654,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"Canada","iso":"CA","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Manufacturing & Industrial Software"],"lifecycle":[{"event":"open","at":"2026-09-29T00:45:40Z"}],"visa":[],"liveness":{"score":75,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.831,"p_room":0.9,"age_days":29,"expected_fill_days":77,"reasons":["conf:0","win:mid"],"computed_at":"2026-10-28T05:59:10Z"},"pay":null,"html_url":"https://alion.io/job/novara-senior-ai-quality-engineer","json_url":"https://alion.io/job/novara-senior-ai-quality-engineer.json","meta":{"generated_at":"2026-10-29T04:24:49Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":6837,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-30T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":679743},"rest":"https://alion.io/mcp/rest/get_company?id=679743"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fnovara-senior-ai-quality-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fnovara-senior-ai-quality-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fnovara-senior-ai-quality-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/novara-senior-ai-quality-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fnovara-senior-ai-quality-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":1428952}},{"tool":"search_jobs","arguments":{"company":"novara.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":1428952},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}