{"id":93771,"url":"https://alion.io/job/rootly-ai-quality-engineer","title":"AI Quality Engineer","company":{"id":35600,"name":"Rootly","domain":"rootly.com","url":"https://alion.io/company/rootly","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Gem","truth_index":{"grade":"B","score":79,"open_postings":15,"ghost_share":0.267,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":286,"computed_at":"2026-10-16T05:45:00Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Toronto, Canada"],"countries":["CA"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":76000,"max_usd":162000,"period":"year","method":"role_country_seniority_unknown","sample_n":181},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Canva","optional":false},{"name":"Dropbox","optional":false},{"name":"Figma","optional":false},{"name":"GitHub","optional":false},{"name":"Incident Management","optional":false},{"name":"LLM","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Python","optional":false},{"name":"Braintrust","optional":true},{"name":"LangSmith","optional":true},{"name":"Playwright","optional":true}],"status":"live","first_seen_at":"2026-06-16T20:55:07Z","employer_posted_date":"2026-06-16","last_verified_at":"2026-10-17T00:14:14Z","board_verified":true,"closed_at":null,"days_open":122,"trust":{"level":"ok","repost_count":0,"flags":["company_stale"],"days_open":121},"description":"About Rootly\nAt Rootly, we are on a mission to be the go-to way companies respond when things go wrong, helping every organization be more reliable. We do this by building an industry-leading incident management platform that allows companies around the world to consistently and quickly resolve incidents. We are not simply transforming an industry, we are carving an entirely new +$B segment ourselves and need incredible talent to achieve this ambitious goal together.\nCustomers love Rootly. Some of the fastest growing companies around the world such as NVIDIA, Figma, Canva, Tripadvisor, Squarespace and more rely on Rootly to power their critical incident management process. They obsess over our delightful enterprise-ready platform and unique partnership model. See why our customers have reviewed us 5 stars on G2.\nInvestors love Rootly. We are backed by some of the most respected funds in the world from Y Combinator to operators like the CTO of Dropbox and GitHub. We'd be happy to disclose our entire funding and profitability picture live during the interview. As a culture we relentlessly put transparency first. We conduct monthly financial reviews as a team so everyone has a pulse on the health of the business and publish what we are building in our weekly changelog.\nRootly is building the AI-native future of incident management, and we need someone who can push our AI to its limits before our customers do. As our AI Quality Engineer, you'll own the evaluation and optimization of Rootly's agentic AI features -- designing test scenarios, running adversarial prompts, interpreting outputs, and working directly with engineering and product to close the loop on performance.\nThis isn't traditional QA. You'll spend your days thinking like an attacker, a confused user, and a power user all at once -- probing how our AI agents reason, make decisions, and handle edge cases across complex incident workflows.\nWhat you'll do\nDesign and execute prompt-based test scenarios that cover happy paths, edge cases, and adversarial inputs across Rootly's agentic AI features\nEvaluate AI outputs for accuracy, relevance, consistency, and alignment with expected workflow behaviour\nBuild and maintain an evaluation framework; structured test libraries, scoring rubrics, and regression suites to track AI performance over time\nIdentify failure modes, hallucinations, reasoning gaps, and unexpected agent behaviours; document findings and work with engineers to resolve them\nPartner with Product and Engineering on new AI feature releases, contributing to acceptance criteria and quality gates before launch\nDefine and track quality metrics (accuracy rates, failure frequency, regression trends) and report findings to stakeholders\nStay current on LLM evaluation techniques, prompt engineering best practices, and agentic testing methodologies\nWhat we're looking for\n+5 years in QA, product operations, AI/ML evaluation, or a closely related role\nHands-on experience testing or evaluating LLM-powered or agentic AI products\nStrong prompt engineering instincts -- you understand how wording, context, and structure affect model behaviour\nComfortable writing scripts or working with evaluation tools (Python a plus; not required to be a full-stack engineer)\nSharp analytical thinking; you can spot a subtle reasoning failure and articulate exactly why it's a problem\nClear written communicator; able to translate AI behaviour findings for both technical and non-technical audiences\nNice to have\nFamiliarity with incident management, DevOps, or IT operations workflows is a strong asset\nExperience with evaluation frameworks (e.g. LangSmith, PromptFlow, Braintrust, or similar)\nExposure to red-teaming or adversarial testing of AI systems\nComfortable writing E2E tests with Playwright\nBackground working at a B2B SaaS or developer-tools company\nFamiliar with mobile app testing (iOS/Android)\nWhy Rootly?\nWe’re not just another startup. We’re building something category-defining and want teammates who crave ownership, love solving hard problems, and thrive in a high-bar, high-impact environment.\nHere’s what you can expect when you join Rootly:\nCompetitive compensation and early equity in a fast-growing, venture-backed company.\nComprehensive medical, dental, and vision coverage.\n3 weeks of vacation, plus unlimited sick and mental health days, and a company-wide end-of-year shutdown to recharge.\n$500 stipend for home office setup.\nUnlimited token usage and access to AI tools\nA fast-moving, high-impact environment where your leadership and ideas directly shape the future of the company.\nIf this sounds like the kind of challenge and opportunity you’re looking for, apply now and let’s build something great together.\nRootly is an equal opportunity employer. We aim to create an environment where every team member at Rootly feels like they belong so they can have a greater impact on our business and customers. We do not discriminate on the basis of race, religion, colour, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.","description_format":"text","description_chars":5092,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Equity","Home office"],"hiring_locations":[{"name":"Canada","iso":"CA","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Incident Response","Observability & APM"],"lifecycle":[{"event":"open","at":"2026-06-16T20:55:07Z"}],"visa":[],"liveness":{"score":55,"band":"ok","label":"Likely open","p_open":1,"p_active":0.765,"p_room":0.72,"age_days":121,"expected_fill_days":286,"reasons":["conf:3","win:mid","crowd:"],"computed_at":"2026-10-16T05:45:00Z"},"pay":null,"html_url":"https://alion.io/job/rootly-ai-quality-engineer","json_url":"https://alion.io/job/rootly-ai-quality-engineer.json","meta":{"generated_at":"2026-10-17T04:21:13Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":5695,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":35600},"rest":"https://alion.io/mcp/rest/get_company?id=35600"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Frootly-ai-quality-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Frootly-ai-quality-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Frootly-ai-quality-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/rootly-ai-quality-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Frootly-ai-quality-engineer"}]}