{"id":2610837,"url":"https://alion.io/job/jayanalytix-qa-lead-ai-systems-models-testing","title":"QA Lead — AI Systems & Models Testing","company":{"id":4416722,"name":"Jayanalytix","domain":"jayanalytix.ca","url":"https://alion.io/company/jayanalytix","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Breezy","truth_index":null},"role":"QA","role_family":"QA","seniority":"lead","employment_type":"contractor","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"explicit","locations":["Montreal, Canada"],"countries":["CA"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":56000,"max_usd":118000,"period":"year","method":"role_seniority_country_cell","sample_n":9},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Embeddings","optional":false},{"name":"LangChain","optional":false},{"name":"LangGraph","optional":false},{"name":"LLM","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"RAG","optional":false},{"name":"Tokenization","optional":false},{"name":"AWS","optional":true},{"name":"Azure","optional":true},{"name":"CI/CD","optional":true},{"name":"DeepEval","optional":true},{"name":"GCP","optional":true},{"name":"Ragas","optional":true}],"status":"live","first_seen_at":"2026-05-12T15:00:21Z","employer_posted_date":"2026-05-12","last_verified_at":"2026-10-15T19:34:37Z","board_verified":true,"closed_at":null,"days_open":156,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":156},"description":"QA Lead - AI Systems & Models Testing\nQuality Assurance\nArtificial Intelligence\nContract Position\n\nContract\nMontreal, QC\nAI / ML Testing\nLLM / RAG / LangChain\nABOUT THE ROLE\nWe are seeking an experienced QA Lead with deep expertise in AI systems testing to join our team on a contract basis in Montreal, Québec. This role sits at the intersection of quality engineering and artificial intelligence, requiring hands-on proficiency in LLM behavior analysis, RAG pipeline validation, and modern AI orchestration frameworks. You will own the end-to-end test strategy for complex AI products and help define quality standards in a rapidly evolving space.\nMUST-HAVE SKILLS\n Proven QA leadership experience designing and executing test strategies for AI/ML systems or LLM-powered applications.\nStrong understanding of LLM internals: tokenization, embeddings, attention mechanisms, and inference behavior to anticipate and diagnose failure modes.\nHands-on experience with prompt engineering - constructing effective prompts, detecting hallucinations, and evaluating outputs across accuracy, tone, coherence, and bias dimensions.\nExperience testing RAG pipelines and knowledge base integrations, including validation of data quality and retrieval accuracy as they impact model outputs.\nFamiliarity with vector database mechanics: similarity search thresholds, embedding drift, near-duplicate documents, and sparse vs. dense embeddings.\nPractical experience with LangChain and/or LangGraph - able to read chain/graph construction code, identify failure points, and write test harnesses.\nAbility to validate MCP (Model Context Protocol) integration points, including tool availability and error-handling scenarios.\nProficiency applying generative AI evaluation metrics and establishing quality thresholds appropriate for production AI systems.\nExcellent written and verbal communication in English; bilingualism (English/French) is a plus for the Montreal market.\nNICE-TO-HAVE SKILLS\nExperience with bias detection and safety testing frameworks for AI systems.\nExposure to performance and scalability testing of vector databases under high load.\nFamiliarity with CI/CD pipelines for ML model deployment and automated regression testing.\nKnowledge of responsible AI principles and AI governance frameworks.\nContributions to or experience with open-source AI testing or evaluation tooling (e.g., DeepEval, Ragas, PromptFlow).\nBackground in data engineering or data quality practices relevant to AI pipeline inputs.\nCloud platform experience (AWS, Azure, or GCP) in the context of deploying or testing AI workloads.\nKEY RESPONSIBILITIES\nLead design and execution of comprehensive test strategies across AI systems, including prompt evaluation, output quality assessment, and bias/safety analysis.\nDevelop and maintain test harnesses for LangChain and LangGraph-based applications; review chain and graph construction code to proactively surface integration risks.\nValidate RAG pipeline integrity - data ingestion, chunking, retrieval accuracy, and embedding consistency - and define edge-case coverage for vector database interactions.\nEstablish and track generative AI quality metrics and thresholds; report on model output quality across multiple evaluation dimensions.\nCollaborate with ML engineers, data scientists, and product teams to embed quality practices throughout the AI development lifecycle.\nDocument test findings clearly for both technical and non-technical stakeholders.\nContract position based in Montreal, Québec, Canada\nOn-site / Hybrid","description_format":"text","description_chars":3542,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[{"language":"English","level":"Upper-Intermediate (B2)","optional":false}]},"benefits":[],"hiring_locations":[{"name":"Canada","iso":"CA","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence"],"lifecycle":[{"event":"open","at":"2026-10-15T10:57:52Z"}],"visa":[],"liveness":{"score":8,"band":"cold","label":"Long shot","p_open":1,"p_active":0.287,"p_room":0.28,"age_days":156,"expected_fill_days":24,"reasons":["conf:3","win:tail","crowd:"],"computed_at":"2026-10-15T22:35:57Z"},"pay":null,"html_url":"https://alion.io/job/jayanalytix-qa-lead-ai-systems-models-testing","json_url":"https://alion.io/job/jayanalytix-qa-lead-ai-systems-models-testing.json","meta":{"generated_at":"2026-10-15T22:35:57Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":34249,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-16T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":4416722},"rest":"https://alion.io/mcp/rest/get_company?id=4416722"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fjayanalytix-qa-lead-ai-systems-models-testing"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fjayanalytix-qa-lead-ai-systems-models-testing"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fjayanalytix-qa-lead-ai-systems-models-testing"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/jayanalytix-qa-lead-ai-systems-models-testing\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fjayanalytix-qa-lead-ai-systems-models-testing"}]}