{"id":1748618,"url":"https://alion.io/job/calgary-ab-ai-test-engineer-vlabs","title":"AI Test Engineer - VLabs","company":{"id":1862112,"name":"Vialto Partners","domain":"vialtopartners.com","url":"https://alion.io/company/vialtopartners","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":{"grade":"B","score":79,"open_postings":45,"ghost_share":0,"stale_share":0.844,"repost_share":0,"time_to_fill_p50_days":40,"computed_at":"2026-10-13T05:45:00Z"}},"role":"QA","role_family":"QA","seniority":"senior","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":11500,"max_usd":29000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":20},"experience_years_min":7,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AWS","optional":false},{"name":"AWS Bedrock","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"DeepEval","optional":false},{"name":"Hallucination","optional":false},{"name":"HTTPX","optional":false},{"name":"LangChain","optional":false},{"name":"LlamaIndex","optional":false},{"name":"LLM","optional":false},{"name":"OCR","optional":false},{"name":"Postman","optional":false},{"name":"Promptfoo","optional":false},{"name":"Python","optional":false},{"name":"SQL","optional":false},{"name":"VLM","optional":false}],"status":"live","first_seen_at":"2026-09-07T00:00:00Z","employer_posted_date":"2026-09-07","last_verified_at":"2026-10-13T16:36:01Z","board_verified":true,"closed_at":null,"days_open":36,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":36},"description":"About Vialto Labs (VLabs)\nVialto Labs (VLabs) is responsible for redesigning how work is delivered in the tax and immigration service lines, as well as driving operational efficiency across Vialto’s functionalareas using AI. The team builds and deploys novel AI-enabled solutions that directly improve productivity and increase delivery quality for our clients. VLabs isaccountable for rapidly turning innovative experiments into production-ready deliverables at scale and embedding them into day-to-day operations. This team focuses on the highest-impact workflows, creating standardized, repeatable capabilities that can be deployed globally. Operating with a mandate for speed and measurable outcomes, VLabs worksalongside service line, product,and platform leaders.\nAbout the Role\nAI Test Engineering isa hands-on role within VLabs QualityEngineering, responsible for validating theperformance, reliability, and integrity of AI-enabled solutions in production environments. Thisrole operates atthe intersection of AI engineering and quality assurance, ensuring that outputs from LLMs, OCR pipelines, document classification models, and agentic workflows perform as expected at scale and meet defined business performance thresholds. Workingclosely with the Programme TestManager and partnering with engineering, product, and delivery teams, this role translates AI testing strategy into executable frameworks, evaluation pipelines, and reusable assets embedded into the delivery lifecycle.\nSuccess requires independent execution, strong technical depth, and the ability to proactively identify risks,patterns, and performance gaps while enabling rapid, production-grade deployment of AI capabilities.\nKey Responsibilities\nAI Evaluation & Test Design\nTranslate AI testing strategy into executable test scenarios across LLM outputs, document classification, extraction accuracy, agent workflows, and edge cases\n\nDesign adversarial and boundary test inputs to expose hallucination, misclassification, and failure modes\n\nValidate AI outputs for structure, consistency, accuracy, and production readiness against defined performance thresholds\n\nEvaluation Engineering & Automation\nBuild reusable Python-based evaluation frameworks, including output validation, hallucination detection, and scoring mechanisms\n\nDevelop parameterized test scripts reusable across features, models, and releases\n\nImplement AI-as-Judge frameworks, including prompt design, scoring logic, and calibration of evaluation reliability\n\nEmbed evaluation frameworks into CI/CD pipelines to support continuous testing and deployment\n\nDrift Detection & Quality Monitoring\nDesignand operate driftdetection frameworks using fixed baseline datasets and scheduled re-evaluation\n\nEstablish thresholds to distinguish acceptable variation from performance degradation\n\nEnable release gatingby identifying regressionsprior to production deployment\n\nGround Truth & Data Quality\nBuildand maintain groundtruth datasets in partnership with subject matter experts\n\nDefine standards for classification, extraction accuracy, and acceptable outputcharacteristics\n\nContinuously update datasets to reflect evolving business requirements and use cases\n\nWorkflow & Integration Testing\nTest end-to-end agenticworkflows, validating dataintegrity, error propagation, and fallback behavior\n\nPerform API-level testing of AI pipeline endpoints using Python and Postman/Newman\n\nValidate data persistence and integrity across system layers using SQL\n\nPartner with engineering teams to ensure testability, observability, and system reliability\n\nStandardization & Scaling\nDefine and scale standardized AI evaluation patterns and reusable quality frameworksacross VLabs\n\nContribute to enterprise AI quality standards andreference architectures\n\nGovernance & Responsible AI\nEnsure adherence to Responsible AI, data privacy, and governance requirements\n\nSupport auditability, traceability, and transparency of AI outputs and evaluation processes\n\nStakeholder Enablement\nTranslate evaluation results into actionable insights for engineering, product, and business stakeholders\n\nSupport decision-making on model readiness, release risk, and performance trade-offs\n\nProactively identify risks,patterns, and systemic issues and escalate appropriately\n\nQualifications & Experience\nProfessional Experience\n7+ years in software testing, including 2-3 years focused on AI/ML-enabled systems in production environments\n\nProven experience designing and executing AI evaluation frameworks and quality strategies\n\nStrong track record building ground truth datasets, drift detection systems, and scalable evaluation pipelines\n\nExperience testing multi-step agentic workflows and AI-driven automation systems\n\nExperience operating infast-paced, iterative delivery environments\n\nBackground in regulated or compliance-driven environments preferred\n\nTechnical Expertise\nAdvanced Python programming for evaluation frameworks, batch processing, and data analysis\n\nExperience with LLM evaluation tools such as deepeval,RAGAS, promptfoo,or similar\n\nStrong capabilities in:\n\nAI output validation, hallucination detection, and grounding checks\n\nDrift detection frameworks and statistical evaluation methods\n\nOCR, VLM, and document AI testing (classification, extraction, edge cases)\n\nAPI testing using Python (requests/httpx) and Postman/Newman\n\nSQL for data validation and pipeline integrity checks\n\nFamiliaritywith LangChain, LlamaIndex,or similar frameworks\n\nExperience with cloud AI platforms such as Azure AI Foundry or AWS Bedrock preferred\n\nOperating Capabilities\nAbilityto operate independentlyin fast-moving, ambiguous environments\n\nStrong analytical mindset with attention to detail and quality rigor\n\nAbility to balance speed and rigor in AI evaluation and delivery cycles\n\nProactive communicatorwho identifies risksand drives resolution\n\nAbility to translate technical findings into business-relevant insights\n\nEducation\nBachelor’s degree required\n\nAdvanced degree in Computer Science, Data Science, or related field preferred\n\nAdditional Information:\nThis job is based in our Bangalore office with the possibility of hybrid work mode\n\nWe are an equal opportunity employer that does not discriminate on the basis of any legally protected status\n\nPlease note, AI is used as part of the application process","description_format":"text","description_chars":6329,"description_truncated":false,"requirements":{"experience_years_min":7,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["Hybrid work"],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Professional Services","Business Travel"],"lifecycle":[{"event":"open","at":"2026-10-03T07:20:22Z"}],"visa":[],"liveness":{"score":38,"band":"fade","label":"Fading","p_open":1,"p_active":0.503,"p_room":0.75,"age_days":36,"expected_fill_days":40,"reasons":["conf:11","stale_co","win:late","comp:brand"],"computed_at":"2026-10-13T05:45:00Z"},"pay":null,"html_url":"https://alion.io/job/calgary-ab-ai-test-engineer-vlabs","json_url":"https://alion.io/job/calgary-ab-ai-test-engineer-vlabs.json","meta":{"generated_at":"2026-10-13T16:42:30Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"search","counted_by":"address","units_charged":0,"used_today":0,"day_limit":null,"remaining_today":null,"minute_limit":null,"resets_at":"2026-10-14T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1862112},"rest":"https://alion.io/mcp/rest/get_company?id=1862112"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcalgary-ab-ai-test-engineer-vlabs"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcalgary-ab-ai-test-engineer-vlabs"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcalgary-ab-ai-test-engineer-vlabs"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/calgary-ab-ai-test-engineer-vlabs\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcalgary-ab-ai-test-engineer-vlabs"}]}