{"id":4078151,"url":"https://alion.io/job/hexaware-technologies-test-analyst-ai-assurance","title":"Test Analyst - AI Assurance","company":{"id":1478,"name":"Hexaware Technologies","domain":"hexaware.com","url":"https://alion.io/company/hexaware-technologies","size_band":"5000+","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"Oracle","truth_index":{"grade":"B","score":75,"open_postings":13,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":22,"computed_at":"2026-11-11T05:49:22Z"}},"role":"QA","role_family":"QA","seniority":"middle","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Arize Phoenix","optional":false},{"name":"CI/CD","optional":false},{"name":"DeepEval","optional":false},{"name":"Galileo","optional":false},{"name":"Langfuse","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Ragas","optional":false}],"status":"live","first_seen_at":"2026-11-11T06:02:56Z","employer_posted_date":"2026-11-11","last_verified_at":"2026-11-11T16:09:45Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"Test Analyst - AI Assurance\nRole Overview\nStrong understanding of GenAI system architecture, including Retrieval-Augmented Generation (RAG) and Agent-based systems, with hands-on experience in implementing Python-based AI evaluation frameworks such as Ragas, DeepEval, and similar tools. Proven ability to perform Model and Agent quality evaluations using platforms like Arize, LangFuse, and Confident AI, while driving Responsible AI (RAI) validation and reporting key RAI metrics. Experienced in golden dataset curation, synthetic dataset generation, and establishing AI observability and tracing through OpenTelemetry (OTEL) instrumentation to enable effective monitoring, diagnostics, and continuous quality improvement of AI systems.\nKey Responsibilities\nAbility to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior\nDesign AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions\nExperience in Golden dataset curation & Synthetic dataset generation\nExperience in implementing AI evaluation frameworks like Ragas, DeepEval, etc.\nExperience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.\nExperience in instrumentation to establish AI Observability / tracing using OTEL libraries\nExperience in Integrating AI evaluation into CI/CD pipeline\nExperience in Responsible AI validation\nRequired Skills and Capabilities\nStrong understanding of GenAI System Architecture of RAG & Agents\nProficiency in Python & AI evaluation SDKs\nAI Evaluation Frameworks (Ragas, DeepEval)\nModel Evaluation\nAgent Evaluation\nResponsible AI (RAI) Validation\nRAI Metrics Reporting\nGolden Dataset Curation\nSynthetic Dataset Generation\nAI Observability\nAI Tracing\nOpenTelemetry (OTEL)\nLLM Evaluation\nAI Quality Engineering\nAI Governance\nAI Monitoring\nAI Testing & Assurance\nRoot Cause Analysis (AI Systems)\nKnowledge of regulatory considerations for AI systems\nRole-Specific Requirements\nProficiency in Python & AI evaluation SDKs\nStrong understanding of RAG & agentic system architecture\nAbility to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior\nDesign AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions\nExperience in Golden dataset curation & Synthetic dataset generation\nExperience in implementing AI evaluation frameworks like Ragas, DeepEval, etc.\nExperience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.\nExperience in instrumentation to establish AI Observability / tracing using OTEL libraries\nExperience in Integrating AI evaluation into CI/CD pipeline\nExperience in Responsible AI validation\nKnowledge of regulatory considerations for AI systems\nExperience Requirements\nMore than 4 years of experience in AI Assurance Testing","description_format":"text","description_chars":2928,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","IT Consulting & Digital Transformation","Business Process Outsourcing (BPO)","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-11-11T16:09:45Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":22,"reasons":["conf:2","win:early"],"computed_at":"2026-11-11T18:33:18Z"},"pay":null,"html_url":"https://alion.io/job/hexaware-technologies-test-analyst-ai-assurance","json_url":"https://alion.io/job/hexaware-technologies-test-analyst-ai-assurance.json","meta":{"generated_at":"2026-11-11T18:33:18Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":35707,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-12T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1478},"rest":"https://alion.io/mcp/rest/get_company?id=1478"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fhexaware-technologies-test-analyst-ai-assurance"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fhexaware-technologies-test-analyst-ai-assurance"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fhexaware-technologies-test-analyst-ai-assurance"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/hexaware-technologies-test-analyst-ai-assurance\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fhexaware-technologies-test-analyst-ai-assurance"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4078151}},{"tool":"search_jobs","arguments":{"company":"hexaware.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4078151},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}