{"id":2771931,"url":"https://alion.io/job/ust-qa-engineer-generative-ai-agentic-ai","title":"QA Engineer – Generative AI & Agentic AI","company":{"id":1774626,"name":"UST","domain":"ust.com","url":"https://alion.io/company/ust-com","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"RippleHire","truth_index":null},"role":"QA","role_family":"QA","seniority":null,"employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":12500,"max_usd":33000,"period":"year","method":"role_country_seniority_unknown","sample_n":62},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"Hallucination","optional":false},{"name":"LLM","optional":false},{"name":"OpenAI","optional":false},{"name":"Platform Engineering","optional":false},{"name":"RAG","optional":false},{"name":"Snowflake","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-10-07T00:00:00Z","employer_posted_date":"2026-10-17","last_verified_at":"2026-10-17T16:24:56Z","board_verified":true,"closed_at":null,"days_open":10,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":10},"description":"We are looking for an experienced QA Engineer - Generative AI & Agentic AI to lead quality assurance activities for GenAI and Agentic AI solutions.\nThis role goes beyond traditional functional testing. The successful candidate will be responsible for validating the correctness, reliability, quality, and production-readiness of AI-generated outputs, including multi-step agent workflows, Text-to-SQL pipelines, Retrieval-Augmented Generation (RAG) systems, and LLM-based applications.\nThe ideal candidate will have hands-on, production-grade experience testing GenAI/Agentic AI applications end-to-end and should have successfully supported at least one production release of a GenAI or Agentic AI solution.\nDesign, develop, and execute comprehensive test strategies for GenAI and Agentic AI applications, covering functional, non-functional, integration, and AI output-quality testing.Test multi-step agentic workflows, validating agent orchestration, tool/function calls, task completion, chained decision-making, reasoning consistency, and outputs across individual agent hops.Evaluate AI-generated outputs for correctness, relevance, coherence, hallucination, consistency, and reliability.Natural language to SQL generation accuracySQL syntax and logicSchema/table/column alignmentQuery executionResult accuracy against Snowflake data modelsRetrieval relevanceContext groundingChunking effectivenessContext utilization\nFaithfulness of generated responses to retrieved information\n\nApply GenAI evaluation metrics and methodologies such as:Embedding similarity\nROUGE\nBLEU\nLLM-as-a-Judge\nOther relevant GenAI quality and accuracy metrics\n\nDesign and maintain LLM-as-a-Judge evaluation frameworks and scoring rubrics for scalable and automated assessment of GenAI outputs.\nCreate and maintain golden datasets, evaluation datasets, regression suites, and reusable AI testing frameworks.\nDevelop test harnesses/scripts for batch evaluation and automated quality validation of LLM-generated outputs.\nOwn QA activities for end-to-end production releases, including test planning, execution, regression testing, release quality assessment, sign-off, and post-production monitoring.\nPartner closely with Data Science, Platform Engineering, Product, and other stakeholders to establish acceptance criteria, quality gates, and go/no-go release benchmarks.Identify LLM-specific edge cases and failure modes, including hallucinations, prompt injection risks, bias, inconsistent responses, and agent/tool failures.Document and track defects through resolution and provide clear evidence of AI application quality.Maintain traceability of test coverage, evaluation results, quality metrics, and release reports for stakeholders and audit requirements.Mandatory Skills &\nGenAI / Agentic AI TestingStrong, demonstrable hands-on experience with:\nGenerative AI / LLM application testingAgentic AI and multi-step agent workflowsAgent orchestration and tool/function-call validationText-to-SQL testingRetrieval-Augmented Generation (RAG) testingLLM-as-a-Judge evaluationPrompt and LLM behavior validationHallucination and response-grounding assessmentContext-window and context-handling conceptsAI output-quality evaluation beyond traditional pass/fail testingPlatformsDataiku: Strong hands-on experience is highly preferred and considered the first preferenceExposure to platforms such as OpenAI workspace, Leena AI, or similar GenAI/Agentic AI platformsData / DatabaseStrong hands-on working experience with SnowflakeAbility to understand and validate data models, schemas, generated SQL queries, and query resultsAI EvaluationPractical experience with at least some of the following:\nEmbedding SimilarityRO","description_format":"text","description_chars":3692,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","IT Consulting & Digital Transformation","IT Outsourcing & Dedicated Teams"],"lifecycle":[{"event":"open","at":"2026-10-17T06:35:48Z"}],"visa":[],"liveness":{"score":51,"band":"ok","label":"Likely open","p_open":1,"p_active":0.677,"p_room":0.75,"age_days":10,"expected_fill_days":14,"reasons":["conf:2","win:late","comp:brand"],"computed_at":"2026-10-17T18:50:18Z"},"pay":null,"html_url":"https://alion.io/job/ust-qa-engineer-generative-ai-agentic-ai","json_url":"https://alion.io/job/ust-qa-engineer-generative-ai-agentic-ai.json","meta":{"generated_at":"2026-10-17T18:50:18Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":1842,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1774626},"rest":"https://alion.io/mcp/rest/get_company?id=1774626"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fust-qa-engineer-generative-ai-agentic-ai"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fust-qa-engineer-generative-ai-agentic-ai"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fust-qa-engineer-generative-ai-agentic-ai"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/ust-qa-engineer-generative-ai-agentic-ai\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fust-qa-engineer-generative-ai-agentic-ai"}]}