{"id":3453417,"url":"https://alion.io/job/dataart-senior-ai-quality-engineer-hyderabad-based","title":"Senior AI Quality Engineer (Hyderabad-based)","company":{"id":46984,"name":"DataArt","domain":"dataart.com","url":"https://alion.io/company/dataart-com","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Schema","truth_index":{"grade":"B","score":75,"open_postings":18,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":27,"computed_at":"2026-11-01T06:10:25Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":20000,"max_usd":42000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":41},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"AWS","optional":false},{"name":"AWS Bedrock","optional":false},{"name":"AWS Bedrock AgentCore","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"Cursor","optional":false},{"name":"DeepEval","optional":false},{"name":"Embeddings","optional":false},{"name":"Function Calling","optional":false},{"name":"GCP","optional":false},{"name":"Git","optional":false},{"name":"GitHub","optional":false},{"name":"GitHub Actions","optional":false},{"name":"Hallucination","optional":false},{"name":"Jira","optional":false},{"name":"LambdaTest","optional":false},{"name":"LangChain","optional":false},{"name":"Langfuse","optional":false},{"name":"LangGraph","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"LLM Guardrails","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Playwright","optional":false},{"name":"Promptfoo","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Ragas","optional":false},{"name":"Reinforcement Learning","optional":false},{"name":"Rest API","optional":false},{"name":"Self-Healing","optional":false},{"name":"Semantic Kernel","optional":false},{"name":"TestRail","optional":false},{"name":"TypeScript","optional":false}],"status":"live","first_seen_at":"2026-09-30T00:00:00Z","employer_posted_date":"2026-10-28","last_verified_at":"2026-11-01T17:32:09Z","board_verified":true,"closed_at":null,"days_open":32,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":32},"description":"We are looking for a Senior AI Quality Engineer (L3+) with strong Python skills and hands-on experience testing AI and ML systems. You will analyze requirements, define evaluation strategies, build automated checks, validate models and outputs, test APIs and data pipelines, verify defects through resolution, and contribute to release readiness for your assigned product area. You will both evaluate AI systems and design, build, and operate QE agents for the software testing life cycle.\nAI-assisted work is part of everyday QE practice on this engagement, with Ragas, DeepEval, promptfoo, Cursor, and other AI and ML tools used for evaluation, scripting, test generation, analysis, and maintenance.\nYou will also build golden datasets, calibrate LLM-as-judge methods, test agentic workflows and RAG systems, implement self-healing and intelligent test approaches, and diagnose quality, safety, performance, and cost failures across distributed AI services.\nThis position is intended for Hyderabad-based candidates who are available to work from the office.\nThe client delivers AI agents, retrieval systems, models, and data pipelines across a broad technology landscape. Quality is part of delivery through evaluation gates, automated regression, model and data validation, security and privacy checks, and evidence that a release is ready.\nPython is the primary language for AI evaluation, model validation, API testing, data-pipeline quality, and QE agent design. The role combines deterministic automation with probabilistic evaluation to assess AI behavior across models, data, retrieval, tools, and integration surfaces, and designs and operates agents that support the software testing life cycle.\nAnalyze requirements and define evaluation strategies, golden datasets, regression benchmarks, baselines, thresholds, and blocking metrics with domain-expert review.\nDesign, build, and orchestrate AI agents, persona-based workflows, and reusable skills for the STLC using LangChain, LangGraph, or Semantic Kernel.\nBuild automated evaluations and benchmarks for LLMs, agents, RAG, retrieval, tool trajectories, and contracts using Ragas, DeepEval, and promptfoo.\nCalibrate LLM-as-judge against human grading, manage non-determinism through repeated runs, and detect regression and drift.\nWrite and maintain Python and pytest automation for AI, ML, API, data, and integration workflows.\nUse Cursor and agentic tools to analyze requirements, generate and review tests, and maintain evaluation suites.\nValidate deep-learning, reinforcement-learning, and predictive models using LangChain.\nBuild AI-assisted STLC workflows for requirements analysis, test design, script generation, execution, self-healing, failure analysis, and defect triage, plus intelligent test selection and defect prediction.\nIntegrate LLMs, APIs, embeddings, vector databases, retrieval pipelines, and enterprise data sources into QE frameworks using Python.\nTest REST and streaming APIs, agent tools, and RAG and MCP integrations, including authentication, errors, retries, and recovery.\nTest data and retrieval pipelines for schema integrity, lineage, idempotency, fallback, promotion gates, and rollback.\nIntegrate evaluations, benchmarks, and agent workflows into QE frameworks and CI/CD on GitHub Actions, review Allure results, and keep TestRail aligned with automated coverage.\nTest prompt injection, jailbreaks, unsafe tool use, data exfiltration, access control, PII controls, and guardrails.\nMeasure accuracy, relevance, groundedness, hallucination, retrieval quality, agent reliability, latency, throughput, token use, and cost, and diagnose distributed failures.\nImplement tracing, monitoring, and production evaluation using Langfuse, LangSmith, or a comparable AI observability platform.\nDeploy and operate QE agents on Amazon Bedrock AgentCore or a comparable agent platform, with governance and human oversight.\nDeploy and validate AI and ML solutions on AWS or Azure, including secrets, database services, and cloud-native test infrastructure.\nReport, track, and verify defects in Jira, and work with domain experts, product, and engineering on traceable coverage and untested scope.\nContribute to release readiness for the assigned product area without owning sign-off for the whole team.\nStrong Python proficiency with pytest for test and evaluation automation.\nHands-on experience building and orchestrating AI agents and agentic workflows with LangChain, LangGraph, Semantic Kernel, or comparable frameworks.\nHands-on experience building AI evaluations and benchmarks with Ragas, DeepEval, promptfoo, or equivalent tools.\nHands-on use of Cursor or agentic development tools for test generation and maintenance.\nExperience defining AI evaluation strategies, golden datasets, regression benchmarks, and LLM-as-judge calibration, including non-determinism and drift.\nExperience validating deep-learning, reinforcement-learning, and predictive models with LangChain.\nExperience building AI-assisted STLC workflows for requirements analysis, test design, script generation, execution, self-healing, failure analysis, defect triage, intelligent test selection, and defect prediction.\nExperience integrating LLMs, APIs, embeddings, vector databases, and retrieval pipelines with automation frameworks in Python.\nExperience testing REST and streaming APIs, agent tools, RAG workflows, and MCP integrations.\nExperience testing data and retrieval pipelines for correctness, lineage, and safe promotion.\nExperience integrating AI evaluation and agent workflows with automation frameworks and CI/CD using GitHub Actions, Allure, TestRail, or equivalent tools.\nExperience testing AI security, privacy, access control, and guardrails.\nExperience measuring accuracy, relevance, groundedness, hallucination, retrieval quality, agent reliability, latency, throughput, token use, and cost, and diagnosing distributed failures.\nExperience with AI tracing, monitoring, and production evaluation using Langfuse, LangSmith, or comparable tools.\nFamiliarity with AWS, Azure, or GCP cloud AI services and agent platforms for deploying and validating AI and ML solutions.\nExperience deploying and operating agents on Amazon Bedrock AgentCore or a comparable agent platform, with governance and human oversight.\nExperience managing defects through resolution in Jira or an equivalent tool.\nAbility to work with domain experts, product, and engineering on ground truth and coverage gaps.\nExperience contributing to release readiness for an assigned product area.\nPlaywright and TypeScript experience for UI and API automation.\nReal-device and cross-browser execution using LambdaTest, Sauce Labs, or an equivalent platform.\nKnowledge of AI-driven test optimization and coverage analytics using SeaLights or an equivalent platform.\nExperience contributing reusable components to shared automation frameworks and reviewing automation code through Git and GitHub workflows.\nExperience managing test data and troubleshooting test environments needed for reliable execution.\nExperience triaging security findings and validating operational readiness, monitoring, rollback, and recovery controls.\nExperience producing audit-ready evaluation reports and improving shared evaluation patterns, automation libraries, and datasets.","description_format":"text","description_chars":7290,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"phd","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Outsourcing & Dedicated Teams","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-10-29T12:30:09Z"}],"visa":[],"liveness":{"score":29,"band":"fade","label":"Fading","p_open":1,"p_active":0.525,"p_room":0.55,"age_days":32,"expected_fill_days":27,"reasons":["conf:2","stale_co","velocity","win:tail","comp:brand"],"computed_at":"2026-11-01T06:10:25Z"},"pay":null,"html_url":"https://alion.io/job/dataart-senior-ai-quality-engineer-hyderabad-based","json_url":"https://alion.io/job/dataart-senior-ai-quality-engineer-hyderabad-based.json","meta":{"generated_at":"2026-11-01T23:11:48Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":35318,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-02T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":46984},"rest":"https://alion.io/mcp/rest/get_company?id=46984"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fdataart-senior-ai-quality-engineer-hyderabad-based"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fdataart-senior-ai-quality-engineer-hyderabad-based"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fdataart-senior-ai-quality-engineer-hyderabad-based"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/dataart-senior-ai-quality-engineer-hyderabad-based\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fdataart-senior-ai-quality-engineer-hyderabad-based"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3453417}},{"tool":"search_jobs","arguments":{"company":"dataart.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3453417},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}