{"id":4218465,"url":"https://alion.io/job/ey-agentic-ai-evaluation-engineer","title":"Agentic AI Evaluation Engineer","company":{"id":2859,"name":"EY","domain":"ey.com","url":"https://alion.io/company/ey-com","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"SuccessFactors","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"middle","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Noida, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"A/B Testing","optional":false},{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"Arize Phoenix","optional":false},{"name":"CI/CD","optional":false},{"name":"DeepEval","optional":false},{"name":"Docker","optional":false},{"name":"Embeddings","optional":false},{"name":"Function Calling","optional":false},{"name":"Git","optional":false},{"name":"Hallucination","optional":false},{"name":"Langfuse","optional":false},{"name":"LangSmith","optional":false},{"name":"Least Privilege","optional":false},{"name":"LLM","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"LLM Guardrails","optional":false},{"name":"Multi-Agent Systems","optional":false},{"name":"NLP","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"OWASP Top 10","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Red Teaming","optional":false},{"name":"Azure","optional":true},{"name":"EU AI Act","optional":true},{"name":"NIST AI RMF","optional":true},{"name":"OpenAI","optional":true}],"status":"live","first_seen_at":"2026-11-13T02:00:00Z","employer_posted_date":"2026-11-13","last_verified_at":"2026-11-13T03:37:59Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.\nEY- Assurance - Senior - Digital\nRole: GenAI / Agentic AI Evaluation Engineer (Quality, Safety & Reliability)\nPosition Details\nAs part of EY GDS Assurance Digital, you will help design, build, and scale a standardized evaluation capability, focused on evaluating GenAI, RAG-based, and Agentic AI solutions before deployment.\nThis role sits at the intersection of AI evaluation engineering, Responsible AI, and GenAI security/red teaming. The primary objective is to ensure GenAI/agentic systems are safe, reliable, robust, and fit-for-purpose, by designing evaluation strategies, building repeatable test harnesses, and generating auditable evidence that supports go/no-go decisions.\nYou will work with global stakeholders (product teams, solution architects, risk & compliance, and assurance leadership) to define evaluation requirements, request test datasets from product teams, execute rigorous evaluations (functional + non-functional), and recommend mitigations and controls to reduce risk.\nThis is a core full-time role that requires a hands-on AI Development mindset, strong evaluation mindset, and the ability to translate risk concerns into practical testing strategies and measurable acceptance criteria.\nResponsibilities\nDefine and operationalize evaluation strategies for GenAI systems across use cases like Q&A assistants, summarization, extraction, drafting, agentic systems, and multi-step workflows.\nTranslate business use-cases into a structured evaluation plan: scope, assumptions, success criteria, datasets, metrics, red-team scenarios, thresholds, and reporting requirements.\nDrive standardization: reusable evaluation templates, test case libraries, scoring rubrics, and reporting formats across product teams.\nDesign structured dataset requirements for product teams and ensure coverage across:Core user journeys and primary business intents\nEdge cases (rare prompts, ambiguous queries, incomplete context)\nAdversarial cases (malicious prompts, jailbreak attempts, prompt injections)\nBias & fairness cases (sensitive demographic proxies, protected attributes, stereotyping patterns)\n\nDefine guidance for dataset sufficiency and statistical coverage (e.g., minimum samples, distribution balance, scenario matrices, stratification by intent/risk).\nBuild reusable evaluation pipelines for:Answer quality (correctness, relevance, completeness, clarity)\nGrounding & faithfulness (RAG-specific: faithfulness, context precision/recall, hallucination rate, citation quality)\nAgentic behavior (tool-call accuracy, tool misuse, goal completion, step correctness, unnecessary actions, loop detection, safety of tool outputs)\nOperational quality (latency, cost/token budget, throughput, stability, retries, failure recovery)\n\nCombine LLM-as-judge and human evaluation in a calibrated way (rubric design, sampling plans, agreement checks).\nImplement automated evaluation harnesses in Python (preferred), enabling:batch runs on scenario suites\nconfigurable metric definitions\nreproducible runs with run IDs and artifacts\nstorage of traces and outputs for auditability\n\nExecute structured red teaming aligned to OWASP Top 10 for LLM Applications, covering (examples):Prompt injection (direct + indirect) and tool hijacking\nSensitive data disclosure / PII leakage\nInsecure output handling (downstream injection)\nTraining data leakage / memorization probes\nModel denial-of-service / denial-of-wallet patterns\n\nIntegrate evals into development lifecycle: pre-release regression gates, CI checks, benchmark comparisons across model versions/prompts/tools/retrievers.\nPerform adversarial testing for agentic workflows:tool misuse / over-permissioned tool access\nunauthorized action execution\nexfiltration via tools/connectors\nprompt injection via retrieved documents (RAG poisoning)\n\nRecommend mitigations: input validation, retrieval filtering, tool sandboxing, least-privilege permissions, guardrails, policy prompting, refusal logic, output encoding, monitoring alerts.\nProduce high-quality evaluation reports that are auditable and decision-ready, including:methodology, datasets, metrics, thresholds\nquantitative results\nqualitative results\nrisk assessment summary and recommended control actions\n\nPresent findings to stakeholders in a crisp, risk-informed manner; clearly explain residual risk, limitations, and rationale for go/no-go.\nKey Requirements/Skills & Qualification:\nExcellent academic background, including at a minimum a bachelor’s or a master’s degree in data science, Statistics, Engineering, Operational Research, or other related field with strong focus on modern data architectures, processes, and environments.\n4-7+ years of relevant experience in one or more areas:ML/AI/GenAI/Agentic engineering (NLP/LLMs), evaluation engineering, applied research\nsecurity testing / red teaming\nbuilding and designing evaluation harness that ensures safety, reliability and robustness.\n\nStrong hands-on Python for building evaluation harnesses (data processing, metric computation, orchestration, reporting pipelines).\nPractical understanding of GenAI system architectures: RAG, embeddings/vector search, prompt orchestration, tool calling, multi-agent systems, memory, routing.\nExperience designing metrics and evaluation methods (rubrics, automated scoring, sampling strategy, regression design).\nFamiliarity with LLM risks and mitigations, especially for enterprise contexts (data leakage, hallucinations, prompt injection, unsafe content, bias).\nSecurity / Red Teaming Skills (Strong Preference)\nUnderstanding of OWASP Top 10 for LLM Applications and how to translate it into test cases and controls.\nExperience with adversarial testing approaches: jailbreak prompts, injection patterns, tool misuse scenarios, retrieval poisoning patterns.\nFamiliarity with secure-by-design practices for LLM apps: least privilege, safe tool invocation, output encoding/validation, monitoring.\nEvaluation frameworks and tooling: RAGAS, DeepEval, LangSmith, Phoenix/Arize, custom eval harnesses.\nExperimentation practices: A/B testing mindset, baseline comparisons, statistical rigor for sample sizes.\nObservability/tracing: structured logging, OpenTelemetry, Langfuse-style traces, dashboards.\nBasic DevOps practices: Git, CI/CD, containerization (Docker), reproducible environments.\nStrong written communication to produce clear evaluation plans and reports for technical + non-technical stakeholders.\nAbility to challenge assumptions constructively (“effective challenge”) and influence engineering teams toward remediation.\nComfort operating in ambiguity with fast-evolving GenAI tooling and risk landscape.\nPreferred / Nice-to-Have\nExperience in Assurance/Finance/Regulatory environments (model validation, risk acceptance workflows, audit evidence mindset).\nFamiliarity with responsible AI frameworks (NIST AI RMF, ISO/IEC 42001, EU AI Act concepts).\nExperience evaluating multilingual systems or domain-heavy enterprise assistants.\nHands-on with Azure ecosystem (Azure OpenAI, AI Search, Function Apps, App Insights, Key Vault).\nAdditional skills requirements:\nExcellent written, oral, presentation and facilitation skills\nAbility to coordinate multiple projects and initiatives simultaneously through effective prioritization, organization, flexibility, and self-discipline.\nMust have demonstrated project management experience.\nKnowledge of firm’s reporting tools and processes.\nProactive, organized, and self-sufficient with ability to priorities and multitask.\nAnalyses complex or unusual problems and can deliver insightful and pragmatic solutions.\nAbility to quickly and easily create/ gather/ analyze data from a variety of sources.\nA robust and resilient disposition able to encourage discipline in team behaviors\nWhat we look for\nA Team of people with commercial acumen, technical experience, and enthusiasm to learn new things in this fast-moving environment\nAn opportunity to be a part of market-leading, multi-disciplinary team of 7200 + professionals, in the only integrated global assurance business worldwide.\nOpportunities to work with EY GDS Assurance practices globally with leading businesses across a range of industries\nWhat working at EY offers\nAt EY, we’re dedicated to helping our clients, from startups to Fortune 500 companies - and the work we do with them is as varied as they are.\nYou get to work with inspiring and meaningful projects. Our focus is education and coaching alongside practical experience to ensure your personal development. We value our employees, and you will be able to control your own development with an individual progression plan. You will quickly grow into a responsible role with challenging and stimulating assignments. Moreover, you will be part of an interdisciplinary environment that emphasizes high quality and knowledge exchange. Plus, we offer:\nSupport, coaching and feedback from some of the most engaging colleagues around\nOpportunities to develop new skills and progress your career\nThe freedom and flexibility to handle your role in a way that’s right for you\nEY | Building a better working world\nEY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.\nEnabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.\nWorking across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.","description_format":"text","description_chars":9898,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Audit & Assurance Firms","Financial Advisory, Valuation & Restructuring","Management Consulting","Tax Advisory & Preparation"],"lifecycle":[{"event":"open","at":"2026-11-13T03:37:59Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":25,"reasons":["conf:0","win:early","comp:brand"],"computed_at":"2026-11-13T04:30:08Z"},"pay":null,"html_url":"https://alion.io/job/ey-agentic-ai-evaluation-engineer","json_url":"https://alion.io/job/ey-agentic-ai-evaluation-engineer.json","meta":{"generated_at":"2026-11-13T04:30:08Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":9530,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-14T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2859},"rest":"https://alion.io/mcp/rest/get_company?id=2859"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fey-agentic-ai-evaluation-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fey-agentic-ai-evaluation-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fey-agentic-ai-evaluation-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/ey-agentic-ai-evaluation-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fey-agentic-ai-evaluation-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4218465}},{"tool":"search_jobs","arguments":{"company":"ey.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4218465},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}