{"id":2918506,"url":"https://alion.io/job/resaror-senior-ai-engineer-agentic-ai-evaluation","title":"Senior AI Engineer - Agentic AI Evaluation","company":{"id":1928287,"name":"Resaro®","domain":"resaro.ai","url":"https://alion.io/company/resaro","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Singapore"],"countries":["SG"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":11000,"max":16000,"currency":"SGD","period":"month","gross":true,"usd_annual":150012},"salary_estimate":null,"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"A2A","optional":false},{"name":"AI Agents","optional":false},{"name":"Computer Vision","optional":false},{"name":"EU AI Act","optional":false},{"name":"Function Calling","optional":false},{"name":"ISO 42001","optional":false},{"name":"LLM","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"NIST AI RMF","optional":false},{"name":"Python","optional":false},{"name":"CUDA","optional":true},{"name":"DuckDB","optional":true},{"name":"Helm","optional":true},{"name":"Kubernetes","optional":true},{"name":"RAG","optional":true},{"name":"Vision-Language-Action","optional":true}],"status":"live","first_seen_at":"2026-10-20T00:00:00Z","employer_posted_date":null,"last_verified_at":"2026-10-20T00:00:00Z","board_verified":false,"closed_at":null,"days_open":1,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":1},"description":"About Us\nResaro is an independent, third-party AI assurance company. We build the software and run the evaluations that let enterprises and public-sector bodies deploy AI they can actually trust - across computer vision, LLMs, vision-language models, and, increasingly, agentic and autonomous systems, for clients in government, defence, and regulated commercial sectors.\nOur product, the Approved Intelligence Platform (AIP), is where this becomes real software: customers register the AI systems they want tested, run rigorous evaluations, and produce defensible, auditable evidence and reports. Agentic AI is moving from pilots into operational, high-stakes deployments faster than anyone can assure it - the overwhelming majority of agent pilots still fail before production, on governance, traceability, and reliability gaps rather than raw capability. Closing that gap is the product. That's what you'll build.\nThe Role\nWe're hiring a senior engineer to be our go-to person for agentic AI evaluation - the technical owner of how AIP measures whether tool-using, multi-step agents are fit to deploy. You'll build the evaluation engine behind our agentic MVP and the roadmap beyond it, and you'll be the person the team turns to when an agentic question is hard. As autonomous and embodied systems mature, you'll help extend the same rigour into those emerging areas.\nThis is a hands-on individual-contributor role with domain ownership. You report to the Engineering Manager for the evaluation engine and partner closely with Product, who owns the roadmap. You set the technical bar for this vertical; you don't line-manage.\nWhat You'll Do\nOwn the agentic-evaluation engine. Build the capability that scores agents on the metrics that matter - intent resolution, tool-selection and parameterisation accuracy, task adherence, planning quality, tool-call ordering, response completeness against a reference, and when an agent should defer to a human.\n\nEvaluate stateful, multi-turn behaviour - not single prompts. Agents carry memory and act over long horizons; your harnesses must expose how behaviour drifts, degrades, or becomes unsafe across a session.\n\nMake evaluation trustworthy. Design for evaluator integrity: isolate the runner from the agent, assert on semantics not just structure, count crashed/incomplete tasks as failures, audit ground truth, and build tests that probe the test setup itself. A benchmark you can't trust is worse than none.\n\nBuild the agentic-security surface. Partner on red-teaming and adversarial evaluation - tool-chain exploits, RBAC and permission-inheritance failures, multi-turn safety erosion - so we catch what a motivated adversary would.\n\nClose the lab-to-production gap. Design evaluations that surface where agents that look good in a demo fail in a real, customer-specific deployment - the gap that makes independent assurance necessary.\n\nTranslate governance into engineering. Work with our governance analysts to turn EU AI Act, NIST AI RMF, and ISO 42001 requirements into concrete, testable evaluation logic.\n\nSet the technical bar for the vertical - through spikes, ADRs, and code review - and keep your reasoning legible so the team can build on it.\n\nExtend into emerging autonomy. As embodied and other autonomous systems become real evaluation targets, help us adapt the engine to them.\n\nWhat We're Looking For\n5+ years of professional software engineering, shipping and operating production systems (not prototypes) in Python.\n\nDemonstrated depth in agentic AI - you've built, orchestrated, or rigorously evaluated LLM agents, and understand tool-calling, memory, multi-step control flow, and modern interoperability (MCP for tools, A2A for agent-to-agent) at more than a tutorial level.\n\nA real grasp of how agents are measured - and how measurement fails. You know the standard execution-based benchmarks (GAIA, τ²-bench, OSWorld, SWE-bench Verified, WebArena) and why a high benchmark score routinely overstates production reliability.\n\nStrong engineering judgement - you can take an ambiguous, research-adjacent problem, scope it, decompose it, and drive it to a shipped, tested, observable outcome without close supervision.\n\nA genuine bias to go deep and stay hands-on, paired with the communication to be the person others come to for this domain - you document decisions and leave work others can pick up.\n\nClear written and verbal communication, and comfort being measured against concrete quarterly outcomes, including across a distributed Singapore-Europe team.\n\nNice To Have\nAgentic security / red-teaming experience - adversarial testing, jailbreak/tool-chain exploits, or agent identity and on-behalf-of authorisation.\n\nEvaluation, testing, or assurance tooling for any AI domain — LLM/RAG, agentic, or computer vision - including LLM-as-judge pipelines and their failure modes.\n\nFamiliarity with AI governance frameworks (EU AI Act, NIST AI RMF, ISO 42001) and translating principles into engineering requirements.\n\nEmerging embodied / autonomous AI exposure - vision-language-action models, simulators, or world-model approaches (a plus, not a requirement).\n\nData-intensive pipelines with columnar/lakehouse formats (Parquet/Iceberg) and DuckDB or similar.\n\nContainer-based or serverless execution frameworks (Nuclio or comparable) for running evaluations at scale.\n\nKubernetes/Helm and GPU/CUDA infrastructure for model inference.\n\nExperience where evidence, auditability, and reproducibility were first-class requirements (regulated, safety-critical, or defence contexts).\n\nOur Hiring Blueprint\nWe hire to a high, transparent bar. We look for production-grade code (correct, tested, observable, maintainable by others); communication and handover (you write clearly and document decisions); independent operation (you scope and drive ambiguous problems to shipped outcomes); and a T-shaped profile - deep in agentic AI evaluation, broad enough to operate across the stack and the ML-evaluation domain.\nResaro is an Equal Opportunity Employer. We respect each individual and support the diverse cultures, perspectives, skills and experiences within our teams.\nSkills\nQuality Control, Agents, Scoped, Build Prototypes, Attack Surface Management, Distributed Team Management, Python, Evaluation, Managing Ambiguity, Benchmarking, Low Level Programming, behaviour changes, Agentic AI, Shipping System, Testing, Bias Identification and Mitigation","description_format":"text","description_chars":6418,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","AI Safety & Alignment","AI Evaluation & Observability"],"lifecycle":[{"event":"open","at":"2026-10-20T10:28:15Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":1,"expected_fill_days":27,"reasons":["seen:1","win:early"],"computed_at":"2026-10-21T05:46:15Z"},"pay":{"stated_usd_annual":150012,"is_top_pay":false},"html_url":"https://alion.io/job/resaror-senior-ai-engineer-agentic-ai-evaluation","json_url":"https://alion.io/job/resaror-senior-ai-engineer-agentic-ai-evaluation.json","meta":{"generated_at":"2026-10-21T23:26:22Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":33371,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-22T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1928287},"rest":"https://alion.io/mcp/rest/get_company?id=1928287"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fresaror-senior-ai-engineer-agentic-ai-evaluation"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fresaror-senior-ai-engineer-agentic-ai-evaluation"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fresaror-senior-ai-engineer-agentic-ai-evaluation"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/resaror-senior-ai-engineer-agentic-ai-evaluation\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fresaror-senior-ai-engineer-agentic-ai-evaluation"}]}