{"id":3856177,"url":"https://alion.io/job/thoth-pte-ltd-ai-evaluation-engineer-based-in-eu","title":"AI Evaluation Engineer based in EU","company":{"id":3775644,"name":"Thoth Pte. Ltd.","domain":"aithoth.com","url":"https://alion.io/company/aithoth","size_band":"501-1000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Moka","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"part_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Claude Code","optional":false},{"name":"Docker","optional":false},{"name":"OpenAI Codex","optional":false},{"name":"Python","optional":false},{"name":"Reinforcement Learning","optional":true}],"status":"live","first_seen_at":"2026-09-08T17:06:48Z","employer_posted_date":"2026-09-08","last_verified_at":"2026-11-20T16:22:58Z","board_verified":true,"closed_at":null,"days_open":73,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":73},"description":"About This Opportunity\nWe are currently building a talent pool of experienced AI Evaluation Engineers for an upcoming client project focused on AI model assessment, benchmarking, and quality assurance.\nThis is a freelance, project-based engagement, with project scope, duration, workload, and onboarding timelines currently being finalized. We welcome expressions of interest from qualified professionals with relevant technical and evaluation experience.\nAt this stage, we are proactively connecting with qualified professionals who may be interested in participating in the project. Candidates who are a potential fit will be considered and contacted as the project details are confirmed and the engagement moves forward.\nJob Summary\nWe are seeking an AI Evaluation Engineer to assess, validate, and improve AI task quality across the end-to-end evaluation pipeline. The ideal candidate possesses strong software engineering skills, hands-on experience with Python-based environments, and the ability to identify root causes of issues within AI evaluation, benchmarking, and rollout processes.\nKey Responsibilities\nEvaluate and analyze AI tasks, benchmarks, and execution traces to identify quality issues and performance gaps across the pipeline.\nInvestigate root causes of failures and determine whether issues originate from task design, evaluation metrics, reference solutions, execution environments, or rollout contamination.\nValidate evaluation methodologies, ensuring reproducibility, environment consistency, model discrimination, and proper visible/hidden evaluation isolation.\nExecute and troubleshoot tasks locally using Python, Shell, Docker, and AI coding tools such as Claude Code or Codex.\nReview evaluation results, artifacts, logs, and traces, providing evidence-based recommendations and quality assessments.\nProduce clear Pass, Conditional Pass, or Reject decisions with actionable improvement plans and comprehensive documentation.\nRequirements\nBachelor’s degree in a relevant technical field or equivalent practical experience.\nStrong hands-on programming experience in Python, Shell scripting, and Docker-based environments.\nExperience in AI model evaluation, benchmarking, testing, validation, quality assurance, or research operations.\nAbility to analyze execution traces, debugging logs, evaluation outputs, and identify root causes across complex workflows.\nHands-on experience using AI coding tools such as Claude Code, Codex, or similar tools to test, debug, and validate tasks locally.\nStrong analytical thinking, problem-solving, and technical reporting skills.\nPreferred Qualifications\nExperience in AI/ML research, model evaluation, reinforcement learning, or benchmark development.\nExperience designing evaluation methodologies and ensuring reproducible experiment results.\nKnowledge of model comparison, testing frameworks, and AI quality assessment practices.\nWhat Success Looks Like\nConsistently identify and resolve quality issues within AI evaluation pipelines.\nDeliver accurate, evidence-backed evaluation conclusions.\nImprove benchmark quality, reproducibility, and model assessment reliability.\nSupport the development of high-quality datasets and evaluation standards for AI systems.","description_format":"text","description_chars":3228,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":true},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-11-06T06:33:14Z"}],"visa":[],"liveness":{"score":11,"band":"cold","label":"Long shot","p_open":1,"p_active":0.398,"p_room":0.28,"age_days":72,"expected_fill_days":27,"reasons":["conf:2","win:tail","crowd:"],"computed_at":"2026-11-20T05:46:30Z"},"pay":null,"html_url":"https://alion.io/job/thoth-pte-ltd-ai-evaluation-engineer-based-in-eu","json_url":"https://alion.io/job/thoth-pte-ltd-ai-evaluation-engineer-based-in-eu.json","meta":{"generated_at":"2026-11-20T19:47:28Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":25368,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-21T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3775644},"rest":"https://alion.io/mcp/rest/get_company?id=3775644"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fthoth-pte-ltd-ai-evaluation-engineer-based-in-eu"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fthoth-pte-ltd-ai-evaluation-engineer-based-in-eu"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fthoth-pte-ltd-ai-evaluation-engineer-based-in-eu"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/thoth-pte-ltd-ai-evaluation-engineer-based-in-eu\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fthoth-pte-ltd-ai-evaluation-engineer-based-in-eu"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3856177}},{"tool":"search_jobs","arguments":{"company":"aithoth.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3856177},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}