{"id":603225,"url":"https://alion.io/job/anyone-ai-research-scientist-remoteuslatam","title":"Research Scientist (Remote/US/LATAM)","company":{"id":9137,"name":"Anyone AI","domain":"anyoneai.com","url":"https://alion.io/company/anyone-ai","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"A","score":95,"open_postings":176,"ghost_share":0,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":63,"computed_at":"2026-11-06T05:58:05Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"contractor","work_mode":"remote","remote_scope":"stated_regions","remote_scope_basis":"posting_text","remote_working_hours":null,"hiring_geo_confidence":"explicit","locations":[],"countries":[],"hiring_countries":["US","AR","BR","CO","MX","AI","AG","AW","BS","BB","BZ","BM","BO","VG","BQ","KY","CL","CR","CW","DM","DO","EC","SV","FK","GF","GD","GP","GT","GY","HT","HN","JM","MQ","MS","NI","PA","PY","PE","PR","KN"],"hiring_countries_total":50,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Function Calling","optional":false},{"name":"LLM","optional":false},{"name":"Multimodal AI","optional":false}],"status":"live","first_seen_at":"2026-07-06T14:23:42Z","employer_posted_date":"2026-07-06","last_verified_at":"2026-11-06T10:49:54Z","board_verified":true,"closed_at":null,"days_open":122,"trust":{"level":"ok","repost_count":0,"flags":["company_stale"],"days_open":122},"description":"Research Scientist, LLM Evaluations & Benchmarking\nAnyone AI Labs\nReports to: CEO · Remote / LatAm / US\nThe role Evaluation is one of the hardest open problems in AI: we still don't have reliable ways to measure what frontier models can and can't do, and the field mostly runs on benchmarks that are saturated, contaminated, or measuring the wrong thing. You'll own that problem at Anyone AI, measuring frontier model capability.\nThis is a research role at heart: you decide what a good evaluation is , design the benchmarks that prove it, and defend the methodology under lab scrutiny. You'll build frontier-grade evaluation packages across reasoning, coding, agents, tool use, and multi-modal - grounded in expert-verified truth, validated against multiple models, and QC'd to survive buyer-side review.\nResponsibilities\n\nEvaluation research. Turn eval targets into original benchmark designs. Own the hard measurement questions: construct validity, item discrimination, headroom, reliability, contamination, and capability elicitation. Push toward evals that stay informative as models improve.\n\nBenchmark development. Build evaluation packages with subject-matter experts, each with expert-verified ground truth, multi-model headroom results, and rigorous QC (calibration layers, severity-weighted rubrics, deterministic verifiers).\n\nExperts. Recruit, calibrate, and review a pool across coding, agentic/tool-use, and STEM/reasoning. Be the final arbiter of correctness and frontier difficulty.\n\nLab relationships. Be a technical point of contact for labs, with CEO support. Understand what they're trying to measure and translate it into an evaluation design.\n\nDelivery & dissemination. Turn lab requests into winning sample packages and own pilots end to end. Where the work generalizes, help turn it into public benchmarks and papers: we support publishing at venues like NeurIPS Datasets & Benchmarks, ICLR, and ACL.\nWhat we're looking for\nResearch background in ML evaluation or benchmarking (a track record of published or open benchmarks, eval/measurement research, or equivalent hands-on work that labs have relied on).\nDeep LLM/frontier-model benchmarking expertise, with real strength in code-model and agentic evaluation.\nFluency with the measurement problem itself: construct validity, psychometrics, rubrics, pass rates, headroom, contamination, and what makes a task genuinely discriminate a model.\nInterest in the safety side of evaluation, capability elicitation, robustness, and measuring the things that are hardest to measure honestly.\nProven ability to hold a team or expert pool to a rigorous standard.\nComfort with the full research loop: framing the question, running the study, and writing it up.\nFluent English; Spanish a plus.","description_format":"text","description_chars":2756,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[{"language":"English","level":"Advanced (C1)","optional":false}]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"},{"name":"Argentina","iso":"AR","kind":"region"},{"name":"Brazil","iso":"BR","kind":"region"},{"name":"Colombia","iso":"CO","kind":"region"},{"name":"Mexico","iso":"MX","kind":"region"},{"name":"Anguilla","iso":"AI","kind":"region"},{"name":"Antigua and Barbuda","iso":"AG","kind":"region"},{"name":"Aruba","iso":"AW","kind":"region"},{"name":"Bahamas","iso":"BS","kind":"region"},{"name":"Barbados","iso":"BB","kind":"region"},{"name":"Belize","iso":"BZ","kind":"region"},{"name":"Bermuda","iso":"BM","kind":"region"},{"name":"Bolivia","iso":"BO","kind":"region"},{"name":"British Virgin Islands","iso":"VG","kind":"region"},{"name":"Bonaire, Sint Eustatius and Saba","iso":"BQ","kind":"region"},{"name":"Cayman Islands","iso":"KY","kind":"region"},{"name":"Chile","iso":"CL","kind":"region"},{"name":"Costa Rica","iso":"CR","kind":"region"},{"name":"Curaçao","iso":"CW","kind":"region"},{"name":"Dominica","iso":"DM","kind":"region"},{"name":"Dominican Republic","iso":"DO","kind":"region"},{"name":"Ecuador","iso":"EC","kind":"region"},{"name":"El Salvador","iso":"SV","kind":"region"},{"name":"Falkland Islands","iso":"FK","kind":"region"},{"name":"French Guiana","iso":"GF","kind":"region"},{"name":"Grenada","iso":"GD","kind":"region"},{"name":"Guadeloupe","iso":"GP","kind":"region"},{"name":"Guatemala","iso":"GT","kind":"region"},{"name":"Guyana","iso":"GY","kind":"region"},{"name":"Haiti","iso":"HT","kind":"region"},{"name":"Honduras","iso":"HN","kind":"region"},{"name":"Jamaica","iso":"JM","kind":"region"},{"name":"Martinique","iso":"MQ","kind":"region"},{"name":"Montserrat","iso":"MS","kind":"region"},{"name":"Nicaragua","iso":"NI","kind":"region"},{"name":"Panama","iso":"PA","kind":"region"},{"name":"Paraguay","iso":"PY","kind":"region"},{"name":"Peru","iso":"PE","kind":"region"},{"name":"Puerto Rico","iso":"PR","kind":"region"},{"name":"Saint Kitts and Nevis","iso":"KN","kind":"region"},{"name":"Saint Lucia","iso":"LC","kind":"region"},{"name":"Saint Vincent and the Grenadines","iso":"VC","kind":"region"},{"name":"Saint-Pierre and Miquelon","iso":"PM","kind":"region"},{"name":"Sint Maarten","iso":"SX","kind":"region"},{"name":"Suriname","iso":"SR","kind":"region"},{"name":"Trinidad and Tobago","iso":"TT","kind":"region"},{"name":"Turks and Caicos Islands","iso":"TC","kind":"region"},{"name":"Uruguay","iso":"UY","kind":"region"},{"name":"United States Virgin Islands","iso":"VI","kind":"region"},{"name":"Venezuela","iso":"VE","kind":"region"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Bootcamps","Recruiting & Staffing","AI Training Data & Annotation"],"lifecycle":[{"event":"open","at":"2026-09-10T17:51:36Z"}],"visa":[],"liveness":{"score":20,"band":"cold","label":"Long shot","p_open":1,"p_active":0.554,"p_room":0.36,"age_days":122,"expected_fill_days":63,"reasons":["conf:3","velocity","win:tail","crowd:brand"],"computed_at":"2026-11-06T05:58:05Z"},"pay":null,"html_url":"https://alion.io/job/anyone-ai-research-scientist-remoteuslatam","json_url":"https://alion.io/job/anyone-ai-research-scientist-remoteuslatam.json","meta":{"generated_at":"2026-11-06T11:39:56Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":21226,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-07T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":9137},"rest":"https://alion.io/mcp/rest/get_company?id=9137"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fanyone-ai-research-scientist-remoteuslatam"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fanyone-ai-research-scientist-remoteuslatam"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fanyone-ai-research-scientist-remoteuslatam"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/anyone-ai-research-scientist-remoteuslatam\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fanyone-ai-research-scientist-remoteuslatam"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":603225}},{"tool":"search_jobs","arguments":{"company":"anyoneai.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":603225},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}