{"id":4249904,"url":"https://alion.io/job/weekday-subject-matter-experts-ml-model-evaluation-phd","title":"Subject Matter Experts: ML Model Evaluation (PhD)","company":{"id":7097,"name":"Weekday","domain":"weekday.works","url":"https://alion.io/company/weekday","size_band":"51-200","is_staffing_agency":true,"employer_type":"agency","is_intermediary":false,"listed_via":null,"ats_vendor":"Workable","truth_index":{"grade":"B","score":80,"open_postings":117,"ghost_share":0,"stale_share":0.983,"repost_share":0,"time_to_fill_p50_days":5,"computed_at":"2026-11-12T10:38:37Z"}},"role":"Early Careers","role_family":"Early Careers","seniority":null,"employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"inferred","locations":[],"countries":[],"hiring_countries":["US","GB","CA","NO","CH","IS"],"hiring_countries_total":6,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Machine Learning","optional":false}],"status":"live","first_seen_at":"2026-11-13T13:53:58Z","employer_posted_date":"2026-11-13","last_verified_at":"2026-11-13T13:53:58Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"This role is for one of Weekday’s clients\nEngagement Type: Independent Contractor\nWork Arrangement: Fully Remote\nEstimated Commitment: Approximately 5 Hours\nSchedule: Flexible, Asynchronous\nPayment Frequency: Weekly, based on services rendered\nEligibility:\nApplicants must be from any of these geographies : US, UK, Canada, the EU, Norway, Switzerland and Iceland.\nShould have a finished / running paper in any of these topics : NeurIPS, ICML, ICLR, COLM or RLC\nWe are unable to support H1-B or STEM OPT candidates at this time.\nNote:\nPay : $150 / hr\nThis requirement is super-urgent & would be awesome if you can start on the project within Friday EOD. \nRole Overview\nWe are seeking highly qualified Subject Matter Experts with PhD-level academic or professional expertise across specialised research and professional domains to participate in a paid Machine Learning (ML) Model Evaluation project.\nThe project focuses on assessing the quality, accuracy, and reliability of AI-generated responses to complex research prompts and domain-specific questions. Your evaluations will help identify where current machine learning models perform effectively and where their outputs require further refinement.\nThis opportunity is suited to experienced researchers, academics, data analysts, technical writers, and domain specialists who can critically assess complex information, identify factual inaccuracies, and provide structured, evidence-based feedback.\nRequirements\nKey Responsibilities\nReview machine learning model outputs generated in response to a range of research-focused and domain-specific prompts.\nEvaluate AI-generated responses for factual accuracy, relevance, completeness, and overall quality.\nAssess the technical validity and reliability of information presented in model-generated responses.\nUse a structured rating system to evaluate and score responses consistently.\nProvide concise, clear, and well-reasoned written justifications supporting evaluation scores.\nIdentify factual errors, gaps in reasoning, irrelevant information, and areas requiring improvement.\nComplete approximately five hours of independent evaluation work asynchronously.\nMaintain consistency and attention to detail throughout the evaluation process.\nRequired Qualifications and Skills\nPhD or equivalent advanced subject matter expertise in a relevant academic, technical, or professional discipline.\nStrong academic or professional background in a specialised domain, with the ability to assess complex information critically.\nExcellent analytical, research, and critical-thinking skills.\nAbility to evaluate technical content, research findings, and domain-specific information for factual correctness and relevance.\nStrong written communication skills, particularly in providing clear and structured evaluation justifications.\nHigh attention to detail and the ability to identify subtle inaccuracies, inconsistencies, and gaps in AI-generated responses.\nAbility to work independently and complete approximately five hours of asynchronous evaluation tasks.\nSuitable Candidate Profiles\nProfessionals from the following backgrounds are encouraged to apply:\nAcademic researchers and university faculty\nPhD scholars and doctoral-level subject matter experts\nData analysts and research analysts\nTechnical writers and research professionals\nDomain specialists with experience evaluating technical or research-focused content\nEngagement and Payment Terms\nContract Type: Independent contractor engagement.\nWork Arrangement: Fully remote, with flexible working hours.\nProject Duration: Approximately five hours of evaluation work.\nPayment: Payments are processed weekly based on services rendered.\nProject Continuity: The engagement may be extended, shortened, or concluded early depending on project requirements and individual performance.\nConfidentiality: Participation will not require access to confidential or proprietary information belonging to any current or former employer, client, or academic institution.\n Work Authorisation: H-1B and STEM OPT candidates cannot be supported for this opportunity.\nInterview and Evaluation Expectations\nDuring the interview process, candidates may be assessed on their review practices, critical thinking, and ability to improve the quality of their evaluations. Responses should address the following three habits:\nExchanging unfinished review notes with another reviewer.\nMaintaining a separate log of opinions or assessments that have changed during the review process.\nUsing review notes and feedback to update personal reading lists.\nIdeal Candidate Profile\nThe ideal candidate combines deep subject matter expertise with strong research judgement and analytical precision. They should be comfortable reviewing complex AI-generated content, distinguishing accurate information from flawed reasoning, and communicating their assessments through clear, structured feedback.","description_format":"text","description_chars":4905,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Flexible schedule"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"},{"name":"United Kingdom","iso":"GB","kind":"country"},{"name":"Canada","iso":"CA","kind":"country"},{"name":"Norway","iso":"NO","kind":"country"},{"name":"Switzerland","iso":"CH","kind":"country"},{"name":"Iceland","iso":"IS","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-11-13T13:53:58Z"}],"visa":[],"liveness":{"score":52,"band":"ok","label":"Likely open","p_open":1,"p_active":0.516,"p_room":1,"age_days":0,"expected_fill_days":5,"reasons":["conf:0","agency","win:early","comp:brand"],"computed_at":"2026-11-13T14:31:15Z"},"pay":null,"html_url":"https://alion.io/job/weekday-subject-matter-experts-ml-model-evaluation-phd","json_url":"https://alion.io/job/weekday-subject-matter-experts-ml-model-evaluation-phd.json","meta":{"generated_at":"2026-11-13T14:31:15Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":29108,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-14T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":7097},"rest":"https://alion.io/mcp/rest/get_company?id=7097"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fweekday-subject-matter-experts-ml-model-evaluation-phd"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fweekday-subject-matter-experts-ml-model-evaluation-phd"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fweekday-subject-matter-experts-ml-model-evaluation-phd"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/weekday-subject-matter-experts-ml-model-evaluation-phd\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fweekday-subject-matter-experts-ml-model-evaluation-phd"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4249904}},{"tool":"search_jobs","arguments":{"company":"weekday.works","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4249904},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}