{"id":936289,"url":"https://alion.io/job/anyone-ai-senior-software-engineer-open-source-swe-bench-evaluation","title":"Machine Learning Engineer – ML Evaluation & Experiment Design","company":{"id":9137,"name":"Anyone AI","domain":"anyoneai.com","url":"https://alion.io/company/anyone-ai","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"A","score":95,"open_postings":176,"ghost_share":0,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":63,"computed_at":"2026-11-05T05:54:30Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"middle","employment_type":"contractor","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"posting_text","remote_working_hours":null,"hiring_geo_confidence":"inferred","locations":[],"countries":[],"hiring_countries":["AR","BR","CO","MX","ES","CL","EC","PT","UY"],"hiring_countries_total":9,"salary":{"min":65,"max":65,"currency":"USD","period":"hour","gross":null,"usd_annual":130000},"salary_estimate":null,"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Machine Learning","optional":false},{"name":"RLHF","optional":true},{"name":"Synthetic Data","optional":true}],"status":"live","first_seen_at":"2026-09-15T14:12:41Z","employer_posted_date":"2026-09-15","last_verified_at":"2026-11-05T05:57:27Z","board_verified":true,"closed_at":null,"days_open":50,"trust":{"level":"ok","repost_count":0,"flags":["company_stale"],"days_open":50},"description":"Anyone AI is recruiting experienced Machine Learning Engineers for a specialized project focused on reviewing and evaluating machine learning challenges used in AI model training and evaluation.\nThe work involves analyzing ML experiments, datasets, metrics, and pipelines to determine whether challenges are technically sound, reproducible, appropriately difficult, and genuinely require strong machine learning reasoning.\nWhat You’ll Work On\nYou’ll review ML challenges involving:\nExperiment design and model selection\n\nSmall and synthetic datasets\n\nData quality and preprocessing\n\nDistribution shift and data contamination\n\nLabel noise and feature leakage\n\nModel evaluation and metric selection\n\nHyperparameter tuning\n\nTrain / validation / test methodology\n\nReproducibility and deterministic pipelines\n\nStatistical significance of model improvements\n\nA key part of the role is determining whether a challenge actually rewards good ML reasoning, rather than simply being solvable through brute-force model selection or large hyperparameter searches.\nWhat We’re Looking For\n3+ years of hands-on applied machine learning experience\n\nStrong experience with:\nML experiment design\n\nModel selection\n\nHyperparameter tuning\n\nModel evaluation\n\nData preprocessing and validation\n\nStrong understanding of train, validation, and test splits\n\nAbility to identify:\nData leakage\n\nLabel noise\n\nDistribution shift\n\nSpurious correlations\n\nFeature leakage\n\nData contamination\n\nExperience evaluating whether performance improvements are statistically meaningful rather than random fluctuations\n\nStrong understanding of ML evaluation metrics and when different metrics are appropriate\n\nExperience debugging ML workloads across CPU and GPU environments\n\nAbility to analyze technical problems and provide clear written feedback\n\nNice to Have\nExperience creating or participating in Kaggle, DrivenData, or similar ML competitions\n\nExperience designing benchmark datasets or ML challenges\n\nBackground in data-centric AI or dataset quality\n\nExperience with synthetic data generation and validation\n\nFamiliarity with statistical testing, confidence intervals, and effect sizes\n\nExperience with ML evaluation pipelines, RLHF, or AI model evaluation\n\nExperience developing ML curricula or technical assessments\n\nUnderstanding of common ML failure modes such as:\nShortcut learning\n\nSpurious correlations\n\nGoodhart’s Law\n\nSimpson’s paradox\n\nMetric gaming\n\nWhat You’ll Be Responsible For\nReviewing ML challenges and determining whether they are well designed and technically solvable\n\nEvaluating whether datasets contain meaningful and learnable signals\n\nIdentifying unintended shortcuts or artifacts in synthetic datasets\n\nDetermining whether tasks require genuine diagnosis of the underlying ML problem\n\nReviewing evaluation metrics and improvement thresholds\n\nDetecting metric gaming, data leakage, and evaluation flaws\n\nVerifying reproducibility across the complete data → model → evaluation pipeline\n\nAssessing whether challenge difficulty is appropriately calibrated\n\nProviding clear recommendations for improving, recalibrating, or excluding problematic tasks\n\nEngagement\nWork Type: Remote\nEngagement: Part-time, project-based consulting\nFocus: Applied machine learning, experiment design, data quality, and model evaluation\nThis role is a strong fit for ML engineers who enjoy debugging experiments, understanding why models succeed or fail, identifying problems in datasets and evaluation pipelines, and designing rigorous machine learning experiments.","description_format":"text","description_chars":3530,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"Argentina","iso":"AR","kind":"country"},{"name":"Brazil","iso":"BR","kind":"country"},{"name":"Colombia","iso":"CO","kind":"country"},{"name":"Mexico","iso":"MX","kind":"country"},{"name":"Spain","iso":"ES","kind":"country"},{"name":"Chile","iso":"CL","kind":"country"},{"name":"Ecuador","iso":"EC","kind":"country"},{"name":"Portugal","iso":"PT","kind":"country"},{"name":"Uruguay","iso":"UY","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Bootcamps","Recruiting & Staffing","AI Training Data & Annotation"],"lifecycle":[{"event":"open","at":"2026-09-15T14:47:54Z"}],"visa":[],"liveness":{"score":20,"band":"cold","label":"Long shot","p_open":0.4,"p_active":0.846,"p_room":0.6,"age_days":50,"expected_fill_days":63,"reasons":["seen:50","velocity","win:late","crowd:brand"],"computed_at":"2026-11-05T05:54:30Z"},"pay":{"stated_usd_annual":130000,"is_top_pay":false},"html_url":"https://alion.io/job/anyone-ai-senior-software-engineer-open-source-swe-bench-evaluation","json_url":"https://alion.io/job/anyone-ai-senior-software-engineer-open-source-swe-bench-evaluation.json","meta":{"generated_at":"2026-11-05T09:16:38Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":11756,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-06T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":9137},"rest":"https://alion.io/mcp/rest/get_company?id=9137"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fanyone-ai-senior-software-engineer-open-source-swe-bench-evaluation"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fanyone-ai-senior-software-engineer-open-source-swe-bench-evaluation"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fanyone-ai-senior-software-engineer-open-source-swe-bench-evaluation"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/anyone-ai-senior-software-engineer-open-source-swe-bench-evaluation\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fanyone-ai-senior-software-engineer-open-source-swe-bench-evaluation"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":936289}},{"tool":"search_jobs","arguments":{"company":"anyoneai.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":936289},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}