{"id":96289,"url":"https://alion.io/job/contra-research-scientist-human-data-evaluation-2","title":"Research Scientist, Human Data & Evaluation","company":{"id":36707,"name":"Contra","domain":"contra.com","url":"https://alion.io/company/contra","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":true,"listed_via":null,"ats_vendor":"Ashby","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"middle","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"explicit","locations":["New York, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":150000,"max":200000,"currency":"USD","period":"year","gross":null,"usd_annual":200000},"salary_estimate":null,"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"Multimodal AI","optional":false},{"name":"Post-training","optional":false},{"name":"Python","optional":false},{"name":"RLHF","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-08-17T15:48:54Z","employer_posted_date":"2026-08-17","last_verified_at":"2026-10-17T20:49:50Z","board_verified":true,"closed_at":null,"days_open":61,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":61},"description":"Who we are and what we do\nContra Labs is a human-centered AI lab focused on creative and multimodal outputs, where human taste defines the next generation of AI capabilities.\nWe build industry-leading creative preference datasets that power benchmarking, evaluation, and post-training for the world's leading AI models and applications. Our work helps define what \"good\" looks like across design, video, imagery, and beyond.\nBuilt on top of Contra, the professional network for independent creatives, Contra Labs connects frontier AI labs with a global network of top creative talent. Together, we turn human judgment into the training and evaluation layer that enables AI models and tools to power the next generation of human creativity.\nWhy this role exists\nContra Labs is expanding its research work with frontier AI labs across evaluation, post-training data, benchmarking, and human-centered research.\nWe need a researcher who can take an ambiguous client or research question and independently turn it into a rigorous project: identify the right research direction, develop the methodology, write the proposal, run the work, analyze the results, and produce insights that shape what we or the client should do next.\nThis person should complement our existing project execution capacity by bringing deeper research ownership and the ability to generate net-new ideas and approaches.\nWhat you'll own\nResearch direction & proposals\nTranslate ambiguous client questions into clear research questions, hypotheses, and study designs.\n\nDevelop research proposals and methodologies from scratch rather than relying on a predefined approach.\n\nIdentify promising new research directions and push the team toward ideas we have not already considered.\n\nHelp define what Contra Labs should measure, study, and build next.\n\nEvaluation & data methodology\nDesign human-data collection strategies, evaluation protocols, rubrics, annotation schemes, and experiments.\n\nDevelop strong prompt strategies for eliciting, evaluating, and comparing model behavior.\n\nDetermine appropriate sampling, evaluator selection, calibration, quality-control, and reliability methods.\n\nDesign methodologies that work across creative and multimodal domains such as design, image, video, and agentic workflows.\n\nAnalysis & insight generation\nAnalyze qualitative and quantitative evaluation data.\n\nMove beyond reporting results to explain why patterns occur and what they imply.\n\nSynthesize findings into clear recommendations, research narratives, and new hypotheses.\n\nProduce client-ready reports, presentations, benchmarks, and research publications.\n\nEnd-to-end project ownership\nIndependently own research engagements from initial scoping through final delivery.\n\nWork closely with project leads, creative experts, research engineers, and data/engineering teams.\n\nMaintain a high bar for research rigor and data quality while operating on fast client timelines.\n\nPotentially support multiple concurrent projects at different stages.\n\nClient work\nBe comfortable working on research driven by real client problems and constraints.\n\nParticipate in scoping and proposal development and help establish credibility with sophisticated AI/ML research teams.\n\nDirect client-facing experience is valuable, but this is not primarily an account-management role. You will collaborate with Strategic Project Leads on client delivery.\n\nWhat we're looking for\n3-5+ years of relevant research experience, with a Master’s, PhD, or equivalent industry research depth in HCI, Human-Centered AI, ML, design research, or a related field.\n\nStrong qualitative and quantitative research skills, with experience in human evaluation, experimentation, data collection, UX/HCI, annotation, or model behavior research.\n\nProven ability to independently design and execute rigorous research, including deciding what data to collect, why, and how to analyze it.\n\nAbility to turn data and observations into new insights, research directions, and actionable recommendations.\n\nComfortable analyzing data with Python and/or SQL.\n\nStrong written communication and ability to clearly explain methodological tradeoffs and research decisions.\n\nComfortable operating independently through ambiguity and balancing research rigor with startup speed and pragmatism.\n\nBonus\nFamiliarity with modern ML and generative AI systems.\n\nUnderstanding of RLHF / post-training / preference data / model evaluation.\n\nExperience studying image, video, multimodal, creative, or agentic systems.\n\nExperience designing prompts or systematically studying prompting/model behavior.\n\nExperience working with large-scale human-data or annotation programs.\n\nExperience working directly with AI research or model teams.\n\nStrong publication record or other evidence of consistently generating original research ideas.\n\nStartup / 0-to-1 experience.\n\nRequirements\nBased in NYC, USA\n\nIn-office 5 days/week (Williamsburg)\n\nTotal Comp\nSalary: $150,000 - $200,000 USD + Equity\n\nMedical, Dental, Vision Benefits\n\n401k Matching\n\nWe will provide you with a company laptop on your start date\n\nInterview Process\nInterview with Recruiting Team (20 minutes)\n\nInterview with Head of Research (30 minutes)\n\nTechnical interview with Data Lead (30 minutes)\n\nInterview with Head of Contra Labs (30 minutes)\n\nPaid Case Study and Presentation (60 minutes)\n\nNote: Contra communicates with applicants through @contra.com domains only. We never ask for money from potential employees. For the latest job postings, visit Contra Careers.","description_format":"text","description_chars":5517,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"master","optional":true},"security_clearance":false,"languages":[]},"benefits":["401k plan","Equity"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Recruiting & Staffing","Freelance & Talent Marketplaces"],"lifecycle":[{"event":"open","at":"2026-08-17T15:48:54Z"}],"visa":[],"liveness":{"score":15,"band":"cold","label":"Long shot","p_open":1,"p_active":0.535,"p_room":0.28,"age_days":60,"expected_fill_days":30,"reasons":["conf:0","win:tail","crowd:"],"computed_at":"2026-10-17T05:46:00Z"},"pay":{"stated_usd_annual":200000,"is_top_pay":false},"html_url":"https://alion.io/job/contra-research-scientist-human-data-evaluation-2","json_url":"https://alion.io/job/contra-research-scientist-human-data-evaluation-2.json","meta":{"generated_at":"2026-10-17T21:48:45Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":6580,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":36707},"rest":"https://alion.io/mcp/rest/get_company?id=36707"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcontra-research-scientist-human-data-evaluation-2"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcontra-research-scientist-human-data-evaluation-2"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcontra-research-scientist-human-data-evaluation-2"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/contra-research-scientist-human-data-evaluation-2\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcontra-research-scientist-human-data-evaluation-2"}]}