{"id":661856,"url":"https://alion.io/job/aaru-evaluation-researcher","title":"Researcher, Evaluation","company":{"id":668926,"name":"Aaru","domain":"aaru.com","url":"https://alion.io/company/aaru","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"B","score":80,"open_postings":8,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":10,"computed_at":"2026-11-01T06:10:25Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["New York, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":200000,"max":600000,"currency":"USD","period":"year","gross":null,"usd_annual":600000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":true,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Multi-Agent Systems","optional":false},{"name":"Recommender Systems","optional":false}],"status":"live","first_seen_at":"2026-08-22T15:50:14Z","employer_posted_date":"2026-08-22","last_verified_at":"2026-11-01T09:21:14Z","board_verified":true,"closed_at":null,"days_open":70,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":70},"description":"About Aaru\nAaru builds simulations of human behavior. Each simulation contains a population of AI agents, each representing a person who could plausibly exist in the real world and capable of making decisions within a modeled environment. Companies and institutions use these simulations to test consequential choices before committing-from product launches and pricing decisions to strategic communications and policy changes.\nBuilding a useful simulation requires more than generating plausible text. Populations must represent real people and groups; predictions must be calibrated; simulations must remain coherent as conditions change; and the product must make the resulting evidence legible enough to support real decisions.\nWe are a small, in-person team in New York. We work with urgency, high ownership, and intellectual honesty. We expect people to surface inconvenient evidence, change their minds quickly, and carry important work all the way to a result.\nAbout Evaluation Research\nEvaluation Research determines whether Aaru's populations, predictions, and end-to-end simulations correspond closely enough to the real world to support consequential decisions. The team defines what should be measured, develops the methods for measuring it, and produces the evidence Aaru uses to improve its systems and describe their capabilities.\nThe function builds both rails and carts. Rails are reusable evaluation infrastructure: datasets, harnesses, libraries, experiment standards, leaderboards, reporting systems, and ways to translate technical evidence into decisions. Carts are the specific evaluations that run on those rails: a historical backtest, a prospective forecast study, a population-coherence test, a reproduction of an observed behavioral pattern, or an end-to-end comparison with a resolved outcome.\nEvaluation Research is not conventional QA and it is not benchmark administration. It is an independent research function. The work requires understanding the systems deeply, collaborating closely with their builders, and remaining willing to conclude that an attractive method did not improve what matters.\nThe role\nAs an Evaluation Researcher, you will own difficult measurement problems at the boundary of machine learning, statistics, behavioral science, and product decision-making. You will define constructs, assemble or create evaluation data, design studies, write analysis and evaluation code, inspect individual failures, quantify uncertainty, and communicate what the evidence does and does not support.\nSome projects will build reusable rails used across the research organization. Others will be focused studies intended to resolve one important uncertainty. In both cases, the goal is the same: create an evaluation that is valid enough to trust, diagnostic enough to guide improvement, and clear enough to inform a real decision.\nYou will work closely with Population Research, Prediction Research, Simulation Engineering, Product Engineering, Research Product, and Deployment while protecting the independence and integrity of final measurements.\nWhat you will do\nOwn an important evaluation or measurement area across population construction, predictive systems, individual agent behavior, group dynamics, or end-to-end simulations.\n\nTurn broad questions about realism, accuracy, calibration, usefulness, and decision quality into measurable constructs and explicit decision criteria.\n\nDesign studies using historical backtests, temporal holdouts, prospective outcomes, observational records, controlled experiments, expert judgment, or mixed methods as the problem requires.\n\nBuild tests of individual-profile quality, including internal coherence, contradictions, impossible combinations, unsupported specificity, stability, and whether a profile induces behavior consistent with the represented person.\n\nBuild tests of population quality, including marginal and joint distributions, conditional relationships, coverage of rare but plausible profiles, subgroup fidelity, and sensitivity to sampling choices.\n\nEvaluate predictions using calibration, proper scoring rules, ranking quality, selective prediction, temporal validity, subgroup performance, and the decision cost of different errors.\n\nCompare simulations with transactions, product usage, behavioral traces, operational outcomes, market movements, resolved events, surveys, and longitudinal decisions.\n\nDesign longitudinal and interaction-based evaluations that test how agents change over time, respond to new information, and influence one another.\n\nBuild end-to-end studies that determine whether a component improvement actually changes the quality of the conclusion a customer receives.\n\nEstablish strong baselines and compare agent-based simulation with direct forecasting, conventional statistical models, simpler segment-level methods, and human or market benchmarks where appropriate.\n\nFind failures hidden by aggregate metrics, especially failures concentrated in important subgroups, rare cases, changing environments, or ambiguous labels.\n\nBuild diagnostic evaluations that localize why a system failed and whether a proposed fix generalizes beyond the development set.\n\nDevelop reusable evaluation datasets, harnesses, libraries, graders, experiment schemas, leaderboards, and reporting tools where shared infrastructure will accelerate future research.\n\nWork with Simulation Engineering to version and automate evaluations while keeping protected holdouts and final claims insulated from development leakage.\n\nConvert customer surprises, production incidents, and resolved real-world outcomes into durable test cases.\n\nWrite clear technical reports that distinguish exploratory evidence from claim-supporting evidence and communicate uncertainty, limitations, and alternative interpretations.\n\nReport negative, null, and inconclusive findings with the same care as positive results.\n\nRepresentative research directions\nYou might investigate questions such as:\nConstruct a population using data available at one point in time, then test whether its future transactions, choices, or behavioral outcomes match what later occurred.\n\nRecreate a historical decision environment using only information available before the outcome and compare the simulation with the observed result.\n\nRun prospective evaluations in which Aaru records predictions before outcomes are known and tracks performance as those outcomes resolve.\n\nDetermine whether profile-coherence scores, population-distribution metrics, or forecast calibration predict end-to-end simulation quality.\n\nMeasure whether simulated groups reproduce observed patterns in information diffusion, coordination, influence, polarization, or collective choice.\n\nStudy how population size, heterogeneity, interaction structure, model capability, context length, and computation affect simulation fidelity.\n\nBuild tests for memorization, leakage, prompt sensitivity, unsupported certainty, judge bias, and benchmark-specific overfitting.\n\nDevelop evaluation methods for settings where outcomes are delayed, noisy, partially observed, selected, or open to more than one reasonable interpretation.\n\nCompare automated graders, expert review, behavioral outcomes, and human-subject measurements to determine when each is a valid proxy.\n\nEstimate whether the measured gain is large enough to matter for the decision, not merely statistically distinguishable from zero.\n\nHow we work\nA useful evaluation measures something consequential and helps the company learn. We compare systems with strong alternatives, protect held-out data, quantify uncertainty, and separate exploratory findings from evidence used to support a claim. When ground truth is imperfect, the quality and limits of the outcome data are part of the research problem rather than a footnote.\nEvaluation should make research faster by giving teams clear signals about what improved, what did not, and why. It should also make Aaru more trustworthy by exposing failures early and keeping product and external claims aligned with the evidence.\nResearchers own the full arc of the work: construct definition, data provenance, implementation, analysis, failure inspection, interpretation, communication, and the decision that follows. A polished metric without a valid construct is not success.\nYou might thrive in this role if\nYou have a record of rigorous work in ML evaluation, statistics, behavioral science, computational social science, economics, psychometrics, experimental design, or a related field.\n\nYou have built an evaluation that changed a research direction, model capability, product decision, or scientific conclusion.\n\nYou can define a difficult construct precisely enough to measure it without losing the underlying question.\n\nYou are comfortable with observational data, sampling, statistical power, uncertainty, causal threats, selection, leakage, and condition shift.\n\nYou can write code, analyze large datasets, build evaluation systems, and inspect individual examples rather than relying only on aggregate dashboards.\n\nYou naturally ask what a benchmark actually measures, what behavior it incentivizes, and where it can be gamed or contaminated.\n\nYou can work closely with a system's builders while maintaining independent judgment about its quality.\n\nYou care more about an accurate conclusion than a favorable one and are willing to revise your own evaluation when it proves inadequate.\n\nYou communicate technical evidence clearly to researchers, engineers, product teams, customers, and non-specialists.\n\nYou want to work in person in New York with a team that moves quickly and treats inconvenient evidence as valuable.\n\nStrong candidates may also have\nWork in forecasting evaluation, econometrics, psychometrics, causal inference, survey methodology, experimental economics, measurement theory, or model behavior.\n\nExperience evaluating LLM agents, multi-agent systems, synthetic populations, recommender systems, probabilistic models, simulations, or decision-support tools.\n\nExperience with longitudinal records, transaction data, product analytics, field experiments, prospective studies, or operational validation.\n\nExperience building evaluation platforms, regression suites, experiment-tracking systems, shared datasets, model scorecards, or scientific reporting tools.\n\nExperience with automated graders, human evaluation, rubric design, inter-rater reliability, benchmark contamination, or adversarial evaluation.\n\nA record of finding an important failure that standard metrics missed and developing a better way to measure it.\n\nExperience communicating scientific results in customer-facing, public, policy, legal, or regulatory settings.\n\nCandidates need not have\nPrior experience in population simulation or a job title containing the word “evaluation.”\n\nA PhD, provided you can demonstrate equivalent research depth and empirical rigor.\n\nExpertise in every statistical or machine-learning method listed above. We care most about measurement judgment, technical execution, and intellectual honesty.\n\nWhat success looks like\nYou create evaluations that resolve important uncertainty rather than merely producing additional metrics.\n\nYour measurements are valid enough to support decisions, diagnostic enough to guide improvement, and reproducible enough for others to challenge.\n\nImportant failures are found early, explained clearly, and converted into durable datasets, tests, or research questions.\n\nProtected evidence remains trustworthy while development teams still receive fast feedback.\n\nReusable rails reduce duplicated work and make results comparable across projects, systems, and versions.\n\nProduct and research claims become more precise because their supporting evidence and limitations are explicit.\n\nYour work helps Aaru distinguish a genuine general improvement from overfitting, leakage, a proxy failure, or a favorable anecdote.\n\nColleagues trust your conclusions because you combine scientific rigor with a practical und...","description_format":"text","description_chars":12526,"description_truncated":true,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"phd","optional":false},"security_clearance":false,"languages":[]},"benefits":["Equity"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":true,"industries":["Predictive Analytics","Market Research"],"lifecycle":[{"event":"open","at":"2026-09-10T21:38:19Z"}],"visa":[],"liveness":{"score":4,"band":"cold","label":"Long shot","p_open":1,"p_active":0.138,"p_room":0.28,"age_days":70,"expected_fill_days":10,"reasons":["conf:1","win:tail","crowd:"],"computed_at":"2026-11-01T06:10:25Z"},"pay":{"stated_usd_annual":600000,"is_top_pay":true},"html_url":"https://alion.io/job/aaru-evaluation-researcher","json_url":"https://alion.io/job/aaru-evaluation-researcher.json","meta":{"generated_at":"2026-11-01T09:59:13Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":15313,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-02T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":668926},"rest":"https://alion.io/mcp/rest/get_company?id=668926"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Faaru-evaluation-researcher"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Faaru-evaluation-researcher"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Faaru-evaluation-researcher"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/aaru-evaluation-researcher\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Faaru-evaluation-researcher"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":661856}},{"tool":"search_jobs","arguments":{"company":"aaru.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":661856},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}