{"id":661849,"url":"https://alion.io/job/aaru-evaluation-research-manager","title":"Research Manager, Evaluation","company":{"id":668926,"name":"Aaru","domain":"aaru.com","url":"https://alion.io/company/aaru","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"B","score":80,"open_postings":12,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":10,"computed_at":"2026-11-07T05:47:08Z"}},"role":"Science & Research","role_family":"Science & Research","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["New York, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":280000,"max":425000,"currency":"USD","period":"year","gross":null,"usd_annual":425000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":true,"relocation_package":true,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Multi-Agent Systems","optional":false},{"name":"Recommender Systems","optional":false}],"status":"live","first_seen_at":"2026-08-22T15:51:03Z","employer_posted_date":"2026-08-22","last_verified_at":"2026-11-07T11:11:15Z","board_verified":true,"closed_at":null,"days_open":76,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":76},"description":"About Aaru\nAaru builds simulations of human behavior. Each simulation contains a population of AI agents, each representing a person who could plausibly exist in the real world and capable of making decisions within a modeled environment. Companies and institutions use these simulations to test consequential choices before committing-from product launches and pricing decisions to strategic communications and policy changes.\nBuilding a useful simulation requires more than generating plausible text. Populations must represent real people and groups; predictions must be calibrated; simulations must remain coherent as conditions change; and the product must make the resulting evidence legible enough to support real decisions.\nWe are a small, in-person team in New York. We work with urgency, high ownership, and intellectual honesty. We expect people to surface inconvenient evidence, change their minds quickly, and carry important work all the way to a result.\nAbout Evaluation Research\nEvaluation Research determines whether Aaru's populations, predictions, and end-to-end simulations correspond closely enough to the real world to support consequential decisions. The team defines what should be measured, develops the methods for measuring it, and produces the evidence Aaru uses to improve its systems and describe their capabilities.\nThe function builds both rails and carts. Rails are reusable evaluation infrastructure: datasets, harnesses, libraries, experiment standards, leaderboards, reporting systems, and ways to translate technical evidence into decisions. Carts are the specific evaluations that run on those rails: a historical backtest, a prospective forecast study, a population-coherence test, a reproduction of an observed behavioral pattern, or an end-to-end comparison with a resolved outcome.\nEvaluation Research is not conventional QA and it is not an internal approval service. It is an independent research function that works closely with the teams building Aaru's systems while preserving the ability to reach and communicate inconvenient conclusions.\nThe role\nAs Evaluation Research Manager, you will lead a focused team of Evaluation Researchers and research engineers. You will translate Aaru's evaluation charter into a coherent portfolio of studies, datasets, and shared infrastructure, and you will be accountable for the quality, pace, and usefulness of the team's work.\nManagers at Aaru remain researchers. You will design studies, write analysis code, inspect individual failures, review statistical and measurement choices, and directly contribute to the hardest evaluations. You will also set priorities, hire exceptional people, develop the team, provide candid feedback, and create the operating mechanisms that keep protected evidence independent while making diagnostic evidence available quickly.\nYou will work across Population Research, Prediction Research, Simulation Engineering, Product Engineering, Research Product, Deployment, and company leadership. The job requires both scientific independence and practical judgment: an evaluation must be rigorous enough to support a real claim and diagnostic enough to help a team improve the system.\nWhat you will do\nBuild, lead, and develop a high-performing team of Evaluation Researchers and research engineers.\n\nTurn broad questions about realism, accuracy, calibration, usefulness, and decision quality into measurable constructs, decisive experiments, and explicit decision criteria.\n\nSet a focused evaluation agenda across population construction, predictive systems, individual agent behavior, group dynamics, and end-to-end simulations.\n\nDecide which evaluation infrastructure should become a reusable organizational rail and which questions require a purpose-built study.\n\nEstablish standards for baselines, temporal holdouts, prospective testing, contamination control, statistical power, uncertainty, subgroup analysis, and reproducibility.\n\nBuild tests of population quality that assess individual coherence, joint and conditional distributions, representation of rare but plausible profiles, and whether a profile induces behavior consistent with the person it represents.\n\nEvaluate forecasts and other predictive outputs using calibration, proper scoring rules, ranking quality, selective prediction, temporal validity, subgroup performance, and the real cost of different errors.\n\nCompare simulations with transactions, product usage, behavioral traces, operational outcomes, market movements, resolved events, and longitudinal decisions.\n\nDesign end-to-end studies that determine whether improvements to a component actually improve the decision-relevant output customers receive.\n\nFind failures hidden by aggregate metrics, especially those concentrated in important subgroups, rare cases, changing environments, or ambiguous ground truth.\n\nCreate diagnostic evaluations that help researchers localize why a system failed and distinguish a real general improvement from benchmark-specific optimization.\n\nPartner with Simulation Engineering to make evaluations repeatable, versioned, scalable, and integrated into development and release workflows without compromising protected holdouts.\n\nConvert production incidents, customer surprises, and deployment failures into durable test cases and better measurement methods.\n\nReview evidence used in product, customer, or public claims and ensure that conclusions are reproducible, appropriately scoped, and honest about uncertainty and limits.\n\nCommunicate negative, null, and inconclusive results with the same precision and urgency as positive findings.\n\nRecruit exceptional researchers, set clear expectations, provide direct feedback, develop independent research judgment, and address performance problems early.\n\nRepresentative research and leadership problems\nYou might be responsible for situations such as:\nTwo teams disagree about whether a system improved because they use different metrics and test sets. Identify the underlying construct, choose the right evidence, and create a shared evaluation that resolves the disagreement.\n\nA population looks plausible one profile at a time but fails to reproduce important real-world relationships. Build measurements that expose the gap and help Population Research identify its cause.\n\nA prediction method is well calibrated overall but systematically overconfident for a high-value subgroup. Determine whether the issue is data coverage, model structure, selection, condition shift, or the evaluation itself.\n\nAn offline benchmark has become a development target and is beginning to leak into decisions. Redesign the evaluation system so teams can iterate quickly without exhausting the integrity of the final holdout.\n\nA component metric improves, but customer decisions do not. Determine whether the metric is invalid, the effect is too small, downstream components erase the gain, or the product is presenting the result incorrectly.\n\nGround truth is delayed, noisy, incomplete, or open to multiple interpretations. Design a study that remains useful without pretending that the label is cleaner than it is.\n\nA customer outcome contradicts the simulation. Reconstruct the information available at decision time, identify the relevant comparison, and convert the failure into a fair and reproducible test.\n\nEvaluation work is becoming a collection of bespoke notebooks. Choose the common abstractions, data contracts, and reporting systems that should become durable rails without freezing the research too early.\n\nA favorable result is strategically important but methodologically weak. Communicate the limitation clearly, resist pressure to overclaim, and propose the fastest credible path to stronger evidence.\n\nHow we work\nA useful evaluation measures something consequential and helps the company learn. We compare systems with strong alternatives, protect held-out data, quantify uncertainty, and separate exploratory findings from evidence used to support a claim. When ground truth is imperfect, the quality and limits of the outcome data are part of the research problem rather than a footnote.\nEvaluation should accelerate research by giving teams clear, diagnostic signals about what improved, what did not, and why. It should also make Aaru more trustworthy by exposing failures early and keeping product and external claims aligned with the available evidence.\nManagement in this function requires independence without isolation. The team must understand the systems deeply enough to measure them well, collaborate closely enough to make the results useful, and remain willing to conclude that an attractive idea did not work.\nYou might thrive in this role if\nYou have led evaluation, measurement, or empirical research in machine learning, behavioral science, computational social science, statistics, economics, psychometrics, or a comparably rigorous environment.\n\nYou have built evaluations that changed a research direction, model capability, product decision, or scientific conclusion.\n\nYou can define a difficult construct precisely enough to measure it without reducing away the underlying question.\n\nYou are comfortable with experimental design, observational data, sampling, statistical power, uncertainty, causal threats, leakage, and condition shift.\n\nYou can write code, analyze large datasets, design studies, inspect individual failures, and review the technical work of researchers and engineers.\n\nYou can work closely with builders while reaching independent conclusions about the quality of their systems.\n\nYou care more about an accurate result than a favorable one and are willing to revise your own evaluation when evidence shows it is inadequate.\n\nYou can prioritize a research portfolio and choose which uncertainty is most important to resolve next.\n\nYou have managed or technically led strong researchers, give clear feedback, and can develop independent judgment rather than creating dependence on your review.\n\nYou can explain technical evidence clearly to researchers, engineers, product teams, customers, company leadership, and non-specialists.\n\nYou want to work in person in New York with a team that moves quickly and takes truth-seeking seriously.\n\nStrong candidates may also have\nWork in ML evaluation, model behavior, forecasting, econometrics, psychometrics, causal inference, experimental economics, survey methodology, or measurement theory.\n\nExperience evaluating LLM agents, multi-agent systems, synthetic populations, recommender systems, probabilistic models, simulations, or decision-support tools.\n\nExperience with longitudinal records, transaction data, product analytics, field experiments, prospective studies, backtesting, or validation against operational outcomes.\n\nExperience building evaluation platforms, regression suites, experiment-tracking systems, shared research datasets, model scorecards, or scientific reporting tools.\n\nA record of finding an important failure that standard metrics missed and developing a better measurement method.\n\nExperience communicating scientific results in customer-facing, public, policy, legal, or regulatory settings.\n\nExperience hiring and leading a small, high-talent research team through ambiguous work with short feedback cycles.\n\nCandidates need not have\nA career spent exclusively in AI evaluation or prior experience with Aaru's exact domain.\n\nA PhD, provided you have equivalent evidence of rigorous empirical work and research leadership.\n\nManaged managers or a large organization; this role is about leading a focused team and remaining directly involved in the research.\n\nWhat success looks like\nAaru has clear, widely trusted definitions of quality across populations, predictions, and end-to-end simulations.\n\nCore evaluations are grounded in observed outcomes and reveal whether performance generalizes across time, domains, populations, and subgroups.\n\nResearchers receive fast, diagnostic measurements while protected holdouts preserve the integrity of final...","description_format":"text","description_chars":13297,"description_truncated":true,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"phd","optional":false},"security_clearance":false,"languages":[]},"benefits":["Equity"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":true,"industries":["Predictive Analytics","Market Research"],"lifecycle":[{"event":"open","at":"2026-09-10T21:38:19Z"}],"visa":[{"country":"US","licensed_sponsor":false,"adverts_offering_sponsorship_24m":24,"sanctioned_now":false,"evidence":"Offered visa sponsorship in 24 of its adverts here in 24 months","filings_12m":0,"filings_prev_12m":0,"green_card_filings_12m":0,"median_offered_wage_usd":null,"route":null,"cap_exempt":false,"checked_at":"2026-11-07T07:57:00+00:00","sources":["Alion: sponsorship offered in the employer's own job adverts (24 months)"],"filings_for_role_12m":0}],"liveness":{"score":3,"band":"cold","label":"Long shot","p_open":1,"p_active":0.1,"p_room":0.28,"age_days":76,"expected_fill_days":10,"reasons":["conf:2","stale_co","win:tail","crowd:"],"computed_at":"2026-11-07T05:47:08Z"},"pay":{"stated_usd_annual":425000,"is_top_pay":true},"html_url":"https://alion.io/job/aaru-evaluation-research-manager","json_url":"https://alion.io/job/aaru-evaluation-research-manager.json","meta":{"generated_at":"2026-11-07T11:57:47Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":12773,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-08T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":668926},"rest":"https://alion.io/mcp/rest/get_company?id=668926"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Faaru-evaluation-research-manager"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Faaru-evaluation-research-manager"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Faaru-evaluation-research-manager"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/aaru-evaluation-research-manager\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Faaru-evaluation-research-manager"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":661849}},{"tool":"search_jobs","arguments":{"company":"aaru.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":661849},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}