{"id":4167443,"url":"https://alion.io/job/robots-pencils-staff-applied-scientist-2","title":"Principal Applied Scientist","company":{"id":7662,"name":"Robots & Pencils","domain":"robotsandpencils.com","url":"https://alion.io/company/robots-and-pencils","size_band":"201-500","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"B","score":82,"open_postings":7,"ghost_share":0,"stale_share":0.714,"repost_share":0,"time_to_fill_p50_days":32,"computed_at":"2026-11-12T10:38:37Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"lead","employment_type":null,"work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"posting_text","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":["US"],"hiring_countries_total":1,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Claude Code","optional":false},{"name":"Langfuse","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"Prompt Caching","optional":false},{"name":"Python","optional":false}],"status":"live","first_seen_at":"2026-11-12T13:33:04Z","employer_posted_date":"2026-11-12","last_verified_at":"2026-11-12T20:20:17Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"We're looking for a Principal Applied Scientist to lead the design and build of an evaluation framework for an enterprise planner-based agent. You will work directly with the client's engineering team to benchmark their agent across planner models, executor LLMs and inference optimization levers. Every proposed change, whether a model upgrade, a harness change or a routing strategy, becomes a controlled, statistically sound experiment with a measurable dollar outcome.\nThis is a research-heavy, customer-facing role with a significant build component. You will own both the experimental method and the tooling that makes it repeatable: a scenario-driven user simulator, isolated experiment execution, statistical comparison, and automated insights in a locally hosted dashboard the client can adopt as their own. The work runs in three overlapping streams (evaluation pipeline, grading and statistics, dashboard and experiment campaign) and ends in a recommendation the client can act on.\nWhy This Role Matters\nAt Robots & Pencils, we design AI systems for a human world. Our name says it all. Robots and pencils means engineering paired with creativity, because every agent we ship has to work for real people in real workflows. That balance is baked into how we operate.\nAgents are only as good as the evidence behind them. Today, most teams pick a model or switch on a cost lever on instinct. Here, you replace instinct with data: you build the system that tells the client which planner model, which executor and which levers to roll out, how much quality each change costs or saves, and what it is worth in dollars at production volume. You also show how well the planner carries over from data analytics to tasks. This is a research role with real build accountability, and your work becomes the client's own decision-making tool. It demands people who take ownership, lead with craft, and care deeply about what they put their name on.\nWhat You'll Do\nEvaluation Pipeline & Simulation\nDesign the end-to-end evaluation workflow for a planner-executor agent: scenarios, user simulation, configuration, execution, grading, comparison and insights\nDefine the scenario schema, grading spec and validators with the client's team, who own scenario creation, so their data plugs directly into the framework (each scenario has a goal, persona, starting state and expected end state)\nBuild an LLM-based user simulator with a hidden goal and a stop rule that holds realistic multi-turn conversations, and check its realism against individual production logs\nBuild isolated, parallel experiment execution: config files, one run per config and scenario, and a harness adapter that routes all model traffic through the client's existing LLM gateway\nPackage optimization levers as plug-ins and implement a shortlist of 3 to 4 for v1, such as cascade-on-failure, cache-aware model switching and prompt caching\nTag every model call at the gateway and send traces to a trace store, so tokens, cache usage, latency and cost are attributed to each request\nGrading, Statistics & Insights\nGrade each run against the expected end state as the primary signal, with optional LLM judges for answer and plan quality\nCompute the core metrics: verified success, cost per success, p95 latency, and failures traced to pipeline stage\nCompare experiments on the same scenarios with paired methods: confidence intervals, win/loss/tie per scenario, and a non-inferiority test that a cost lever does not drop quality past an agreed margin. Say clearly when a result is inconclusive\nConvert differences into projected dollar savings at production volume using real deployment costs, not just list token prices, and plot the cost-versus-quality frontier\nBuild an automated insight engine where a statistics layer computes every number and an LLM only writes them up, so each ranked finding links back to its evidence\nDashboard, Experiments & Readout\nBuild the locally hosted dashboard (Launch, Live, Results, Compare, Insights and Traces views), API-first with one-command local deployment so it can be integrated into the client's platform later\nRun the staged campaign: production baseline, model screen, lever screen, then confirm the best combination\nAnalyze generalization by scenario category, optionally alongside a small data-analytics anchor set\nDeliver a final recommendation with confidence intervals and dollar impact, plus the runbook and integration guide that support go/no-go decisions\nCollaboration & Technical Leadership\nPartner with the client's engineers and leadership to agree on scenarios, margins and success criteria, and explain findings to both scientists and business stakeholders\nWrite clear technical reports and leadership summaries that translate results into business impact\nSet the experimental standards for the pod and mentor other engineers on rigorous agent evaluation, including AI-assisted development tools like Claude Code\nWhat You'll Bring\nExperience with LLM and agent evaluation methods, such as programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring and user simulation\nExperience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production, not just prototypes\nA strong background in experimental design and statistical inference, including paired designs, non-inferiority testing and power analysis, and the judgment to call a result inconclusive\nExperience with multi-objective optimization or cost-aware ML, including Pareto frontier analysis and converting results into projected dollar savings\nFamiliarity with LLM routing, model cascades or model selection, and an understanding of LLM inference internals and cost levers (prompt and plan caching, context compaction, reasoning budgets, cache-aware model switching)\nExperience with observability and tracing for LLM systems, including attributing tokens, cache usage, latency and cost per request\nThe ability to build production-quality research tooling: Python services, containerized and sandboxed execution, and data pipelines\nPrior customer-facing or consulting experience where you scoped and delivered research engagements\nThe ability to explain technical findings clearly to non-technical audiences and translate them into business impact, and to go a level deeper when someone pushes on your answer instead of staying abstract\nHelpful Extras and Unique Skills\nPublications or open-source contributions in LLM evaluation, agents or efficient inference\nExperience building internal dashboards or web UIs that a client team can adopt as their own\nHands-on use of an LLM gateway or tracing stack, such as LangFuse or LangSmith, to attribute cost and latency at the request level\nExperience in a regulated or high-stakes enterprise domain where evaluation rigor carries real consequences\nA point of view on cost-quality tradeoffs gained from running them in production rather than reading about them\nOur salary range is $200,00 - $300,000","description_format":"text","description_chars":7000,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["UI/UX Design","Custom Software Development","AI Consulting & Integration"],"lifecycle":[{"event":"open","at":"2026-11-12T13:48:52Z"}],"visa":[{"country":"US","licensed_sponsor":true,"adverts_offering_sponsorship_24m":0,"sanctioned_now":false,"evidence":"Filed H-1B visas in the last two years","filings_12m":0,"filings_prev_12m":1,"green_card_filings_12m":0,"median_offered_wage_usd":null,"route":null,"cap_exempt":false,"checked_at":"2026-11-12T07:36:15+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":0}],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":32,"reasons":["conf:2","win:early"],"computed_at":"2026-11-12T23:14:18Z"},"pay":null,"html_url":"https://alion.io/job/robots-pencils-staff-applied-scientist-2","json_url":"https://alion.io/job/robots-pencils-staff-applied-scientist-2.json","meta":{"generated_at":"2026-11-12T23:14:18Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":49131,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-13T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":7662},"rest":"https://alion.io/mcp/rest/get_company?id=7662"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Frobots-pencils-staff-applied-scientist-2"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Frobots-pencils-staff-applied-scientist-2"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Frobots-pencils-staff-applied-scientist-2"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/robots-pencils-staff-applied-scientist-2\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Frobots-pencils-staff-applied-scientist-2"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4167443}},{"tool":"search_jobs","arguments":{"company":"robotsandpencils.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4167443},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}