{"id":1310812,"url":"https://alion.io/job/aslan-protects-inc-evals-lead","title":"Evals Lead","company":{"id":2390703,"name":"Aslan protects, Inc.","domain":"aslanprotects.com","url":"https://alion.io/company/aslanprotects","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"B","score":75,"open_postings":3,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-10-22T06:10:10Z"}},"role":"Field & Service","role_family":"Field & Service","seniority":"lead","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Washington, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":50000,"max_usd":105000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":1070},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Arize Phoenix","optional":false},{"name":"Braintrust","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"Promptfoo","optional":false},{"name":"Python","optional":false},{"name":"Reward Modeling","optional":false}],"status":"live","first_seen_at":"2026-08-10T04:16:11Z","employer_posted_date":"2026-08-10","last_verified_at":"2026-10-22T21:39:12Z","board_verified":true,"closed_at":null,"days_open":73,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":73},"description":"About Aslan\nOur national security apparatus was designed for an adversary that congregated and operate in the physical world. The threats disrupting our way of life today have gone faceless, borderless, and beyond the reach of any human operator. Aslan builds the autonomous agents that reach them, unmask them, and stop them, before a life can be lost, a household can be bankrupted, innocence can be stolen, or our can be nation subverted.\nIn 12 months we've gone from founding to live operational tasking and pilots across multiple national security and law enforcement partners. We've proven it works. This role makes it permanent.\nWe’ve raised $20M to date from Khosla Ventures, XYZ, BoxGroup, 2048, Liquid 2, Precursor Ventures, and others.\nRole\nAslan builds autonomous agents that run for weeks at a time across many external systems, plan their own next steps, and work toward an objective. Scoring them the usual way doesn't work. Individual outputs look fine. Results come back late or not at all. What we need to know is whether the agent chose well at the moment it chose, and nothing off the shelf measures that.\nThis is our first quality role. You'll operate the platform daily, build the tooling that catches what you catch by hand, and own the decision to ship. We deploy on-premise on a fixed cadence and can't push fixes after delivery.\nResponsibilities\nRun the platform yourself every day and log what's wrong in enough detail that an engineer can reproduce it.\n\nSample agent plans, get them rated on whether the agent made the right call, and turn the ratings into a versioned dataset. Engineering uses it as a regression suite. The training side uses it as labels.\n\nDiagnose whether a bad decision came from the model or from the harness handing it the wrong context, and route each to the right owner.\n\nWrite and run checks over each agent's full history, and over properties that only hold across the whole set of running agents.\n\nBuild a simulation harness so a week of agent operation runs overnight against a candidate build.\n\nDefine what counts as a regression between builds. Measure both whether agents last and whether they get anywhere. A build that only makes them cautious is a failed build.\n\nWho You Are\nYou've owned evals for a shipped LLM or agent product, and you can say what your suite caught and what it missed\n\nYou've built a labeled dataset that got used for training, not just for measurement\n\nYou've picked a proxy metric because the real signal wasn't available, and you can defend the choice\n\nYou know when LLM-as-judge works and when it doesn't\n\nYou'd rather write the check than file the ticket\n\nWhat You Bring\nStrong Python\n\nAn evals framework you've run in production: Inspect, Promptfoo, Braintrust, Arize, LangSmith or equivalent\n\nFamiliarity with agent harness internals: memory, context assembly, tool selection\n\nPreference data, process supervision, or reward modeling experience\n\nEnough statistics to tell a regression from noise on small samples\n\nBonus Points\nEvaluating agents that run for days or weeks, where per-episode metrics don't work\n\nSystems you can't roll back or reset\n\nGraph stores, provenance, lineage tracking\n\nEligible for a US security clearance","description_format":"text","description_chars":3219,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Homeland Security","Security Services"],"lifecycle":[{"event":"open","at":"2026-09-26T16:06:46Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 1","filings_12m":1,"filings_prev_12m":0,"green_card_filings_12m":0,"median_offered_wage_usd":127500,"route":null,"cap_exempt":false,"checked_at":"2026-10-15T11:24:14+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":0}],"liveness":{"score":14,"band":"cold","label":"Long shot","p_open":1,"p_active":0.496,"p_room":0.28,"age_days":73,"expected_fill_days":27,"reasons":["conf:2","win:tail","crowd:"],"computed_at":"2026-10-22T06:10:10Z"},"pay":null,"html_url":"https://alion.io/job/aslan-protects-inc-evals-lead","json_url":"https://alion.io/job/aslan-protects-inc-evals-lead.json","meta":{"generated_at":"2026-10-23T00:38:29Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":1131,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-24T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2390703},"rest":"https://alion.io/mcp/rest/get_company?id=2390703"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Faslan-protects-inc-evals-lead"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Faslan-protects-inc-evals-lead"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Faslan-protects-inc-evals-lead"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/aslan-protects-inc-evals-lead\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Faslan-protects-inc-evals-lead"}]}