{"id":1510328,"url":"https://alion.io/job/firecrawl-research-engineer-benchmarks","title":"Research Engineer - Benchmarks","company":{"id":2391,"name":"Firecrawl","domain":"firecrawl.dev","url":"https://alion.io/company/firecrawl","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"middle","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"explicit","locations":["San Francisco, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":250000,"max":290000,"currency":"USD","period":"year","gross":null,"usd_annual":290000},"salary_estimate":null,"experience_years_min":4,"visa_sponsorship":true,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Firecrawl","optional":false},{"name":"GitHub","optional":false},{"name":"LLM","optional":false},{"name":"Python","optional":false}],"status":"live","first_seen_at":"2026-09-29T23:12:06Z","employer_posted_date":"2026-09-29","last_verified_at":"2026-10-01T03:10:32Z","board_verified":true,"closed_at":null,"days_open":1,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":1},"description":"Research Engineer - Benchmarks\nEvery week, someone asks which data provider is actually best: for company records, for finding the right person, for fresh job listings. Right now the answers come from the vendors themselves. You'll build the independent version. You'll run rigorous, automated, public benchmarks of the providers in Alexandria and beyond, and publish them weekly. The same results will feed straight back into Alexandria so it learns which provider to call for which job.\nThis isn't our internal evals role. You're measuring the market, in public, where every number will be challenged by the vendors it ranks. You'll own it end to end: the datasets, the ground truth, the scoring, the harness, the weekly release, and the loop into Alexandria's provider selection. No one hands you a methodology. You write it, defend it, and ship it every week.\nSalary Range: $250,000-$290,000 USD/year (SF) / $210,000-$224,000 CAD/year (Toronto)\nEquity Range: Competitive equity. Details shared during the process.\nLocation: San Francisco, CA (SF HQ) or Toronto, ON (Toronto Hub). Hybrid, onsite 3+ days a week.\nEquity Range: Competitive equity. Details shared during the process.\nLocation: San Francisco, CA (SF HQ). On-site, five days a week.\nJob Type: Full-Time\nExperience: 4+ years in ML, research engineering, or data engineering, with evaluation or benchmark work you've shipped\nWork Authorization: Must be authorized to work in the United States or Canada. We're not able to sponsor US visas right now. For Canada, we'll consider sponsorship on a case-by-case basis through our Toronto Hub.\nAbout Firecrawl\nFirecrawl is the easiest way to turn the web into data AI agents can use. One API call converts any URL into clean, LLM-ready markdown or structured data. It's the boring-hard problem everyone building with LLMs eventually hits, solved.\nIn September 2026, we raised a $75M Series B led by Smash Capital, and we're spending it building the largest repository of knowledge in the world. We hit 8 figures in ARR in year one and more than doubled it in year two. We have 187k+ GitHub stars, putting us in the top 40 repositories of all time, and developers, agents, and category-defining AI companies build on us every day. Growth like this is rare, and we're just getting started.\nWe're a small team punching far above our weight. Everyone here owns a real piece of the product and company, end to end, and runs it themselves. No hiding behind process or headcount.\nThis is a place for people who want to work at the frontier: an AI company building the infrastructure other AI companies run on, not one bolting AI onto an existing product. We move fast, go deep, and are building the tools superintelligence will rely on to gather data from the web. That library is called Alexandria, and it starts now.\nWhat You'll Do\nDesign and run head-to-head benchmarks of data providers against verified ground truth. For example:\nApollo vs. FullEnrich vs. DataLegion on company details.\n\nFullEnrich vs. DataLegion on finding the right person and their current employer and title.\n\nBuilt In vs. ZipRecruiter on relevant, fresh, non-duplicate job listings.\n\nBuild and maintain the test datasets and ground truth, and keep them from going stale or leaking.\n\nOwn the automated harness and the weekly benchmark release. Every run has to be reproducible, versioned, and defensible.\n\nMeasure what buyers actually care about: accuracy, coverage, freshness, speed, and cost.\n\nWork with marketing to ship public leaderboard pages that are useful and hold up under scrutiny.\n\nClose the loop into Alexandria so benchmark results change which provider gets called for what.\n\nKeep expanding the categories we benchmark, inside Alexandria and beyond it.\n\nWhat We're Looking For\nYou've shipped evals or benchmarks, and you can explain exactly why your results were trustworthy.\n\nYou've done this somewhere that matters: a frontier lab, a data company like Scale, Surge, micro1 or Mercor, a third-party benchmark org, or a public benchmark project. OSS contributors very welcome.\n\nYou're strong in Python and API integrations, and comfortable with messy vendor APIs, rate limits, and inconsistent schemas.\n\nYou know how to build test sets and scoring methods: sampling, labeling, inter-rater agreement, and when to trust an LLM judge and when not to.\n\nYou use AI heavily and keep upgrading your own workflow. You ship without waiting for instructions.\n\nYou write findings clearly for engineers, marketers, and the vendors on the other side of the leaderboard.\n\nWhat We're Not Looking For\nSomeone who wants to run benchmarks someone else designed.\n\nSomeone who picks the metric that makes the story look good. Our numbers have to survive the vendors who lose.\n\nA pure researcher who won't build the harness, or a pure engineer who won't think hard about methodology.\n\nSomeone who needs a fully specced ticket, or a quarter, to ship the first result.\n\nA Note On Pace\nWe operate at an absurd level of urgency because the window for what we're building won't stay open forever. If that excites you, keep reading. If it doesn't, no hard feelings, but this role probably isn't for you.\nBenefits & Perks\nAvailable to all employees\nSalary that makes sense: $250,000-$290,000 USD/year, / $210,000-$224,000 CAD/year (Toronto), based on impact, not tenure\n\nOwn a piece: Gain competitive equity in what you're helping build\n\nGenerous PTO: 15 days mandatory, anything after 24 days, just ask (holidays excluded). Take the time you need to recharge\n\nParental leave: 12 weeks fully paid, for all parents\n\nWellness stipend: $100 USD/month for the gym, therapy, massages, or whatever keeps you human\n\nLearning & Development: Expense up to $1,000 USD/year toward anything that helps you grow professionally\n\nTeam offsites: A change of scenery, minus the trust falls\n\nSabbatical: 3 paid months off after 4 years, do something fun and new\n\nAvailable to US-based full-time employees\nFull coverage, no red tape: Medical, dental, and vision (100% for employees, 50% for partner and kids). No weird loopholes, just care that works\n\nLife & Disability insurance: Employer-paid basic life and AD&D, short-term disability, and long-term disability. Coverage for life's curveballs\n\nVirtual care and a health guide: Teladoc for the couch doctor visit, plus Rightway to answer coverage questions and fight billing errors for you\n\nMental health: Talkspace, therapy and psychiatry on your schedule\n\nFertility and family building: Carrot, covering you and your partner\n\nEAP: Free confidential counseling, legal and financial consults, and online will prep through Guardian\n\n401(k) plan: Retirement might be a ways off, but future-you will thank you\n\nPre-tax benefits: HSA, FSA, and commuter benefits to help your wallet out a bit\n\nSupplemental options: Extra life and AD&D, accident, critical illness, hospital indemnity, plus pet, legal, and identity protection through MetLife\n\nAvailable to Canada-based full-time employees\nFull coverage, no red tape: Extended health, dental, and vision through Manulife (Diamond, the top tier), 100% employer-paid for you, your partner, and your kids\n\nLife & Disability insurance: Employer-paid life, AD&D, short-term disability, and long-term disability. Coverage for life's curveballs\n\nVirtual care: Dialogue Premium, so you can see a doctor or nurse from your couch, any hour\n\nMental health: Talkspace Elite, therapy and psychiatry on your schedule\n\nFertility and family building: Carrot, covering you and your partner\n\nRetirement: Group RRSP through Wealthsimple, so future-you can thank you\n\nAvailable to SF-based employees\nSF HQ perks: Snacks, drinks, team lunches, intense ping pong, and peak startup energy\n\nE-Bike transportation: A loaner electric bike to get you around the city, on us\n\nAvailable to Toronto-based employees\nToronto Hub perks: Snacks, drinks, team lunches, glass-walled views down University Avenue, and a home base steps from Union Station\n\nTransit, covered: A PRESTO card loaded for GO Transit, subway, and streetcar, plus station parking if you drive to the train. Winter-proof, on us\n\nInterview Process\nApplication Review: Send us your work. We want the benchmark, leaderboard, eval, or dataset you built, and how you knew it was right. We care about what you've shipped, not where you went to school.\n\nIntro Chat (~25 min): A quick conversation to get to know each other. We'll cover what you've been working on, what drew you to Firecrawl, and what you want next. Time for your questions too.\n\nTechnical Chat (~45 min): A real problem from our world: design a benchmark that decides whether FullEnrich or DataLegion finds the right person's current title. We'll cover where the ground truth comes from and how you'd defend the result to the vendor who loses. Come ready to think out loud.\n\nWorkflow Chat (~30 min): Show us how you actually work: your AI tools, your setup, and a recent thing you shipped faster than you would have a year ago.\n\nFounder Chat (~25 min): Culture, pace, ownership, and how you like to work. Time for your questions too.\n\nPaid Work Trial (~40 hours): Ship a small benchmark end to end on a real provider category, paid at a contractor rate. It's the truest signal for both sides. Remote-friendly, and we'll flex around your current commitments.\n\nDecision: We move fast after the trial.\n\nIf you want to be the person the whole market checks before picking a data provider, you should join us.\nApply now.","description_format":"text","description_chars":9453,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Equity","Parental leave"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"},{"name":"Canada","iso":"CA","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["AI Development Tools","Vector Search & RAG","Developer Tools","Web Scraping & Proxy Networks"],"lifecycle":[{"event":"open","at":"2026-09-30T07:43:10Z"}],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":1,"expected_fill_days":38,"reasons":["conf:0","win:early"],"computed_at":"2026-10-01T03:25:26Z"},"pay":{"stated_usd_annual":290000,"is_top_pay":true},"html_url":"https://alion.io/job/firecrawl-research-engineer-benchmarks","json_url":"https://alion.io/job/firecrawl-research-engineer-benchmarks.json","meta":{"generated_at":"2026-10-01T03:25:26Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","usage":{"tier":"crawler","counted_by":"address","units_charged":1,"used_today":2871,"day_limit":5000,"remaining_today":2129,"minute_limit":60,"resets_at":"2026-10-02T00:00:00Z"}}}