{"id":1194345,"url":"https://alion.io/job/thinking-machines-lab-web-crawling-research-engineer","title":"Research Engineer, Web Crawling","company":{"id":14382,"name":"Thinking Machines Lab","domain":"thinkingmachines.ai","url":"https://alion.io/company/thinking-machines-lab","size_band":"501-1000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["San Francisco, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":350000,"max":475000,"currency":"USD","period":"year","gross":null,"usd_annual":475000},"salary_estimate":null,"experience_years_min":8,"visa_sponsorship":true,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Python","optional":false},{"name":"Rust","optional":false},{"name":"Machine Learning","optional":true}],"status":"live","first_seen_at":"2026-09-24T16:55:05Z","employer_posted_date":"2026-09-24","last_verified_at":"2026-10-05T01:43:33Z","board_verified":true,"closed_at":null,"days_open":10,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":10},"description":"About Thinking Machines\nThe mission of Thinking Machines is to build AI that extends human will and judgment. We are training frontier models with Inkling, developing Tinker to let people make models their own, and crafting interfaces that broaden human-AI communication. We believe the future worth building is human, and we're hiring people who want to build it.\nAbout the Role\nWe're hiring a Software Engineer to build and own our web-crawling systems, from distributed collection at internet scale through filtering, deduplication, and deciding what data we keep.\nThe ideal candidate has built and scaled a web crawler or large-scale data-acquisition systems. In this role, you'll write and own production systems: the crawler itself, the infrastructure that runs it at scale, and the pipelines that turn raw crawls into usable pretraining data. You'll work closely with our pretraining and data teams to understand what's actually moving model quality, but this is fundamentally an engineering role, not a research one.\nWhat You'll Do\nDesign and scale the web crawler and ingestion infrastructure that sources Inkling's pretraining data\n\nBuild pipelines for large-scale extraction, deduplication, and data quality filtering\n\nBuild specialized crawlers for high-value or hard-to-reach data sources\n\nWork with the pretraining team to understand how changes in crawled data affect model performance\n\nImprove the reliability and efficiency of crawling and ingestion infrastructure at petabyte scale\n\nHelp set technical direction for this area as it grows, and bring other engineers up to speed on what you've learned\n\nSkills & Qualifications\nMinimum Qualifications\n8+ years designing, building, and scaling web crawlers, scrapers, or large-scale distributed data-acquisition systems\n\nA track record of owning crawler or data-acquisition infrastructure at internet scale\n\nStrong software engineering skills in a language such as Python, Go, or Rust, with real experience in distributed systems\n\nWorking knowledge of the practical and legal considerations of large-scale web data collection (robots.txt, rate limiting, licensing)\n\nPreferred Qualifications\nExperience applying machine learning to crawl selection, extraction, or data quality classification at internet scale\n\nExperience setting technical direction for a crawling, data acquisition, or search infrastructure team, whether or not that was your formal title\n\nExperience designing systems for petabyte-scale storage and processing\n\nTrack record of open-source contributions to crawling, scraping, or data infrastructure tools\n\nBackground at a search engine (crawling, indexing) or a frontier AI lab's data acquisition team\n\nLogistics\nLocation: This role is based in San Francisco, CA.\n\nCompensation: Depending on background, skills and experience, the expected annual salary range for this position is $350,000-$475,000 USD.\n\nVisa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.\n\nBenefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.\n\nAs set forth in Thinking Machines' Equal Employment Opportunity policy, we do not discriminate on the basis of any protected group status under any applicable law.\nThinking Machines Lab will consider for employment qualified applicants with criminal histories in a manner consistent with the requirements of the California Fair Chance Act, the San Francisco Fair Chance Ordinance, and any other applicable state or local fair chance ordinance or law.","description_format":"text","description_chars":3675,"description_truncated":false,"requirements":{"experience_years_min":8,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Parental leave","Unlimited PTO"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":true,"industries":["Multimodal AI","Foundation Models","AI Research Labs","AI Development Tools"],"lifecycle":[{"event":"open","at":"2026-09-24T18:22:20Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 30","filings_12m":30,"filings_prev_12m":8,"green_card_filings_12m":0,"median_offered_wage_usd":192500,"route":null,"cap_exempt":false,"checked_at":"2026-10-03T21:08:04+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":28}],"liveness":{"score":87,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.866,"p_room":1,"age_days":9,"expected_fill_days":29,"reasons":["conf:0","velocity","win:early"],"computed_at":"2026-10-04T05:45:00Z"},"pay":{"stated_usd_annual":475000,"is_top_pay":true},"html_url":"https://alion.io/job/thinking-machines-lab-web-crawling-research-engineer","json_url":"https://alion.io/job/thinking-machines-lab-web-crawling-research-engineer.json","meta":{"generated_at":"2026-10-05T02:53:27Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","usage":{"tier":"crawler","counted_by":"address","units_charged":1,"used_today":3983,"day_limit":5000,"remaining_today":1017,"minute_limit":60,"resets_at":"2026-10-06T00:00:00Z"}}}