{"id":2056512,"url":"https://alion.io/job/prometheus-mts-pre-training-data-acquisition-engineer","title":"[Proto] MTS - Software Engineer, Data Acqusition & Crawling","company":{"id":79790,"name":"Prometheus","domain":"prometheus.ae","url":"https://alion.io/company/prometheus","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":null},"role":"Backend","role_family":"Backend","seniority":null,"employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["San Francisco, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":154000,"max_usd":320000,"period":"year","method":"role_country_seniority_unknown","sample_n":7868},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"C++","optional":false},{"name":"Kubernetes","optional":false},{"name":"Python","optional":false},{"name":"Ray","optional":false},{"name":"Rust","optional":false},{"name":"Spark","optional":false}],"status":"live","first_seen_at":"2026-10-07T22:27:22Z","employer_posted_date":"2026-10-07","last_verified_at":"2026-10-12T21:25:15Z","board_verified":true,"closed_at":null,"days_open":4,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":4},"description":"The Role\nThe best models are trained on data nobody else bothered to collect. This role owns how we acquire the internet: a crawler and ingestion system that pulls text, code, images, video, audio, 3D assets, documents, and stranger formats from across the web at petabyte scale, and a standing search for sources other labs overlook.\nYou will design and run the distributed systems that do the fetching, extraction, dedup, and delivery, and you will spend real time hunting for new corpora about the physical world and the economy: engineering and manufacturing data, CAD and mesh repositories, sensor and telemetry streams, scientific archives, industrial and logistics records, public filings, maps and geospatial, and whatever else turns out to teach a model how the world actually works. The job is half systems engineering at scale and half data research: every acquisition decision is measured by what it does to the model.\nWhat you'll do\nCrawl & ingest: build and operate a large-scale, polite, fault-tolerant web crawler and the ingestion pipelines behind it, from scheduling and fetch to extraction, normalization, versioning, and delivery to the pretraining data team.\n\nOdd modalities: build specialized acquisition paths for video, 3D (meshes, point clouds, CAD, scenes), audio, scanned documents, and structured or semi-structured datasets, including the format parsing and storage layout those need.\n\nNew sources: systematically find, assess, and acquire high-value corpora tied to the physical world and the economy, and work with legal on licensing, compliance, robots and rate limits, and privacy.\n\nData research: run experiments on crawling strategy, extraction quality, and coverage; analyze what we have for gaps and redundancy; close the loop with the pretraining and data teams on what moves evals.\n\nInfrastructure: own petabyte-scale storage, indexing and search over the corpus, and deployment on Kubernetes with infrastructure as code.\n\nWhat we're looking for\nDeep expertise in large, stateful, high-throughput distributed systems, with production experience in Rust or C++ [plus Python for glue and analysis].\n\nHands-on experience building or running a large web crawler or a comparable internet-scale acquisition system; you know the practical problems (politeness, dedup, trap detection, extraction, freshness) firsthand.\n\nComfort with multi-TB to PB datasets and with making systems observable, testable, and reproducible at that scale.\n\nGenuine curiosity about data: you enjoy finding unusual sources, reading their formats, and arguing from evidence about whether they will matter for a model.\n\nEnough familiarity with how LLMs are trained and evaluated to connect acquisition choices to model outcomes.\n\nBonus: experience with video, 3D, or scientific data formats and pipelines; background in web archiving, search infrastructure, or data engineering frameworks (Spark, Ray, Beam). [Domain bonus: simulation, CAD, geospatial, or industrial data.]","description_format":"text","description_chars":2979,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Wealth Management & Financial Advisors","Asset Management & Funds"],"lifecycle":[{"event":"open","at":"2026-10-08T00:03:39Z"}],"visa":[],"liveness":{"score":88,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.877,"p_room":1,"age_days":4,"expected_fill_days":20,"reasons":["conf:3","velocity","win:early"],"computed_at":"2026-10-12T06:55:09Z"},"pay":null,"html_url":"https://alion.io/job/prometheus-mts-pre-training-data-acquisition-engineer","json_url":"https://alion.io/job/prometheus-mts-pre-training-data-acquisition-engineer.json","meta":{"generated_at":"2026-10-12T22:03:55Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":36079,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-13T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":79790},"rest":"https://alion.io/mcp/rest/get_company?id=79790"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fprometheus-mts-pre-training-data-acquisition-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fprometheus-mts-pre-training-data-acquisition-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fprometheus-mts-pre-training-data-acquisition-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/prometheus-mts-pre-training-data-acquisition-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fprometheus-mts-pre-training-data-acquisition-engineer"}]}