{"id":61514,"url":"https://alion.io/job/sunset-data-scientist","title":"Data Scientist","company":{"id":28307,"name":"Replay","domain":"replay.ai","url":"https://alion.io/company/replay","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"A","score":100,"open_postings":13,"ghost_share":0,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-10-17T05:46:00Z"}},"role":"Data Science","role_family":"Data Science","seniority":"middle","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["New York, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":139000,"max_usd":262000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":454},"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Multimodal AI","optional":false},{"name":"NER","optional":false},{"name":"Python","optional":false},{"name":"SQL","optional":false},{"name":"Synthetic Data","optional":false}],"status":"live","first_seen_at":"2026-08-13T19:39:13Z","employer_posted_date":"2026-08-13","last_verified_at":"2026-10-17T05:54:41Z","board_verified":true,"closed_at":null,"days_open":64,"trust":{"level":"ok","repost_count":1,"flags":[],"days_open":64},"description":"About Replay\nAt its core, Replay was founded to help founders. We started by supporting startups through shutting down, but we have since expanded into unlocking a new revenue stream for all types of businesses.\nIn 2025, we had a unique insight: the data every company generates each day through collaboration, communication, and building is some of the most valuable training data in the world. Public and synthetic data can only get frontier models so far, so the next generation of model progress depends on real, proprietary data grounded in how actual businesses operate. We are a primary source of it, partnering directly with the frontier AI labs building what comes next.\nWhy Join Replay Now\nWe have scaled from $0 to a multi-eight-figure run rate in a matter of months\n\nWe have raised from top-tier investors, including Floodgate, Afore, Ludlow, and Hustle Fund\n\nWe are small enough that you will carry outsized responsibility and grow as quickly as the company does\n\nYou will partner with and build for some of the fastest and most important companies in the world\n\nYou will help build a massive, category-defining business from the ground floor\n\nThe Role\nReplay turns sensitive internal enterprise data into de-identified datasets without destroying the structure and meaning that make the data valuable. That creates a difficult measurement problem. A system can improve aggregate F1 while missing a high-risk slice, remove more sensitive information while also destroying useful context, or pass one stage while defects escape somewhere else in the pipeline.\nAs Replay's first Data Scientist focused on evaluation, you will establish how we know whether that data is actually getting better. You will build the datasets, experiments, quality measures, and feedback loops that expose hidden failures, accelerate model and pipeline improvement, and give the team confidence in what it delivers.\nThis is a hands-on, zero-to-one role at the intersection of data science, AI, and a real production system. You will write Python and SQL, construct evaluation corpora, study failure patterns, design comparisons, calibrate human and model-based judgments, and turn the result into a clear decision. The questions are scientifically difficult, but the output must be practical enough to change what the team builds and ships.\nYou will work closely with Machine Learning, Product Engineering, Data Engineering, Security, Quality, domain experts, and the team making delivery decisions. Machine Learning Engineers own changing model behavior. You own the credibility of the evidence used to decide whether a model, pipeline, or delivery change actually made the data safer or more useful.\nQuestions You Might Answer\nDid a higher NER or entity-resolution score actually reduce sensitive misses across the messages, documents, tables, and providers that matter?\n\nIs a new model finding more sensitive information, or simply removing more of the useful structure our customers need?\n\nCan we trust a golden dataset, a human review process, or an LLM judge enough to use it for a release decision?\n\nWhich customer, modality, entity, language, or format slices are hidden by a strong aggregate result?\n\nWhere did a quality loss enter between source data, processing, de-identification, review, and delivery?\n\nWhat is the smallest credible experiment that would tell us whether to ship, revise, or stop a change?\n\nWhat You'll Do\nDefine what high-quality and safe-to-deliver data mean across de-identification, structure preservation, semantic coherence, and customer utility\n\nDesign representative samples and build golden, adversarial, replay, and production-like corpora with explicit provenance, labeling policy, agreement, adjudication, and versioning\n\nTurn ambiguous concepts such as “useful,” “clean,” or “safe” into measurable claims with known uncertainty and clear decision consequences\n\nEvaluate detectors, models, prompts, judges, thresholds, review workflows, and pipeline changes using comparisons that can support a real decision\n\nBreak aggregate results into the modalities, providers, entity classes, customer contexts, languages, formats, and risk tiers that reveal consequential failures\n\nConnect local measures to escaped sensitive information, avoidable over-redaction, preserved data utility, review burden, rework, and delivery acceptance\n\nBuild reproducible analysis, evaluation pipelines, and high-fidelity environments using Python, SQL, synthetic data, historical replay, seeded failures, and programmatic verifiers\n\nEstablish holdout and evaluation practices that keep the evidence trustworthy while model and product teams iterate quickly\n\nUse modern AI tools deeply for analysis, corpus development, coding, review, and hypothesis generation while independently verifying their output\n\nWhat Success Looks Like\nThe team has a decision-grade baseline for a priority Clean Data quality claim and trusts it enough to use in model, pipeline, release, and delivery decisions\n\nImprovements are judged by the slices and failure costs that matter, not only by an aggregate benchmark\n\nThe company can distinguish a true gain from label noise, sample bias, leakage, evaluator error, or a shifted workload\n\nChanges that improve one stage cannot hide escaped defects, over-redaction, utility loss, or review burden somewhere else\n\nAt least one consequential decision changes because the evidence reveals a risk, tradeoff, or opportunity that was previously unclear\n\nEvaluation becomes faster and more repeatable without sacrificing independence or rigor\n\nQuality claims communicate uncertainty honestly and remain understandable to engineers, customers, and risk owners\n\nYou Might Thrive Here If\nYou have at least three years of professional experience in applied science, data science, machine learning, quantitative research, or a closely related role\n\nYou have designed evaluations or experiments that changed a product, model, release, or operational decision\n\nYou understand sampling, uncertainty, precision, recall, F1, calibration, agreement, class imbalance, distribution shift, and imperfect labels\n\nYou can investigate messy, multi-stage data systems and determine where an apparent gain or loss actually came from\n\nYou are comfortable writing Python and SQL and building reproducible technical artifacts rather than handing requirements to an engineering team\n\nYou can protect the independence of an evaluation while collaborating closely with the people whose work it evaluates\n\nYou have startup experience and enjoy broad ownership, changing context, and building the measurement foundation while decisions are already moving quickly\n\nYou use AI tools fluently but do not confuse an articulate model output with valid evidence\n\nYou communicate uncertainty and difficult findings directly, without hiding behind false precision\n\nThis Role May Not Be for You If\nYou want to optimize models as your primary job rather than determine whether changes actually improve delivered data\n\nYou prefer descriptive dashboards that stop short of changing a decision\n\nYou treat labels, benchmarks, or model-based judges as ground truth without investigating how they fail\n\nYou need a perfectly defined dataset and research plan before you can make progress\n\nYou are uncomfortable disagreeing with a technically strong team when the evidence does not support its conclusion\n\nYou do not want AI tools to be part of your daily scientific and technical workflow\n\nBonus\nExperience evaluating NER, entity resolution, information extraction, document understanding, multimodal, retrieval, or LLM systems\n\nExperience with privacy, de-identification, data quality, model risk, safety, or other high-trust decision systems\n\nExperience designing human-review, adjudication, weak-supervision, or active-learning systems\n\nExperience building adversarial corpora, replay systems, simulation environments, programmatic verifiers, or model-judge evaluations\n\nExperience connecting offline measures to escaped defects, customer outcomes, review effort, or preserved data utility\n\nExperience measuring quality across multi-stage batch or data pipelines","description_format":"text","description_chars":8144,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-08-13T19:39:13Z"},{"event":"reopen","at":"2026-10-12T17:15:17Z"}],"visa":[],"liveness":{"score":10,"band":"cold","label":"Long shot","p_open":1,"p_active":0.369,"p_room":0.28,"age_days":64,"expected_fill_days":21,"reasons":["conf:3","win:tail","crowd:"],"computed_at":"2026-10-17T05:46:00Z"},"pay":null,"html_url":"https://alion.io/job/sunset-data-scientist","json_url":"https://alion.io/job/sunset-data-scientist.json","meta":{"generated_at":"2026-10-17T07:26:10Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":10121,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":28307},"rest":"https://alion.io/mcp/rest/get_company?id=28307"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsunset-data-scientist"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsunset-data-scientist"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsunset-data-scientist"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/sunset-data-scientist\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsunset-data-scientist"}]}