{"id":61512,"url":"https://alion.io/job/sunset-machine-learning-engineer","title":"Machine Learning Engineer","company":{"id":28307,"name":"Replay","domain":"replay.ai","url":"https://alion.io/company/replay","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"A","score":100,"open_postings":13,"ghost_share":0,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-10-17T05:46:00Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"middle","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["New York, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":184000,"max_usd":346000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":451},"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Data Augmentation","optional":false},{"name":"Embeddings","optional":false},{"name":"Fine-tuning","optional":false},{"name":"Human-in-the-Loop","optional":false},{"name":"Knowledge Distillation","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Model Distillation","optional":false},{"name":"Multimodal AI","optional":false},{"name":"NER","optional":false},{"name":"ONNX Runtime","optional":false},{"name":"Python","optional":false},{"name":"Quantization","optional":false},{"name":"Synthetic Data","optional":false},{"name":"TensorRT","optional":false}],"status":"live","first_seen_at":"2026-08-13T19:39:25Z","employer_posted_date":"2026-08-13","last_verified_at":"2026-10-17T05:54:41Z","board_verified":true,"closed_at":null,"days_open":64,"trust":{"level":"ok","repost_count":1,"flags":[],"days_open":64},"description":"About Replay\nAt its core, Replay was founded to help founders. We started by supporting startups through shutting down, but we have since expanded into unlocking a new revenue stream for all types of businesses.\nIn 2025, we had a unique insight: the data every company generates each day through collaboration, communication, and building is some of the most valuable training data in the world. Public and synthetic data can only get frontier models so far, so the next generation of model progress depends on real, proprietary data grounded in how actual businesses operate. We are a primary source of it, partnering directly with the frontier AI labs building what comes next.\nWhy Join Replay Now\nWe have scaled from $0 to a multi-eight-figure run rate in a matter of months\n\nWe have raised from top-tier investors, including Floodgate, Afore, Ludlow, and Hustle Fund\n\nWe are small enough that you will carry outsized responsibility and grow as quickly as the company does\n\nYou will partner with and build for some of the fastest and most important companies in the world\n\nYou will help build a massive, category-defining business from the ground floor\n\nThe Role\nReplay turns sensitive internal enterprise data into de-identified datasets without destroying the structure and meaning that make the data valuable. The data does not arrive in one clean modality. It spans messages, documents, tables, files, images, metadata, and provider-specific structures, with important context distributed across all of them.\nYou will improve how well our system understands and protects that data. Your initial scope will be a prioritized subset of named-entity recognition, entity and identity resolution, structured extraction, classification, semantic review, or other model-backed parts of the de-identification pipeline. We do not expect one person to be an expert in every modality. The goal is measurable improvement in the areas you own: better precision, recall, F1, high-risk coverage, and preserved data utility across the failure modes that matter.\nThis is an applied, production-facing ML role. You will study errors, form hypotheses, build datasets and experiments, improve or replace models, and ship the result into a live pipeline. Evaluation, reproducibility, observability, and safe releases matter because they let us identify, ship, and verify meaningful model improvements in production.\nWhat You'll Do\nOwn and improve NER, entity resolution, structured or tabular detection, document understanding, semantic review, or related de-identification systems\n\nTransform model failures and capability ceilings into a prioritized improvement roadmap\n\nDesign active-learning loops that combine model sweeps, LLM-assisted review, clustering, and uncertainty signals to identify the examples most worth hand-labeling\n\nBuild representative datasets and benchmarks, and use decision-relevant metrics to reveal strengths, weaknesses, uncertainty, and failure costs\n\nChoose and combine deterministic rules, classical ML, fine-tuning, embeddings, multimodal models, and LLM-based approaches based on the problem and evidence\n\nDesign experiments, tune thresholds, analyze precision-recall and utility tradeoffs, and explain which changes are real, uncertain, or limited to particular conditions\n\nProductionize improvements with reproducible artifacts, evaluation evidence, runtime instrumentation, and safe rollout\n\nOptimize inference cost, latency, and throughput without hiding regressions in quality or high-risk recall\n\nBuild high-fidelity evaluation environments with seeded failure modes and programmatic verifiers that expose subtle regressions\n\nBuild reliable model- or agent-based harnesses with bounded behavior and explicit output verification when the problem calls for them\n\nPartner with Applied Science on measurement and calibration, Data and Product Engineering on pipeline and review systems, and Security and Quality on acceptable risk\n\nUse AI engineering tools deeply to accelerate research, implementation, error analysis, and evaluation while verifying their output\n\nWhat Success Looks Like\nModel improvements generalize beyond the examples used to develop them and hold up in replay, shadow, and production evidence\n\nPriority modalities and entity classes show credible improvements in precision, recall, F1, or other decision-relevant quality measures\n\nHigh-risk misses decline without unacceptable over-redaction or loss of useful structure\n\nNew formats and modalities can be covered without relying on brittle one-off fixes\n\nImprovements reduce meaningful delivery risk, review or rework burden, or loss of data utility rather than moving only an isolated benchmark\n\nThe team can explain why a model changed, where it improved or regressed across consequential failure modes and data segments, and whether the change should ship\n\nThe path from error discovery to a trustworthy production improvement becomes faster and more repeatable\n\nQuality gains remain inside acceptable inference-cost, latency, and operational constraints\n\nYou Might Thrive Here If\nYou have 3+ years of professional machine learning or software engineering experience, including improving models in production\n\nYou have startup experience, enjoy broad ownership, and thrive when requirements are evolving or incomplete\n\nYou use modern AI tools fluently and verify their output\n\nYou have personally moved model quality through error analysis, data work, experimentation, implementation, deployment, and iteration\n\nYou have a strong grasp of precision, recall, F1, calibration, thresholding, class imbalance, imperfect labels, distribution shift, and representative evaluation\n\nYou are an applied engineer first: a strong Python and software engineer who can work inside data pipelines and production systems, not only notebooks\n\nYou have a bias toward action while maintaining scientific and engineering rigor\n\nYou are curious and stay current with relevant state-of-the-art methods\n\nYou choose techniques based on the shape of the problem and can combine deterministic, statistical, neural, and LLM-based approaches\n\nYou communicate uncertainty and tradeoffs clearly to scientists, engineers, and people making delivery or risk decisions\n\nThis Role May Not Be for You If\nYou want to focus on research novelty without owning measurable production improvement\n\nYou prefer optimizing one aggregate benchmark without investigating consequential failure modes, data segments, and failure costs\n\nYou want data preparation, evaluation, deployment, and production diagnosis to belong entirely to other teams\n\nYou reach for a larger model before understanding the errors, constraints, and simpler alternatives\n\nYou do not want AI tools to be part of your daily engineering and research workflow\n\nBonus\nExperience with NER, entity resolution, information extraction, document understanding, multimodal systems, or privacy-preserving ML\n\nExperience with hyperparameter tuning, data augmentation, model merging, ensembles, knowledge distillation, or multimodal model training\n\nExperience fine-tuning or adapting transformer, GLiNER, embedding, vision-language, or small specialized models\n\nExperience with active learning, uncertainty sampling, weak supervision, human-in-the-loop review, or LLM-assisted evaluation pipelines\n\nExperience building goldens, adversarial corpora, replay systems, model bakeoffs, agentic harnesses, or programmatic evaluation environments\n\nExperience with difficult ML or labeling problems\n\nExperience with ONNX Runtime, TensorRT, model pruning, quantization, or other CPU/GPU inference optimization\n\nExperience with sensitive enterprise data or other high-trust production systems\n\nExperience with synthetic data generation and managing the synth-to-real gap","description_format":"text","description_chars":7770,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-08-13T19:39:25Z"},{"event":"reopen","at":"2026-10-12T17:15:17Z"}],"visa":[],"liveness":{"score":10,"band":"cold","label":"Long shot","p_open":1,"p_active":0.369,"p_room":0.28,"age_days":64,"expected_fill_days":21,"reasons":["conf:3","win:tail","crowd:"],"computed_at":"2026-10-17T05:46:00Z"},"pay":null,"html_url":"https://alion.io/job/sunset-machine-learning-engineer","json_url":"https://alion.io/job/sunset-machine-learning-engineer.json","meta":{"generated_at":"2026-10-17T08:36:19Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":11427,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":28307},"rest":"https://alion.io/mcp/rest/get_company?id=28307"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsunset-machine-learning-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsunset-machine-learning-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsunset-machine-learning-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/sunset-machine-learning-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsunset-machine-learning-engineer"}]}