{"id":1647976,"url":"https://alion.io/job/institute-of-foundation-models-eval360-error-analysis-engineer","title":"Eval360 - Error Analysis Engineer","company":{"id":99421,"name":"Institute of Foundation Models","domain":"ifm.ai","url":"https://alion.io/company/institute-of-foundation-models","size_band":"501-1000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Lever","truth_index":{"grade":"A","score":95,"open_postings":41,"ghost_share":0,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":482,"computed_at":"2026-10-28T05:59:10Z"}},"role":"Backend","role_family":"Backend","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Sunnyvale, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":150000,"max":450000,"currency":"USD","period":"year","gross":null,"usd_annual":450000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Angular","optional":false},{"name":"C#","optional":false},{"name":"CI/CD","optional":false},{"name":"Django","optional":false},{"name":"FastAPI","optional":false},{"name":"Flask","optional":false},{"name":"Git","optional":false},{"name":"GitHub","optional":false},{"name":"Interpretability","optional":false},{"name":"JavaScript","optional":false},{"name":"Machine Learning","optional":false},{"name":"MySQL","optional":false},{"name":"Node JS","optional":false},{"name":"PostgreSQL","optional":false},{"name":"Python","optional":false},{"name":"React.js","optional":false},{"name":"TypeScript","optional":false},{"name":"Multimodal AI","optional":true},{"name":"NumPy","optional":true}],"status":"live","first_seen_at":"2026-06-19T21:05:13Z","employer_posted_date":"2026-06-19","last_verified_at":"2026-10-28T04:29:20Z","board_verified":true,"closed_at":null,"days_open":130,"trust":{"level":"ok","repost_count":0,"flags":["company_stale"],"days_open":130},"description":"About the Institute of Foundation Models\nThe Institute of Foundation Models is a dedicated research lab focused on building, understanding, using, and risk-managing foundation models. Our mission is to advance AI research, support the next generation of AI builders, and develop impactful systems that improve how frontier models are trained, evaluated, deployed, and governed.\nAs part of our team, you will work closely with researchers, machine learning engineers, data scientists, software engineers, and product teams on some of the most important challenges in AI development. You will contribute to systems that help measure model quality, identify failure modes, and improve the reliability, safety, and readiness of model releases.\nThe Role\nWe are looking for an Eval360 - Error Analysis Engineer to help build, improve, and operate Eval360, an evaluation service that serves as a quality gate for AI models. This person will focus specifically on error analysis: understanding where models fail, why they fail, how those failures should be categorized, and how evaluation systems can better detect, measure, and prevent these issues before models are released.\nYou will collaborate with researchers, machine learning engineers, product managers, data scientists, and platform teams to develop AI evaluation applications and internal tools based on next-generation AI research. You will be part of a cross-functional team responsible for the full software development lifecycle, from requirements gathering and system design to implementation, deployment, monitoring, debugging, documentation, and continuous improvement.\nThe ideal candidate is comfortable working across the stack, including front-end interfaces for reviewing errors, back-end evaluation pipelines, data analysis workflows, model evaluation infrastructure, databases, dashboards, and APIs. This person should have strong software engineering skills, excellent analytical judgment, and the ability to turn ambiguous model failures into structured insights that improve evaluation quality.\nKey Responsibilities\nCollaborate with researchers, machine learning engineers, data scientists, product managers, and internal stakeholders to implement innovative software solutions for Eval360 and related model evaluation workflows.\nBuild and improve Eval360 as an evaluation service that acts as a quality gate for model development, model comparison, and model release decisions.\nPerform deep error analysis on model outputs, including identifying failure patterns, categorizing issues, tracing root causes, and proposing improvements to evaluation methodology.\nDevelop tools, workflows, and dashboards that make it easier for researchers and engineers to inspect model failures, compare model behavior, and understand quality regressions.\nDesign and implement client-side and server-side architecture for evaluation review systems, error analysis interfaces, reporting tools, and internal evaluation applications.\nDevelop responsive, usable interfaces that support error triage, annotation review, evaluation debugging, and model quality investigation.\nBuild and maintain back-end services, APIs, data pipelines, and integrations that support evaluation execution, results storage, analysis, and reporting.\nTest software to ensure responsiveness, correctness, reliability, and efficiency across evaluation workflows.\nTroubleshoot, debug, and upgrade evaluation systems, including identifying issues in data processing, evaluation metrics, model output handling, job orchestration, and user-facing analysis tools.\nCreate and maintain security, access control, and data protection settings for evaluation data, model outputs, annotations, and internal tooling.\nWrite clear technical documentation for Eval360 systems, error taxonomies, evaluation workflows, debugging procedures, and user-facing tools.\nWork with researchers, data scientists, analysts, and machine learning engineers to improve evaluation quality, model diagnostics, and failure-mode visibility.\nKeep track of new development tools, evaluation frameworks, model analysis methods, data quality techniques, and architectures relevant to AI evaluation systems.\nContribute to the design of error taxonomies, evaluation rubrics, quality thresholds, regression detection methods, and model readiness criteria.\nHelp ensure Eval360 produces reliable, interpretable, and actionable signals for model quality gates.\nContribute to research publications, technical reports, internal knowledge sharing, and external presentations where appropriate.\nContribute to intellectual property and thought leadership in AI evaluation, error analysis, model quality measurement, and evaluation infrastructure.\nPerform all other duties as reasonably directed by the line manager that are aligned with these functional objectives.\nAcademic Qualifications\nBachelor's degree in Computer Science, Machine Learning, Data Science, Software Engineering, Statistics, or a related technical field required.\nMaster's or Ph.D. in Computer Science, Machine Learning, Artificial Intelligence, Data Science, or a related field preferred.\nProfessional Experience\nProven experience as a Software Engineer, Full Stack Developer, Machine Learning Evaluation Engineer, Data Scientist, AI Engineer, or similar role.\nExperience building software systems for AI, machine learning, data analysis, evaluation, annotation, experimentation, or model monitoring.\nExperience working with AI algorithms and the ability to develop systems that accommodate AI-related requirements.\nExperience performing error analysis, model evaluation, data quality analysis, or failure-mode investigation for machine learning or language model systems.\nExperience developing internal applications, dashboards, review tools, or web-based workflows for technical users.\nFamiliarity with common software stacks, including front-end frameworks, back-end services, databases, APIs, and cloud or internal infrastructure.\nFamiliarity with GitHub, Git, CI/CD workflows, and collaborative software development practices.\nKnowledge of front-end languages and libraries such as HTML, CSS, JavaScript, TypeScript, React, Angular, or similar technologies.\nKnowledge of back-end languages and frameworks such as Python, Java, C#, Node.js, FastAPI, Flask, Django, or similar technologies.\nFamiliarity with databases such as MySQL, PostgreSQL, MongoDB, or other structured and unstructured data stores.\nFamiliarity with evaluation frameworks, experiment tracking systems, data pipelines, or machine learning infrastructure is strongly preferred.\nAbility to analyze complex model outputs and translate qualitative failures into structured, measurable categories.\nStrong problem-solving and troubleshooting skills, especially for ambiguous technical issues involving models, data, metrics, and software systems.\nEffective communication and collaboration skills, with the ability to work across research, engineering, data, and product teams.\nStrong attention to detail and a high bar for evaluation quality, reliability, and interpretability.\nPreferred Qualifications\nExperience with large language models, foundation models, multimodal models, or model evaluation systems.\nExperience designing or using error taxonomies, evaluation rubrics, benchmark datasets, human evaluation workflows, or automated grading systems.\nExperience with Python-based data analysis tools such as pandas, NumPy, Jupyter, or similar.\nExperience with visualization or dashboarding tools for model quality analysis.\nExperience with distributed systems, job queues, workflow orchestration, or large-scale data processing.\nExperience working in a research environment or with fast-moving AI product and model teams.","description_format":"text","description_chars":7737,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Foundation Models","AI Research Labs"],"lifecycle":[{"event":"open","at":"2026-10-02T00:08:08Z"}],"visa":[],"liveness":{"score":72,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":0.8,"age_days":130,"expected_fill_days":482,"reasons":["conf:1","velocity","win:early","crowd:"],"computed_at":"2026-10-28T05:59:10Z"},"pay":{"stated_usd_annual":450000,"is_top_pay":true},"html_url":"https://alion.io/job/institute-of-foundation-models-eval360-error-analysis-engineer","json_url":"https://alion.io/job/institute-of-foundation-models-eval360-error-analysis-engineer.json","meta":{"generated_at":"2026-10-28T06:11:12Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":8060,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-29T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":99421},"rest":"https://alion.io/mcp/rest/get_company?id=99421"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Finstitute-of-foundation-models-eval360-error-analysis-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Finstitute-of-foundation-models-eval360-error-analysis-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Finstitute-of-foundation-models-eval360-error-analysis-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/institute-of-foundation-models-eval360-error-analysis-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Finstitute-of-foundation-models-eval360-error-analysis-engineer"}]}