{"id":856067,"url":"https://alion.io/job/openteams-senior-aiml-test-and-evaluation-engineer","title":"Senior AI/ML Test and Evaluation Engineer","company":{"id":678978,"name":"OpenTeams","domain":"openteams.com","url":"https://alion.io/company/openteams","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":null,"work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Austin, United States","United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":145000,"max":250000,"currency":"USD","period":"year","gross":null,"usd_annual":250000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"Computer Vision","optional":false},{"name":"Hugging Face","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"Machine Learning","optional":false},{"name":"Python","optional":false},{"name":"PyTorch","optional":false},{"name":"NumPy","optional":true},{"name":"SciPy","optional":true},{"name":"Travis CI","optional":true}],"status":"live","first_seen_at":"2026-08-25T18:10:32Z","employer_posted_date":"2026-10-07","last_verified_at":"2026-10-21T15:53:51Z","board_verified":true,"closed_at":null,"days_open":56,"trust":{"level":"ok","repost_count":1,"flags":[],"days_open":56},"description":"Who We Are\nEvery organization runs on intelligence: years of accumulated knowledge, decisions, and context. As AI takes on more of that work, companies face a choice: rent that intelligence from vendors who keep the data, the context, and the results, or own it.\nOpenTeams exists to make ownership possible.\nFounded by Travis Oliphant, creator of NumPy and SciPy, and built by people with deep roots across the open-source ecosystem, including NumPy, SciPy, PyTorch, and Jupyter, we help enterprises and governments build AI they control, govern, and evolve themselves.\nIf that sounds like your kind of work, we'd like to meet you.\nSenior AI/ML Test and Evaluation Engineer\nLocation: U.S - Remote OR Hybrid - Washington, DC, Denver, CO or Colorado Springs, CO. \nWork Authorization: U.S. citizenship required\nClearance: U.S.-Remote Opening: An active clearance is not required. Candidates must be eligible and willing to obtain and maintain a U.S. security clearance. Hybrid Opening: An active TS/SCI clearance is preferred. Candidates may also be considered if they previously held a TS/SCI with CI polygraph or currently hold an active TS or Secret clearance.\nSalary Range: $145,000-$250,000 USD, dependent on experience level and location\nOpenings: Two positions are available:\nOne hybrid position: Candidates must be located in or willing to work hybrid from Washington, DC; Denver, CO; or Colorado Springs, CO. An active TS/SCI clearance is required. Up to 15% travel is required.\nOne U.S.-remote position: Candidates may work remotely from anywhere in the United States. An active clearance is not required, but candidates must be willing and able to undergo the process required to obtain and maintain a U.S. security clearance.\nAbout the Role\nWe're looking for a Senior AI/ML Test and Evaluation Engineer to build and operate the benchmarking and evaluation capability at the core of an AI platform. This is a role for someone who is more interested in what a model gets wrong than in what it gets right.\nYou build the evaluation harnesses - automated metrics paired with structured human expert judgment, applied to candidate models and to the agentic workflows built on top of them. You develop repeatable methodologies for comparing performance against current operational baselines, which means the comparison holds up when someone runs it again in six months with a different model. And you document the limitations and surface the failure modes that matter, including the ones nobody asked about.\nYour reports go to senior stakeholders and inform decisions about which capabilities are ready to field. That's the weight of the job: a benchmark that looks good and hides a failure mode is worse than no benchmark at all, and you're the check against that.\nThis is hands-on engineering on an open-source toolchain.\nKey Responsibilities\nDesign and implement a platform for test & evaluation of AI models and Agentic systems\nDevelop evaluation methodologies combining human and AI expert judging and multiple input and output modalities\nDesign and enhance user interfaces such that platform customers can express workflows in an intuitive and generic manner\nCollaborate closely with subject matter experts, Model/Agent developers, and evaluation designers to ensure coverage over existing and to be discovered requirements.\nArchitect and implement robust experiment provenance and result tracking systems.\nEnsure performance across integrated tools and scaling systems.\nProvide technical mentorship, code review, design guidance, and clear documentation to support team knowledge transfer and user onboarding.\n\nRequired Skills & Experience\nProven experience building and deploying software platforms with complex integration surface, preferably handling Machine Learning or agentic workloads.\nDeep experience using python, including API frameworks and agentic harnesses.\nHigh level knowledge of Computer vision, language modeling, and/or task specific agent evaluation.\nHands on experience with common frameworks and tooling, eg PyTorch and Hugging Face ecosystem.\nStrong communication skills and experience collaborating effectively with cross functional teams including external stakeholders.\nDemonstrated experience or potential for leadership, especially in making architectural decisions, coordinating the efforts of other engineers, and developing standards and best practices.\nNice to Have\nCurrently hold or eligible to hold U.S. Security Clearance (Secret or higher)\nPrior experience designing test and evaluation systems for AI\nExperience deploying complex software systems in IL 4 or higher environments.\nExperience participating in security assessment and authorization, hardening systems, and remediating vulnerabilities. \nPrevious experience contributing to open source projects and participating in open source communities.\nExposure to the intelligence community or department of defence subject matter experts. \nGrow With Us\nAt OpenTeams, growth isn’t just about the company-it’s about you.\nWe believe the best careers are built at the edge of your potential. That is where new tools, ideas, and technologies change the world. Here, you’ll work alongside pioneers of AI, solving problems that matter: making AI more transparent, more ethical, and more empowering. As your skills grow, our career framework provides a pathway and recognition of that increased impact.\nOpportunities aren’t limited by geography. You’ll collaborate with global experts, contribute to open source projects that power the world’s technology, and stretch your skills daily. That global perspective and diversity makes our solution more universal and robust. We are committed to continuing to celebrate diversity on our team.\nSupported people are successful people. We offer 100% employer paid medical premiums for employees and self-managed PTO with a minimum time off requirement, so that our teams are able to do their best work.\nWe invest in curiosity, creativity, and ownership. That means you’ll be trusted to boldly innovate, supported to learn fast, and celebrated for successful collaboration.\nCommitment to diversity, equity, inclusion, and belonging\nOpenTeams understands that valuing diverse creative practices and forms of knowledge is crucial to and enriches the company’s core mission. We encourage applications from everyone, including members of all equity-seeking communities, such as (but certainly not limited to) women, racialized and Indigenous persons, disabled people, persons of all sexual orientations, gender identities and expressions.\nWe are an equal opportunity employer - all qualified applicants will receive equal consideration for recruitment, interviews, employment, training, compensation, promotion, and related activities. We do not discriminate based on race, religion, gender, gender identity, gender expression, color, national origin, pregnancy, ancestry, domestic partner status, disability, sexual orientation, age, genetic predisposition, medical condition, marital status, citizenship status, military or veteran status, or any other basis covered by applicable laws. OpenTeams will not tolerate discrimination or harassment based on these characteristics or any other unlawful behavior, conduct, or purpose.","description_format":"text","description_chars":7254,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"phd","optional":false},"security_clearance":true,"languages":[]},"benefits":["Equity"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"},{"name":"Washington","iso":null,"kind":"city"},{"name":"Colorado","iso":null,"kind":"city"}],"hiring_excludes":[],"relocation_offered":false,"industries":["AI Consulting & Integration","Open Source Projects & Foundations"],"lifecycle":[{"event":"open","at":"2026-09-13T06:44:15Z"},{"event":"close","at":"2026-09-24T07:25:29Z"},{"event":"reopen","at":"2026-10-07T16:34:36Z"}],"visa":[],"liveness":{"score":25,"band":"fade","label":"Fading","p_open":1,"p_active":0.559,"p_room":0.45,"age_days":56,"expected_fill_days":29,"reasons":["conf:0","win:tail"],"computed_at":"2026-10-21T05:46:15Z"},"pay":{"stated_usd_annual":250000,"is_top_pay":true},"html_url":"https://alion.io/job/openteams-senior-aiml-test-and-evaluation-engineer","json_url":"https://alion.io/job/openteams-senior-aiml-test-and-evaluation-engineer.json","meta":{"generated_at":"2026-10-21T16:22:06Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":23233,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-22T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":678978},"rest":"https://alion.io/mcp/rest/get_company?id=678978"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fopenteams-senior-aiml-test-and-evaluation-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fopenteams-senior-aiml-test-and-evaluation-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fopenteams-senior-aiml-test-and-evaluation-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/openteams-senior-aiml-test-and-evaluation-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fopenteams-senior-aiml-test-and-evaluation-engineer"}]}