{"id":1650847,"url":"https://alion.io/job/steampunk-ai-evaluation-scientist","title":"AI Evaluation Scientist","company":{"id":3854754,"name":"Steampunk","domain":"steampunk.com","url":"https://alion.io/company/steampunk-5","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"iCIMS","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"junior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["McLean, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":105000,"max":145000,"currency":"USD","period":"year","gross":null,"usd_annual":145000},"salary_estimate":null,"experience_years_min":2,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"DeepEval","optional":false},{"name":"Embeddings","optional":false},{"name":"Hallucination","optional":false},{"name":"Hugging Face","optional":false},{"name":"Human-in-the-Loop","optional":false},{"name":"LangChain","optional":false},{"name":"LLM","optional":false},{"name":"LLMOps","optional":false},{"name":"Machine Learning","optional":false},{"name":"NLP","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Python","optional":false},{"name":"PyTorch","optional":false},{"name":"RAG","optional":false},{"name":"Ragas","optional":false},{"name":"Scikit-learn","optional":false},{"name":"AWS","optional":true},{"name":"Azure","optional":true},{"name":"NIST AI RMF","optional":true},{"name":"OWASP","optional":true}],"status":"live","first_seen_at":"2026-10-02T00:14:38Z","employer_posted_date":"2026-10-02","last_verified_at":"2026-10-16T07:32:29Z","board_verified":true,"closed_at":null,"days_open":14,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":14},"description":"Overview\nWe are looking for an AI Evaluation Scientist to design and execute evaluation processes that ensure our predictive and generative AI systems are accurate, reliable, safe, and aligned with mission requirements. This role is essential for establishing trust in AI solutions and supporting continuous improvement across the AI lifecycle. The AI Evaluation Scientist will work closely with engineers, data scientists, governance analysts, and product teams to develop evaluation metrics, build test harnesses, analyze model behavior, and support responsible deployment.\nContributions\nImplement evaluation frameworks for AI models, including accuracy, robustness, relevance, bias, hallucination rate, and safety metrics.\nBuild and maintain automated evaluation scripts, tests, and pipelines that assess AI model outputs and detect performance drift over time. \nDevelop benchmark datasets, challenge sets, and scenario-based test cases tailored to mission and user needs. \nPerform structured error analysis and behavioral audits of LLMs, retrieval-augmented generation (RAG) systems, and predictive models, documenting findings and improvement recommendations. \nCollaborate with AI Developers, LLMOps Engineers, and Data Scientists to support iterative experimentation, model hardening, and quality improvements. \nContribute to the design of human-in-the-loop evaluation workflows, integrating qualitative and quantitative insight into evaluation reports. \nAssist in mapping evaluation outcomes to responsible AI principles such as fairness, transparency, reliability, and safety. \nPartner with AI Governance Analysts to ensure evaluation outputs support compliance, documentation, and risk assessments. \nStay current with emerging evaluation tools, frameworks, metrics, and research related to LLM assessment and generative AI reliability. \nDocument evaluation processes, criteria, and results for both technical and non-technical audiences. \nYou will contribute to the growth of our AI & Data Exploitation Practice! \nQualifications\nAbility to hold a position of public trust with the U.S. government. \nBachelor’s or Master’s degree in Computer Science, Statistics, Machine Learning, Cognitive Science, Human-Computer Interaction, Data Science, or a related field. \n2+ years of experience evaluating machine learning models, NLP systems, or generative AI models (LLMs preferred). \nFamiliarity with evaluation metrics, statistical testing, dataset creation, and experimental design for AI systems. \nProficiency in Python and relevant libraries such as PyTorch, Hugging Face, scikit-learn, LangChain. \nProficiency in AI evaluation frameworks such as Ragas or DeepEval. \nProficiency in AI traceability/observability tools based on the OpenTelemetry protocol. \nExperience analyzing structured and unstructured data, including text, documents, and embeddings. \nUnderstanding of LLM behavior, prompt evaluation, retrieval pipelines, or RAG architectures. \nExposure to responsible AI concepts and governance-aligned evaluation criteria (e.g., fairness, transparency, reliability). \nStrong analytical skills with the ability to interpret model weaknesses, extract insights, and recommend actionable improvements. \nExcellent written and verbal communication skills, with the ability to present evaluation findings clearly to technical and non-technical stakeholders. \nExperience working in agile or iterative development environments is a plus. \nFamiliarity with OWASP LLM Top 10 Risks.\nRelevant certifications (helpful but not required): NIST AI RMF (AISIC), INFORMS CAP, AWS/Azure/Google ML Certifications.\nAbout steampunk\nSteampunk relies on several factors to determine salary, including but not limited to geographic location, contractual requirements, education, knowledge, skills, competencies, and experience. The projected compensation range for this position is $105,000 to $145,000. The estimate displayed represents a typical annual salary range for this position. Annual salary is just one aspect of Steampunk’s total compensation package for employees. Learn more about additional Steampunk benefits here.\nIdentity Statement\nAs part of the application process, you are expected to be on camera during interviews and assessments. We reserve the right to take your picture to verify your identity and prevent fraud.\nSteampunk is a Change Agent in the Federal contracting industry, bringing new thinking to clients in the Homeland, Federal Civilian, Health and DoD sectors. Through our Human-Centered delivery methodology, we are fundamentally changing the expectations our Federal clients have for true shared accountability in solving their toughest mission challenges. If you want to learn more about our story, visit http://www.steampunk.com.\nWe are an equal opportunity employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability status, protected veteran status, or any other characteristic protected by law. Steampunk participates in the E-Verify program.","description_format":"text","description_chars":5063,"description_truncated":false,"requirements":{"experience_years_min":2,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-10-02T00:14:38Z"}],"visa":[],"liveness":{"score":12,"band":"cold","label":"Long shot","p_open":1,"p_active":0.329,"p_room":0.35,"age_days":14,"expected_fill_days":5,"reasons":["conf:2","velocity","win:tail","comp:junior,brand"],"computed_at":"2026-10-16T05:45:00Z"},"pay":{"stated_usd_annual":145000,"is_top_pay":false},"html_url":"https://alion.io/job/steampunk-ai-evaluation-scientist","json_url":"https://alion.io/job/steampunk-ai-evaluation-scientist.json","meta":{"generated_at":"2026-10-16T11:29:43Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":10340,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-17T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3854754},"rest":"https://alion.io/mcp/rest/get_company?id=3854754"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsteampunk-ai-evaluation-scientist"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsteampunk-ai-evaluation-scientist"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsteampunk-ai-evaluation-scientist"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/steampunk-ai-evaluation-scientist\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsteampunk-ai-evaluation-scientist"}]}