LLM Evaluation Jobs - Remote & On-site

LLM Evaluation jobs hiring now at vetted startups and product companies on Alion, updated daily. Average salary around $230,281/year. Compare remote, hybrid and on-site positions with relocation and visa sponsorship. Browse and apply today.

AI/ML
Data Science
Backend
Frontend
Mobile
DevOps
Web3
Games
Hardware
Robotics
Security
QA
Executive
Networking
Product
Design
Analytics
Support
Enterprise Apps
Quantum

Intent Lab

intentlab.ai
Turn your intent into production systems.
Open 94 days 4 months ago

Agent Engineer / Forward Deployed Engineer

$111k – $222k per year (Estimated) • Middle • 3+ years expIn office • Relocation • Full-Time • Middle
Go
Python
TypeScript
AI/ML
Prompt Engineering
Function Calling
LLM Evaluation
DevOps
AWS
Azure
CI/CD
GCP
Apply
Report

Abby Care

abbycare.org
Abby Care is a home health care company focused on children with complex medical needs and disabilities. The company trains and employs family members as paid caregivers under Medicaid programmes, supporting them with nurse oversight, care planning, and its own clinical software. It handles enrollment, credentialing, and claims for those families, positioning parent caregivers as a fix for the shortage of home nursing staff.
abbycare.org • San Francisco • Worcester • Philadelphia • Boston • Newark • Medical AI • Artificial Intelligence • Nursing Care • Digital Health • Health Care • Home Healthcare
San Francisco • Worcester • Philadelphia • Boston • Newark • Medical AI • Artificial Intelligence • Nursing Care • Digital Health • Health Care • Home Healthcare
Open 96 daysTop 25% pay 4 months ago

Senior Applied AI Engineer

$190k – $250k per year • Senior • 5+ years expIn office (San Francisco, United States) • Relocation • Full-Time • Senior
Fine-tuning
Multimodal AI
Reinforcement Learning
Synthetic Data
Knowledge Graph
LLM Evaluation
Post-training
PPO
Apply
Report

Socure

socure.com
Socure is a leading digital identity verification and fraud prevention platform that uses artificial intelligence and machine learning to verify consumer identities in real time. The company analyzes thousands of predictive data points - including biometrics, device intelligence, and email or phone attributes - to accurately authenticate individuals during online onboarding. By providing high accuracy rates across demographic groups, it enables financial institutions, digital healthcare providers, and online platforms to reduce fraud while streamlining user conversion.
socure.com • HQ: New York, United States • Artificial Intelligence • Fraud Detection • Identity Management • 501-1000 employees • Est. 2012
HQ: New York, United States • Artificial Intelligence • Fraud Detection • Identity Management • 501-1000 employees • Est. 2012
Top 25% payVerified live · 5 hours ago 4 months ago

Senior AI Engineer

$200k – $230k per year • Senior • 8+ years expRemote/Hybrid (New York, United States) • Full-Time • Senior
Python
AI/ML
AutoGen
AWS Bedrock
CrewAI
Function Calling
LangGraph
LangSmith
LLM
Model Context Protocol
RAG
Weights & Biases
LangChain
Amazon SageMaker
AWS Bedrock AgentCore
AWS Strands Agents
Human-in-the-Loop
Knowledge Graph
LLM Evaluation
LLM Guardrails
AI Agents
Arize Phoenix
DevOps
AWS
AWS Lambda
Vector
Apply
Top 25% payVerified live · 5 hours ago 4 months ago

AI Engineer

$220k – $290k per year • Senior • 8+ years expRemote/Hybrid • Full-Time • Staff
Python
AI/ML
AutoGen
AWS Bedrock
CrewAI
Function Calling
LangGraph
LangSmith
LLM
Model Context Protocol
RAG
Weights & Biases
LangChain
Amazon SageMaker
AWS Bedrock AgentCore
AWS Strands Agents
Human-in-the-Loop
Knowledge Graph
LLM Evaluation
LLM Guardrails
AI Agents
Arize Phoenix
DevOps
AWS
AWS Lambda
Vector
Apply
Report

Harper

harpercorp.com
harpercorp.com • San Francisco • Manufacturing
San Francisco • Manufacturing
Open 97 daysTop 25% pay 4 months ago

Product Manager

$125k – $170k per year • Junior • 1+ year expIn office (San Francisco, United States) • Relocation • Full-Time • Junior
AI Agents
Claude
Claude Code
Cursor
Context Engineering
LLM Evaluation
Lovable
Apply
Report

Machinify

machinify.com
Machinify is an artificial intelligence software company headquartered in Palo Alto, California, and founded in 2016. The company provides an AI-powered operating system designed to automate healthcare payment integrity processes, including claims auditing, subrogation, and pharmacy payment accuracy. It serves major health plans across the United States, managing data for over 270 million members to identify cost savings and reduce administrative friction for payers and providers.
machinify.com • HQ: Palo Alto, United States • Business Process Automation (BPA) • Software • Insurance • Financial Services • Machine Learning • Health Care • Digital Health • Artificial Intelligence • Est. 2016
HQ: Palo Alto, United States • Business Process Automation (BPA) • Software • Insurance • Financial Services • Machine Learning • Health Care • Digital Health • Artificial Intelligence • Est. 2016
Verified live · 16 min ago 4 months ago

Sr. Director | Engineering, ML

$124k – $257k per year (Estimated) • Executive • 8+ years exp • Remote (United States) • Bachelor's Degree • Senior
Java
Python
Rust
Scala
AI/ML
LLM
LLM Evaluation
Semantic Search
Apply
Top 25% payVerified live · 16 min ago 7 months ago

Staff Data Scientist | NLP

$180k – $230k per year • Staff+ • 3+ years exp • Remote (United States) • Staff
Python
SQL
Databases
pgvector
Pinecone
Weaviate
PostgreSQL
AI/ML
Fine-tuning
JAX
LLM
LoRA
NLP
PyTorch
QLoRA
RAG
Semantic Search
Sentence-Transformers
spaCy
Transformers
PEFT
AI Agents
Hugging Face
Human-in-the-Loop
LLM Evaluation
Semantic Search
Multimodal AI
DevOps
AWS
Azure
GCP
Vector
Analytics
A/B Testing
Apply
Top 25% payVerified live · 16 min ago 7 months ago

AI Engineer | Agentic Systems

$130k – $200k per year • Junior • 2+ years exp • Remote (United States) • Bachelor's Degree • Junior
Python
Python
Pydantic
AI/ML
Anthropic SDK
Claude
Claude Code
Gemini
LangGraph
LLM
LangChain
Anthropic
LLM Evaluation
OpenAI
OpenAI Agents SDK
OpenAI Codex
Structured Outputs
AI Agents
Multimodal AI
OCR
Apply
Report

Altaml

altaml.ai
altaml.ai • Edmonton • Dallas • Calgary • Toronto
Edmonton • Dallas • Calgary • Toronto
Open 103 daysVerified live · 1 day ago 4 months ago

Intermediate Full Stack Software Engineer

$65k – $79k per year • Middle • 3+ years expRemote/Hybrid (Calgary, Canada) • Full-Time • Middle
C#
JavaScript
Python
TypeScript
AI/ML
AI Agents
Claude
Claude Code
Copilot
Cursor
LLM
RAG
Function Calling
LLM Evaluation
LLM Guardrails
OpenAI
AutoGen
LangChain
LangGraph
Frontend
Angular
React.js
Vue.js
DevOps
Azure
Docker
Git
GitHub
Azure DevOps
CI/CD
GitHub Actions
QA
Postman
Pytest
Apply
Open 131 dayVerified live · 1 day ago 5 months ago

Senior Full Stack Software Engineer

$94k – $108k per year • Senior • 5+ years expRemote/Hybrid (Toronto, Canada) • Full-Time • Senior
AI Agents
Claude
Claude Code
LLM
Function Calling
LLM Evaluation
Model Context Protocol
OpenAI Agents SDK
DevOps
Azure
Docker
QA
Postman
Apply
Report

Coupa Software

coupa.com
Coupa Software is a global enterprise software leader in AI-driven Business Spend Management (BSM) and supply chain design, headquartered in Foster City, California. Founded in 2006 by Dave Stephens and Noah Eisner (and taken private by Thoma Bravo in an $8B transaction in 2023), the platform oversees trillions of dollars in transactional spend across a network of over 3,000 corporate clients globally.
coupa.com • HQ: Foster City, United States • Transportation & Logistics • Supply Chain • Software • Business Process Automation (BPA) • 201-500 employees • Est. 2006
HQ: Foster City, United States • Transportation & Logistics • Supply Chain • Software • Business Process Automation (BPA) • 201-500 employees • Est. 2006
Verified live · 1 day ago 4 months ago

Forward Deployed Engineer (FDE) - 11385

$141k – $197k per year • Remote (United Kingdom)
JavaScript
Node JS
Python
AI/ML
AI Agents
AutoGen
Claude
LangChain
LLM
Model Context Protocol
Human-in-the-Loop
LLM Evaluation
LLM Guardrails
OpenAI
Frontend
GraphQL
Apply
Verified live · 1 day ago 5 months ago

Forward Deployed Engineer (FDE) - 11396

$81k – $182k per year (Estimated) • Remote (United Kingdom)
JavaScript
Node JS
Python
AI/ML
AI Agents
AutoGen
Claude
LangChain
LLM
Model Context Protocol
Human-in-the-Loop
LLM Evaluation
LLM Guardrails
OpenAI
Frontend
GraphQL
Apply
Report

Labelbox

labelbox.com
Labelbox is a San Francisco company founded in 2018 that provides a data engine for training and evaluating machine learning models. Its platform combines annotation tooling, model-assisted labelling and quality analytics, and it operates an expert workforce marketplace for tasks that require domain knowledge. Customers range from computer vision teams to frontier labs sourcing human preference data.
labelbox.com • HQ: San Francisco, United States • Information Security • Cybersecurity • Artificial Intelligence • Est. 2018
HQ: San Francisco, United States • Information Security • Cybersecurity • Artificial Intelligence • Est. 2018
Open 140 daysVerified live · 10 hours ago 5 months ago

Forward Deployed Research Scientist

$155k – $339k per year (Estimated) • Equity • In office (San Francisco, United States) • Master's Degree
Databricks
AI/ML
Fine-tuning
Llama
LLM
Mistral
NLP
Qwen
Reinforcement Learning
RLHF
DPO
LLM Evaluation
SFT
Apply
Report

Xsolla

xsolla.com
Xsolla is a global video game commerce company that provides specialized financial and operational tools tailored for the gaming industry. The firm helps game developers and publishers fund, launch, market, and monetize their titles across PC, mobile, web, and cloud platforms. By operating as a merchant of record with support for over 1,000 local payment methods, it enables direct-to-consumer sales and seamless cross-border transactions for gaming studios worldwide.
xsolla.com • HQ: Los Angeles, United States • Financial Services • Payments • Game Development • Gaming • 1001-5000 employees • Est. 2006
HQ: Los Angeles, United States • Financial Services • Payments • Game Development • Gaming • 1001-5000 employees • Est. 2006
Open 109 daysVerified live · 1 day ago 4 months ago

Principal Engineer — DevX & AI

$108k – $141k per year • Lead • 10+ years expMontreal • Remote (Canada, Serbia) • Full-Time • Principal
JavaScript
TypeScript
AI/ML
AI Agents
Claude
Claude Code
Cursor
LLM
Model Context Protocol
LLM Evaluation
Frontend
React.js
DevOps
AWS
CI/CD
GCP
Design
Figma
Apply
Report

Guidehouse

guidehouse.com
Guidehouse is a leading global professional services firm that provides advisory, technology, and managed services to both government agencies and commercial enterprises. Formed initially from the separation of PwC's U.S. public sector practice and expanded through key acquisitions, it specializes in highly regulated markets such as defense, healthcare, energy, and financial services. The company leverages artificial intelligence, cybersecurity, and management expertise to help organizations navigate complex regulatory environments, modernize operations, and drive transformation.
guidehouse.com • HQ: McLean, United States • Financial Services • Government • Professional Services • 1001-5000 employees
HQ: McLean, United States • Financial Services • Government • Professional Services • 1001-5000 employees
Open 111 daysVerified live · 3 hours ago 4 months ago

AI/ML Solutions Architect (Agentic Systems)

$130k – $216k per year • Staff+ • 8+ years expIn office (Tysons, United States) • Bachelor's Degree • Full-Time • Architect
Python
SQL
Python
Asyncio
Boto3
Databases
DuckDB
ElasticSearch
OpenSearch
PostgreSQL
AI/ML
CrewAI
Embeddings
Haystack
LangChain
NLTK
NumPy
OpenCV
Pandas
Prompt Engineering
Pydantic AI
PyTorch
RAG
Scikit-learn
TensorFlow
AI Agents
AWS Strands Agents
Knowledge Graph
LLM Evaluation
LLM Guardrails
Model Context Protocol
CUDA
CUDA Toolkit
Human-in-the-Loop
DevOps
AWS
CI/CD
Vector
IAM
Cybersecurity
Least Privilege
Apply
Report

Magnet Forensics

magnetforensics.com
Magnet Forensics is a company that specializes in providing digital forensics and incident response tools. Their solutions, such as Magnet AXIOM and Magnet GrayKey, are used by law enforcement, military, intelligence, and enterprise organizations to recover, analyze, and manage digital evidence. They offer a wide range of tools to support digital investigations, including data extraction from mobile devices, video recovery, and workflow automation.
magnetforensics.com • Toronto • Ontario • Gothenburg • Waterloo • Ottawa • Homeland Security • National Security • Incident Response
Toronto • Ontario • Gothenburg • Waterloo • Ottawa • Homeland Security • National Security • Incident Response
Verified live · 8 hours ago 4 months ago

Senior AI/ML Engineer (Magnet-Griffeye)

$56k – $141k per year (Estimated) • Senior • 5+ years expRemote/Hybrid (Gothenburg, Sweden) • Bachelor's Degree • Full-Time • Senior
Python
AI/ML
Haystack
LLM
PyTorch
RAG
LLM Evaluation
AI Agents
Claude
Computer Vision
Copilot
NLP
DevOps
AWS
Azure
CI/CD
Vector
GitHub
Apply
Report

United Tech AI

unitedtech.ai
United Tech AI is a technology company specializing in artificial intelligence solutions, enterprise software development, and digital transformation services. The organization focuses on leveraging machine learning, cloud infrastructure, and advanced automation to help businesses streamline operations and drive innovation. By delivering tailored AI-driven platforms, it enables clients across various industries to make data-driven decisions and enhance their competitive edge.
unitedtech.ai • HQ: Kyiv, Ukraine • Software • Mobile App Development • Dating & Relationships
HQ: Kyiv, Ukraine • Software • Mobile App Development • Dating & Relationships
Open 112 daysVerified live · 1 day ago 4 months ago

AI Engineer

Senior • 5+ years expKyiv • Remote (Ukraine) • Full-Time • Senior • English: B2
Python
SQL
Databases
pgvector
Pinecone
PostgreSQL
AI/ML
AWS Bedrock
Claude
Embeddings
Function Calling
Hybrid Search
LangSmith
LLM
Prompt Engineering
Promptfoo
RAG
Ragas
Anthropic
LLM Evaluation
OpenAI
Structured Outputs
AI Agents
DevOps
AWS
AWS Lambda
CI/CD
Git
Vector
Amazon S3
IAM
GitHub
GitLab
Docker
Management
Confluence
Jira
Slack
n8n
Apply
Report

KOS

kos.ai
Kos reads the pay app, ties out the contract, and cuts the check. A virtual financial operations employee for companies that build the physical world.
kos.ai • San Francisco
San Francisco
Open 116 days 4 months ago

Software Engineer Intern

Intern • In office (San Francisco, United States) • Relocation • Visa sponsorship • Master's Degree • Internship • Intern
Python
TypeScript
AI/ML
Computer Vision
LLM Evaluation
OpenAI
DevOps
Azure
Apply
Open 116 daysTop 25% pay 4 months ago

Founding AI Engineer

$170k – $250k per year • Junior • 2+ years expRemote/Hybrid (San Francisco, United States) • Visa sponsorship • Bachelor's Degree • Full-Time • Junior
Computer Vision
Fine-tuning
LLM
LLM Evaluation
OpenAI
AI Agents
Function Calling
DevOps
Azure
Apply
Report

BrainCo

brainco.tech
BrainCo is a neurotechnology company headquartered in Somerville, Massachusetts, and was founded in 2015. The company develops brain-computer interface technology, including wearable EEG headbands for cognitive training and advanced prosthetic limbs controlled by muscle signals. It operates internationally across the education, healthcare, and wellness sectors, providing specialized tools for focus enhancement, physical rehabilitation, and STEM learning.
brainco.tech • HQ: Somerville, United States • Health & Medical Electronics • Medical & Healthcare Robotics • Robotics • STEM Education • Education • Medical Devices • Health Care • Consumer Electronics • Est. 2015
HQ: Somerville, United States • Health & Medical Electronics • Medical & Healthcare Robotics • Robotics • STEM Education • Education • Medical Devices • Health Care • Consumer Electronics • Est. 2015
Open 117 daysVerified live · 2 min ago 4 months ago

AI Platform Engineer, Agentic Engineering

$161k – $313k per year (Estimated) • Senior • 5+ years expRemote/Hybrid (San Francisco, United States) • Full-Time • Senior
Go
Python
Rust
TypeScript
Databases
Databricks
AI/ML
LLM
LLM Evaluation
LLM Guardrails
AI Agents
DevOps
Kubernetes
Cybersecurity
HIPAA
SOC 2
Apply
Report

Jonas Software

jonassoftware.com
Jonas Software is a Canadian software holding founded in 1990 and based in Markham. It owns dozens of vertical software businesses across many industries. The group is part of Constellation Software.
jonassoftware.com • HQ: Markham, Canada • 5000+ employees • Est. 1990
HQ: Markham, Canada • 5000+ employees • Est. 1990
Open 119 days 4 months ago

Technical Lead, Central AI Lab

$107k – $221k per year (Estimated) • Lead • 7+ years expUnited Kingdom • Remote (United Kingdom) • Full-Time • Staff
LLM
RAG
LLM Evaluation
Model Context Protocol
DevOps
AWS
Azure
CI/CD
Cybersecurity
Least Privilege
Apply
Report

Edisyl

edisyl.com
Flipside Crypto is a blockchain analytics company founded in 2017 and headquartered in Boston. It provides free SQL access to curated on-chain datasets and rewards a large community of analysts for producing dashboards and research. Protocol teams use its growth analytics to measure user behaviour and target ecosystem incentives.
edisyl.com • HQ: Boston, United States • Artificial Intelligence
HQ: Boston, United States • Artificial Intelligence
Open 125 days 5 months ago

AI Agent Architect

$166k – $310k per year (Estimated) • Staff+ • 6+ years expBoston • Remote (United States) • Full-Time • Architect
Python
AI/ML
AI Agents
AutoGen
Fine-tuning
Hallucination
LangChain
LlamaIndex
LLM
RAG
RLHF
Human-in-the-Loop
LLM Evaluation
Function Calling
DevOps
Vector
Apply
Report
Antioch University spans coast to coast, offering bachelor’s completion, master’s, and doctoral programs, plus certificates and endorsements online, on campus, and low-residency.
antioch.edu • New York • Health Care • Education • Higher Education
New York • Health Care • Education • Higher Education
Open 126 daysVerified live · 1 hour ago 5 months ago

Member of Technical Staff, Core Platform

$180k – $230k per year • Staff+ • In office (New York, United States) • Full-Time • Staff
C++
Python
Rust
AI/ML
AI Agents
Model Context Protocol
Claude
LLM
LLM Evaluation
LLM Guardrails
OpenAI Codex
DevOps
AWS
Azure
GCP
Kubernetes
Robotics
Digital Twin
Apply
Report

Architect AI

architect.ai
Architect AI is an artificial intelligence platform designed to transform urban design, architecture, and interior concepts into high-resolution visual renderings. The tool allows architects, designers, and real estate professionals to generate detailed 3D designs and explore spatial ideas rapidly using text prompts, sketch transformations, and mood boards. By streamlining the initial conceptual and visualization stages of building design, it helps creative teams accelerate project planning and present polished design options to clients.
architect.ai • Palo Alto • Bengaluru
Palo Alto • Bengaluru
Open 130 daysVerified live · 1 day ago 5 months ago

Member of Technical Staff - Applied AI

$172k – $348k per year (Estimated) • Equity • Staff+ • In office (Palo Alto, United States) • Master's Degree • Full-Time • Staff
Python
TypeScript
AI/ML
AI Agents
LLM
Anthropic
Edge AI
LLM Evaluation
Post-training
Function Calling
Apply
Report

Reflection AI

reflection.ai
Reflection AI is a company founded in 2024 by former Google DeepMind researchers who worked on AlphaGo and large language models. It builds autonomous coding agents and has committed to releasing frontier open-weight models as an American counterweight to Chinese open model releases. The company raised a very large round in 2025 to fund training at frontier scale.
reflection.ai • HQ: New York, United States • 501-1000 employees • Est. 2024
HQ: New York, United States • 501-1000 employees • Est. 2024
Open 132 daysVerified live · 7 hours ago 5 months ago

Forward Deployed Engineer - LLM Post-training

$117k – $248k per year (Estimated) • Equity • Middle • 3+ years expIn office (San Francisco, New York, United States) • Visa sponsorship • Full-Time • Middle
Python
AI/ML
Fine-tuning
RLHF
DPO
LLM Evaluation
Post-training
SFT
Apply
Open 306 daysVerified live · 7 hours ago 11 months ago

Forward Deployed Engineer, Lead - LLM Post-training

$155k – $332k per year (Estimated) • Equity • Lead • 6+ years expIn office (New York, San Francisco, United States) • Visa sponsorship • Full-Time • Staff
LLM
Reinforcement Learning
Synthetic Data
LLM Evaluation
Post-training
Apply
Report

Fieldguide

fieldguide.io
Fieldguide is an AI-powered automation platform designed specifically for risk, compliance, and audit advisory firms. Founded in 2020 and headquartered in San Francisco, the company provides cloud software that streamlines engagement workflows for cybersecurity and compliance frameworks like SOC 2, ISO 27001, and NIST. By automating data collection, testing, and report drafting, Fieldguide enables accounting and risk advisory professionals to deliver higher-quality audit services in less time.
fieldguide.io • HQ: San Francisco, United States • Artificial Intelligence • Financial Services • Audit • Est. 2020
HQ: San Francisco, United States • Artificial Intelligence • Financial Services • Audit • Est. 2020
Open 133 daysTop 25% pay 5 months ago

Senior AI Engineer, Quality

$200k – $250k per year • Senior • Remote/Hybrid (San Francisco, United States) • Full-Time • Senior
Python
TypeScript
Databases
PostgreSQL
AI/ML
AI Agents
LangGraph
LangSmith
LangChain
LLM Evaluation
LLM Guardrails
Langfuse
LLM
RAG
Lovable
DevOps
Vector
Apply
Report
LLM Evaluation Jobs - Remote & On-site
Frequently asked questions
How many LLM Evaluation jobs are available on Alion?
There are currently 491 active listings from 297 hiring companies on Alion, refreshed daily.
What salary do LLM Evaluation jobs pay?
The average salary is around $230,281 per year based on the current listings.
Are remote options available for LLM Evaluation jobs?
Yes, many of these roles offer remote or hybrid work, and you can filter listings by remote, relocation and visa sponsorship.
How do I apply for LLM Evaluation jobs?
Open any listing on Alion and apply directly with the employer through the provided link - no lengthy sign-up required.