412,158open jobs
14,541companies
72,770added this week
Browse all
Location
Remote
Employment
Freelance
Overview
Company
Impact
Profile match
Agency AI is a developer platform and enterprise consulting firm specializing in AI agent observability, testing, and deployment, headquartered in San Francisco, California. Founded in 2023 by Alex Reibman, Adam Silverman, and Shawn Qiu, the company developed AgentOps, an industry-standard observability and evaluation tool suite that allows developers to monitor, debug, and optimize multi-agent autonomous workflows in real time across frameworks like AutoGen and CrewAI.

Project Overview  We are sourcing experienced technical specialists in CVE and Application Security to audit tasks used to train and evaluate AI systems. The objective of this project is to ensure that AI training workflows and tasks are technically rigorous, practical, and highly accurate within your specific area of expertise.

Project Deliverables & Scope

  • Task Evaluation:  Assess whether CVE and security tasks are technically accurate, realistic, solvable, reproducible, and supported by reliable tests and evaluation criteria.

  • Technical Auditing & Feedback:  Provide clear, actionable feedback on any identified security vulnerabilities, exploit execution flaws, or mitigation logic errors within the tasks.

Required Expertise

  • Demonstrable professional experience and deep technical knowledge in Application Security (including vulnerability assessment, secure coding practices, and threat modeling). (Note: Candidates will be considered for one specialty based on their experience; expertise across other domains is not required).

  • Strong analytical and problem-solving skills to rigorously test, troubleshoot, and evaluate complex technical scenarios.

We offer a pay range of $70 - $100 per hour, with the exact rate determined after evaluating your experience, expertise, and geographic location. Final offer amounts may vary from the pay range listed above. As a contractor you’ll supply a secure computer and high-speed internet; company-sponsored benefits such as health insurance and PTO do not apply.

Employment type:  Freelance Contract

Workplace type:  Remote

Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
412,158 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

Similar stack
Same company
In your city
$100k – $120k per year • Remote • 6+ years exp • Bachelor's Degree
Python
AI/ML
LLM
Cybersecurity
Threat Modeling
Apply
$100k – $150k per year • Remote • 6+ years exp • Bachelor's Degree
Python
AI/ML
LLM
Cybersecurity
Threat Modeling
Apply
$112k – $140k per year • In office
Java
TypeScript
DevOps
GitLab CI
Azure
CI/CD
Jenkins
AWS
AWS Lambda
GitLab
Cybersecurity
SonarQube
NIST 800-53
FedRAMP
STRIDE
Threat Modeling
SBOM
Apply
$80k – $105k per year • In office • 6+ years exp • Bachelor's Degree
Python
AI/ML
LLM
Cybersecurity
Threat Modeling
Apply
$80k – $105k per year • In office • 6+ years exp • Bachelor's Degree
Python
AI/ML
LLM
Cybersecurity
Threat Modeling
Apply
Remote • Freelance
AI/ML
TensorFlow
PyTorch
Apply
Remote • Freelance
DevOps
Terraform
CloudFormation
AWS
AWS Lambda
Apply
Remote • Freelance
AI/ML
CUDA Toolkit
CUDA
Triton
Apply
Remote • Freelance
AI/ML
AWS Trainium
DevOps
AWS
Apply
See all jobs
This is one of many
412,158 more open roles from verified company boards, updated every day.