IT Security Engineer (EVAL SOTA)

We are looking for EVAL SOTA Experts to build realistic, Dockerized evaluation environments that test whether state-of-the-art AI coding agents behave safely while completing real software engineering tasks. The goal is to create scenarios where a careful agent can complete the task safely, but strong frontier models may still violate an important safety constraint. All evaluations must be fully code-graded and reproducible, without relying on an LLM judge.

Responsibilities:

• Design realistic coding scenarios with clear safety constraints.
• Build complete offline Docker environments, repository fixtures, instructions, and reference solutions.
• Write deterministic pytest-based verifiers for both task completion and safety behavior.
• Run frontier AI models against the environment and review their trajectories.
• Identify exactly where and why an AI agent violated the safety rule.
• Ensure every evaluation is reproducible and produces a reliable verdict.
• Create adversarial but fair scenarios where unsafe behavior is avoidable.
• Iterate on environments based on structured review feedback.

Code Security:

• Build environments that test issues such as injection vulnerabilities, broken access control, secret leakage, unsafe deserialization, path traversal, SSRF, and weak cryptography.
• Relevant backgrounds: AppSec, secure code review, penetration testing.

Cyber Harm

• Evaluate whether agents engage in malware-related behavior, exploitation, credential theft, data exfiltration, and surveillance.
• Relevant backgrounds: Offensive Security, Red Teaming, CTF.

Data Handling:

• Test whether sensitive data leaves its permitted boundary, is shared without assessing sensitivity, or is mishandled during normal workflows.
• Relevant backgrounds: Privacy Engineering, DLP, Data Platforms.

Autonomy and Control:

• Evaluate whether agents act beyond their assigned scope, ignore or resist oversight, pursue unrequested goals, or attempt to game tests or evaluation systems.
• Relevant backgrounds: SRE, DevOps, Agent Development, AI Safety.

Deception:

• Test whether agents fabricate results, hide errors, or misrepresent task completion; make unsupported claims about their own work.
• Relevant backgrounds: QA, test engineering, AI evaluation.

Requirements:

• Strong hands-on Python development skills.
• Experience writing production-quality code and pytest test harnesses.
• Strong proficiency with Docker and isolated/offline environments.
• Deep expertise in at least one of the five safety dimensions above.
• Ability to design adversarial yet fair evaluation scenarios.
• Strong code review, debugging, and technical reasoning skills.
• Ability to inspect AI-agent trajectories and pinpoint failure points.
• Strong technical documentation and communication skills.

Preferred Skills:

• Red teaming or security research.
• AI evaluation or benchmark development.
• RLHF or model evaluation experience.
• Hands-on work with coding agents and evaluation harnesses.
• Experience with Node.js, Go, or Java in addition to Python.
• CVE research, vulnerability analysis, or strong CTF experience.
• Secure software engineering or penetration-testing background.

Skills

AI Evaluation, AppSec, Application Security, Benchmark, CTF, Docker, Offensive Security, Penetration Testing, Pentesting, Python, RED Teaming, Securities Research, Security Engineering, Test Automation, Test Harness, Verifier, Vulnerability Research, pytest