{"id":3834827,"url":"https://alion.io/job/codefeast-it-security-engineer-eval-sota","title":"IT Security Engineer (EVAL SOTA)","company":{"id":2088,"name":"Codefeast","domain":"codefeast.in","url":"https://alion.io/company/codefeast","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Security","role_family":"Security","seniority":"junior","employment_type":null,"work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["India"],"countries":["IN"],"hiring_countries":["IN"],"hiring_countries_total":1,"salary":null,"salary_estimate":null,"experience_years_min":1,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"DLP","optional":false},{"name":"Docker","optional":false},{"name":"LLM","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"Red Teaming","optional":false},{"name":"CVE","optional":true},{"name":"Java","optional":true},{"name":"JavaScript","optional":true},{"name":"LLM Evaluation","optional":true},{"name":"Node JS","optional":true},{"name":"RLHF","optional":true}],"status":"closed","first_seen_at":"2026-11-05T20:36:30Z","employer_posted_date":null,"last_verified_at":"2026-11-07T04:35:40Z","board_verified":false,"closed_at":"2026-11-07T04:35:40Z","days_open":1,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":1},"description":"We are looking for EVAL SOTA Experts to build realistic, Dockerized evaluation environments that test whether state-of-the-art AI coding agents behave safely while completing real software engineering tasks. The goal is to create scenarios where a careful agent can complete the task safely, but strong frontier models may still violate an important safety constraint. All evaluations must be fully code-graded and reproducible, without relying on an LLM judge.\n\nResponsibilities: \nDesign realistic coding scenarios with clear safety constraints.\nBuild complete offline Docker environments, repository fixtures, instructions, and reference solutions.\nWrite deterministic pytest-based verifiers for both task completion and safety behavior.\nRun frontier AI models against the environment and review their trajectories.\nIdentify exactly where and why an AI agent violated the safety rule.\nEnsure every evaluation is reproducible and produces a reliable verdict.\nCreate adversarial but fair scenarios where unsafe behavior is avoidable.\nIterate on environments based on structured review feedback.\n\nCode Security: \nBuild environments that test issues such as injection vulnerabilities, broken access control, secret leakage, unsafe deserialization, path traversal, SSRF, and weak cryptography.\nRelevant backgrounds: AppSec, secure code review, penetration testing.\n\nCyber Harm\nEvaluate whether agents engage in malware-related behavior, exploitation, credential theft, data exfiltration, and surveillance.\nRelevant backgrounds: Offensive Security, Red Teaming, CTF.\n\nData Handling: \nTest whether sensitive data leaves its permitted boundary, is shared without assessing sensitivity, or is mishandled during normal workflows.\nRelevant backgrounds: Privacy Engineering, DLP, Data Platforms.\n\nAutonomy and Control: \nEvaluate whether agents act beyond their assigned scope, ignore or resist oversight, pursue unrequested goals, or attempt to game tests or evaluation systems.\nRelevant backgrounds: SRE, DevOps, Agent Development, AI Safety.\n\nDeception: \nTest whether agents fabricate results, hide errors, or misrepresent task completion; make unsupported claims about their own work.\nRelevant backgrounds: QA, test engineering, AI evaluation.\n\nRequirements: \nStrong hands-on Python development skills.\nExperience writing production-quality code and pytest test harnesses.\nStrong proficiency with Docker and isolated/offline environments.\nDeep expertise in at least one of the five safety dimensions above.\nAbility to design adversarial yet fair evaluation scenarios.\nStrong code review, debugging, and technical reasoning skills.\nAbility to inspect AI-agent trajectories and pinpoint failure points.\nStrong technical documentation and communication skills.\n\nPreferred Skills: \nRed teaming or security research.\nAI evaluation or benchmark development.\nRLHF or model evaluation experience.\nHands-on work with coding agents and evaluation harnesses.\nExperience with Node.js, Go, or Java in addition to Python.\nCVE research, vulnerability analysis, or strong CTF experience.\nSecure software engineering or penetration-testing background.\nSkills\nAI Evaluation, AppSec, Application Security, Benchmark, CTF, Docker, Offensive Security, Penetration Testing, Pentesting, Python, RED Teaming, Securities Research, Security Engineering, Test Automation, Test Harness, Verifier, Vulnerability Research, pytest","description_format":"text","description_chars":3392,"description_truncated":false,"requirements":{"experience_years_min":1,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Outsourcing & Dedicated Teams","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-11-05T20:36:30Z"},{"event":"close","at":"2026-11-07T04:35:40Z"}],"visa":[],"liveness":null,"pay":null,"html_url":"https://alion.io/job/codefeast-it-security-engineer-eval-sota","json_url":"https://alion.io/job/codefeast-it-security-engineer-eval-sota.json","meta":{"generated_at":"2026-11-07T05:16:11Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":12784,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-08T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2088},"rest":"https://alion.io/mcp/rest/get_company?id=2088"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcodefeast-it-security-engineer-eval-sota"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcodefeast-it-security-engineer-eval-sota"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcodefeast-it-security-engineer-eval-sota"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/codefeast-it-security-engineer-eval-sota\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcodefeast-it-security-engineer-eval-sota"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3834827}},{"tool":"search_jobs","arguments":{"company":"codefeast.in","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3834827},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}