{"id":2695296,"url":"https://alion.io/job/patronus-ai-software-engineer-ai-evaluation-expert-contractual","title":"Software Engineer – AI Evaluation Expert (Contractual)","company":{"id":40918,"name":"Patronus AI","domain":"patronus.ai","url":"https://alion.io/company/patronus","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":null},"role":"Backend","role_family":"Backend","seniority":"senior","employment_type":"contractor","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["India"],"countries":["IN"],"hiring_countries":["IN"],"hiring_countries_total":1,"salary":{"min":30,"max":100,"currency":"USD","period":"hour","gross":null,"usd_annual":200000},"salary_estimate":null,"experience_years_min":2,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"C++","optional":false},{"name":"JavaScript","optional":false},{"name":"LLM","optional":false},{"name":"Python","optional":false},{"name":"TypeScript","optional":false}],"status":"live","first_seen_at":"2026-10-16T08:06:22Z","employer_posted_date":"2026-10-16","last_verified_at":"2026-10-18T08:05:22Z","board_verified":true,"closed_at":null,"days_open":2,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":2},"description":"About Patronus AI\nPatronus AI is a frontier lab developing simulation research and infrastructure to accelerate progress toward human-aligned AGI. We are on a mission to simulate all of the world’s intelligence.\nWe are the team behind some of the earliest and most influential research in AI evaluation likeFinanceBench,Lynx,SimpleSafetyTests,CopyrightCatcher,Humanity’s Last Exam, and more. We are formerly AI researchers and engineers from companies like Meta AI, Amazon AGI, and Google. Our customers include foundation model labs and Fortune 500 enterprises like Adobe. We are backed by top-tier investors like Lightspeed Venture Partners, Notable Capital, Stanford University, Noam Brown, Gokul Rajaram, and more.\nAbout the role \n We are looking for experienced software engineers (task-based contract) to design realistic, challenging tasks for frontier AI models and to judge how well those models handle them. You'll apply your day-to-day engineering judgment to show where AI gets code, reasoning and technical instructions right, and where it falls short.\nWhat you’ll do\nDepending on the project, you may:\nDesign realistic, multi-step coding and technical tasks that test an AI model's ability to reason, write code and follow precise instructions.\nBuild tasks around realistic workspace materials such as codebases, technical specs, architecture docs and datasets.\nEvaluate AI-generated code and technical content for correctness, efficiency, clarity and alignment with real-world engineering practice.\nGrade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.\nHelp improve how AI understands programming, system design, data workflows and engineering best practices.\nWork independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high.\nTest and refine content to reflect real-world complexity.\nReview other contributors’ work for accuracy, clarity, and practical relevance.\nIncorporate feedback promptly and flag gaps or unclear guidance.\nWhat we’re looking for\n2+ years of professional experience in software engineering, data science or analytics.\nExperience with system design, data pipelines, ML or LLM-based applications.\nPrior work in code review, technical writing, AI training or model evaluation.\nStrong coding skills in at least one major language, such as Python, Java, C++, JavaScript, TypeScript or Go.\nExperience building, reviewing and debugging production-grade code.\nAbility to follow detailed technical guidelines, work independently, respond to feedback, and meet deadlines.\nExcellent written communication and close attention to detail.\nA bachelor's degree in computer science, engineering or a related field, or equivalent experience.\nOur standards\nAll tasks and solutions must be human-authored. LLM-generated content is not permitted.\nTasks must be original and non-searchable.\nAccurate, carefully checked contributions.\nQuality over volume.\nPay: $30-$100 per hour and some tasks would have a flat rate\nLocation: Remote (India)\nType: Hourly, project/task-based contract (not a full-time role)\nCommitment: Flexible, typically 15-20 hours per week; can be done alongside a full-time job\nApplications are reviewed on a rolling basis. Placement depends on project needs and assessment results.\nPatronus AI is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.\nBy clicking ‘Apply’, you agree to Greenhouse's Terms of Service and Privacy Policy.\nBy clicking 'Apply', you agree to Patronus AI, Inc. Privacy Policy.","description_format":"text","description_chars":3984,"description_truncated":false,"requirements":{"experience_years_min":2,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","AI Safety & Alignment","AI Research Labs","AI Evaluation & Observability"],"lifecycle":[{"event":"open","at":"2026-10-16T09:57:33Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":45,"reasons":["conf:3","win:early"],"computed_at":"2026-10-17T05:46:00Z"},"pay":{"stated_usd_annual":200000,"is_top_pay":true},"html_url":"https://alion.io/job/patronus-ai-software-engineer-ai-evaluation-expert-contractual","json_url":"https://alion.io/job/patronus-ai-software-engineer-ai-evaluation-expert-contractual.json","meta":{"generated_at":"2026-10-18T10:16:05Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":15336,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-19T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":40918},"rest":"https://alion.io/mcp/rest/get_company?id=40918"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fpatronus-ai-software-engineer-ai-evaluation-expert-contractual"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fpatronus-ai-software-engineer-ai-evaluation-expert-contractual"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fpatronus-ai-software-engineer-ai-evaluation-expert-contractual"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/patronus-ai-software-engineer-ai-evaluation-expert-contractual\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fpatronus-ai-software-engineer-ai-evaluation-expert-contractual"}]}