{"id":2909179,"url":"https://alion.io/job/applied-retention-founding-ai-engineer-ai-evaluations-and-systems","title":"Founding AI Engineer - AI Evaluations and Systems","company":{"id":19453,"name":"Applied Retention","domain":"appliedretention.com","url":"https://alion.io/company/applied-retention","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"junior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Mumbai, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":10000,"max_usd":26000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":18},"experience_years_min":2,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Anthropic","optional":false},{"name":"Asyncio","optional":false},{"name":"AWS","optional":false},{"name":"Azure","optional":false},{"name":"Braintrust","optional":false},{"name":"CI/CD","optional":false},{"name":"DeepEval","optional":false},{"name":"Docker","optional":false},{"name":"Function Calling","optional":false},{"name":"GCP","optional":false},{"name":"GitHub Actions","optional":false},{"name":"HTTPX","optional":false},{"name":"Langfuse","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"OpenAI","optional":false},{"name":"PostgreSQL","optional":false},{"name":"Promptfoo","optional":false},{"name":"Pydantic","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Ragas","optional":false},{"name":"SQL","optional":false},{"name":"Structured Outputs","optional":false},{"name":"TruLens","optional":false},{"name":"Git","optional":true},{"name":"Machine Learning","optional":true}],"status":"live","first_seen_at":"2026-10-20T08:48:15Z","employer_posted_date":null,"last_verified_at":"2026-10-20T08:48:15Z","board_verified":false,"closed_at":null,"days_open":11,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":11},"description":"We are building a retention product for online merchants, with LLM-drafted content that is validated and human-approved before anything reaches a customer. The question this role answers: How do we know a change to a prompt, a model, or the code made the system better, not worse? You will build the evaluation suites that decide whether a change may ship, and help get that change to production safely. This role suits an engineer early in their career who has already built real evaluations and shipped something, and who wants to grow into owning both quality and reliability.\n\nThe candidate will have responsibilities across the following functions: \n\nMake AI Quality Measurable: \nBuild golden sets and behavioural assertions for LLM outputs.\nRead real outputs and traces first, and group failures into a taxonomy before choosing metrics.\nSet pass thresholds and track hallucinated or unsupported claims.\nTurn every production failure into a regression case.\n\nBuild Evaluation Pipelines That Run in CI: \nBuild regression suites for prompts, models, and tools.\nRun them in GitHub Actions or an equivalent platform so a regression blocks the merge.\nPin model versions and rerun suites on provider upgrades.\nRepeat runs and compare results properly, so noise is not mistaken for progress.\nReview changes against what their descriptions and tests claim, and add checks that would have caught any gaps.\nMake evaluation part of engineering, not a final QA step.\n\nEngineer LLM-as-a-Judge Systems: \nWrite structured rubrics and scoring mechanisms.\nCalibrate the judge against human-labelled examples and report agreement using precision, recall, and Cohen's kappa.\nMonitor judge bias, including position, verbosity, and self-preference.\n\nKeep Evaluation Affordable: \nUse deterministic checks for simple cases and reserve LLM judges for more complex cases.\nBatch and parallelise evaluation runs.\nTrack cost and latency per run.\n\nShip and Run It Reliably: \nContainerise services with Docker and automate builds and deployments.\nKeep deployments reversible through rollback steps, health checks, and backups that have been restored at least once.\nAdd tracing and cost tracking for model calls, along with actionable alerts.\nWrite runbooks and concise technical decision notes.\n\nRequirements: \n2+ years as a software engineer, with recent hands-on work on an LLM feature that people have used. RAG or agent products count.\nAt least one evaluation suite you built and ran, with a clear explanation of what it caught and what you changed because of it.\nPractical experience with LLM-as-a-judge, including rubrics, human labels, and calibration.\nHands-on experience with at least one of: promptfoo, DeepEval, Ragas, TruLens, LangSmith, Langfuse, Braintrust, Inspect, or an equivalent tool.\nFamiliarity with common evaluation measures: faithfulness, answer relevance, task completion, precision and recall, and confidence intervals.\n\nTechnical Skills: \nPython 3. x: pytest, Pydantic, httpx or requests, and asyncio basics.\nPostgreSQL / SQL: joins, aggregates, and window functions.\nLLM APIs: Anthropic and/or OpenAI SDKs, structured outputs, and tool calling.\nCI/CD: GitHub Actions or an equivalent platform, including a check that blocks a merge.\nDocker: Building and running containerised services.\nDeployment automation: Basic deploy scripts or pipeline jobs, environment configuration, and secrets handling.\nCloud: Working-level knowledge of any one of AWS, GCP, or Azure. AWS is a plus. If you have not used a cloud yet, Docker plus solid deployment automation is enough to start.\nGit: Pull-request workflows and code reviews.\nSkills\nLLMs, Machine Learning, Python, Regression Testing","description_format":"text","description_chars":3677,"description_truncated":false,"requirements":{"experience_years_min":2,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Marketing Automation","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-10-20T08:48:15Z"}],"visa":[],"liveness":{"score":74,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.819,"p_room":0.9,"age_days":10,"expected_fill_days":21,"reasons":["seen:10","win:mid","comp:junior,attention"],"computed_at":"2026-10-31T05:58:55Z"},"pay":null,"html_url":"https://alion.io/job/applied-retention-founding-ai-engineer-ai-evaluations-and-systems","json_url":"https://alion.io/job/applied-retention-founding-ai-engineer-ai-evaluations-and-systems.json","meta":{"generated_at":"2026-10-31T13:30:53Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":20972,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-01T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":19453},"rest":"https://alion.io/mcp/rest/get_company?id=19453"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fapplied-retention-founding-ai-engineer-ai-evaluations-and-systems"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fapplied-retention-founding-ai-engineer-ai-evaluations-and-systems"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fapplied-retention-founding-ai-engineer-ai-evaluations-and-systems"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/applied-retention-founding-ai-engineer-ai-evaluations-and-systems\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fapplied-retention-founding-ai-engineer-ai-evaluations-and-systems"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":2909179}},{"tool":"search_jobs","arguments":{"company":"appliedretention.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":2909179},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}