{"id":40244,"url":"https://alion.io/job/sarvam-ai-data-scientist-evaluations-chanakya","title":"Data Scientist - Evaluations, Chanakya","company":{"id":229,"name":"Sarvam AI","domain":"sarvam.ai","url":"https://alion.io/company/sarvam-ai","size_band":"501-1000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"A","score":95,"open_postings":37,"ghost_share":0,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":91,"computed_at":"2026-10-29T05:49:05Z"}},"role":"Data Science","role_family":"Data Science","seniority":"middle","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":15500,"max_usd":41000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":21},"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Fine-tuning","optional":false},{"name":"Hugging Face","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"NumPy","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Python","optional":false},{"name":"RLHF","optional":false}],"status":"live","first_seen_at":"2026-04-17T11:56:48Z","employer_posted_date":"2026-04-17","last_verified_at":"2026-10-29T13:20:26Z","board_verified":true,"closed_at":null,"days_open":195,"trust":{"level":"ok","repost_count":0,"flags":[],"days_open":194},"description":"About Sarvam\nSarvam is building the bedrock of Sovereign AI for India. The company is developing India's full-stack sovereign AI platform, building across research, models, infrastructure and applications with a singular focus on making AI genuinely work for India. Sarvam works with leading enterprises and public institutions and is backed by Lightspeed, Peak XV, and Khosla Ventures. Sarvam partners with India's leading brands, including Tata Capital, SBI Life, CRED, IDFC, and LIC.\nAbout the Role\nThis is an intellectually demanding role. The Data Scientist anchors the evaluations function for this vertical - designing, building, and maintaining evaluation frameworks that measure model and system quality in operational context. You are not running standard benchmarks. You are building domain-specific eval harnesses for high-stakes use cases where a wrong answer carries real consequences.\nYou will work closely with the MLOps Engineer, the PM, and the deployment team. The evaluations you design are the mechanism by which the team determines whether what we've built is good enough to deploy - and whether it stays good after deployment.\nWhat You'll Do\nDesign and build evaluation frameworks for Sarvam's AI outputs across domain-specific requirements: document comprehension, command summarisation, geospatial reasoning, enterprise workflow automation, and others as they emerge\nDefine quality metrics in collaboration with domain experts and clients; translate operational requirements into measurable, defensible signals\nRun structured evaluation cycles pre- and post-deployment; build dashboards that surface model quality in production\nIdentify failure modes, edge cases, and distribution shifts - with the bias of someone looking for what's wrong, not confirming what's right\nCollaborate with the MLOps Engineer to operationalise eval pipelines - automated, triggered by deployment events, versioned, and reproducible\nBuild and manage domain-specific datasets for fine-tuning, evaluation, and benchmarking - including human annotation workflows where needed\nPublish internal findings and quality reports that feed the product and engineering roadmap\nWhat We're Looking For\n3-6 years in data science, ML research, or applied AI; at least 2 years working with LLMs in production contexts\nStrong statistics and probability fundamentals - you understand what makes an evaluation valid and what makes it misleading\nExperience designing evaluation frameworks from scratch: custom metrics, inter-rater reliability, red-teaming methodologies\nPython proficiency; comfort with pandas, NumPy, HuggingFace datasets, RAGAS, EleutherAI Eval Harness, LangSmith, or equivalent\nExperience with prompt engineering, model fine-tuning, or RLHF in applied settings\nAbility to work with unstructured domain data: PDFs, doctrine documents, transcripts, and field reports\nBonus Points\nPrior work in high-stakes domains (healthcare, legal, defence, finance) where output quality carries real-world consequences\nRed-teaming or adversarial evaluation experience\nNote: We are looking for people who can own the outcomes described here, not people who\nmatch every line of this specification. If this problem excites you and you believe you can do this work, we want to hear from you.\nWhy Sarvam?\nSarvam is a fast-moving, high talent-density team building full-stack AI for India, working on problems that push the frontiers of AI with real population-scale impact.\nWork alongside researchers, engineers, builders, and business leaders who move fast and hold each other to a very high bar\nHigh ownership and high impact, from day one\nEverything we do is AI-first, from the way we build and ship to the way we think about problems\nYou can work on problems that could change how an entire country learns, works, and communicates\nIf you want to work on problems at the frontier of AI in India, Sarvam is the place to be.","description_format":"text","description_chars":3905,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["LLM & Generative AI","Natural Language Processing","Speech & Audio AI","Foundation Models"],"lifecycle":[{"event":"open","at":"2026-04-17T11:56:48Z"}],"visa":[],"liveness":{"score":18,"band":"cold","label":"Long shot","p_open":1,"p_active":0.636,"p_room":0.28,"age_days":194,"expected_fill_days":91,"reasons":["conf:1","velocity","win:tail","crowd:"],"computed_at":"2026-10-29T05:49:05Z"},"pay":null,"html_url":"https://alion.io/job/sarvam-ai-data-scientist-evaluations-chanakya","json_url":"https://alion.io/job/sarvam-ai-data-scientist-evaluations-chanakya.json","meta":{"generated_at":"2026-10-29T13:21:12Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":20035,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-30T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":229},"rest":"https://alion.io/mcp/rest/get_company?id=229"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsarvam-ai-data-scientist-evaluations-chanakya"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsarvam-ai-data-scientist-evaluations-chanakya"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsarvam-ai-data-scientist-evaluations-chanakya"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/sarvam-ai-data-scientist-evaluations-chanakya\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsarvam-ai-data-scientist-evaluations-chanakya"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":40244}},{"tool":"search_jobs","arguments":{"company":"sarvam.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":40244},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}