{"id":4002078,"url":"https://alion.io/job/yalo-staff-data-scientist-agent-improvement-loop","title":"Staff Data Scientist, Agent Improvement Loop","company":{"id":57969,"name":"Yalo","domain":"yalo.ai","url":"https://alion.io/company/yalo","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"staff","employment_type":null,"work_mode":"remote","remote_scope":"stated_regions","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"inferred","locations":[],"countries":[],"hiring_countries":["AR","BR","CO","MX","AI","AG","AW","BS","BB","BZ","BM","BO","VG","BQ","KY","CL","CR","CW","DM","DO","EC","SV","FK","GF","GD","GP","GT","GY","HT","HN","JM","MQ","MS","NI","PA","PY","PE","PR","KN","LC"],"hiring_countries_total":49,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"CI/CD","optional":false},{"name":"LangGraph","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"Python","optional":false},{"name":"SQL","optional":false},{"name":"TypeScript","optional":false},{"name":"A/B Testing","optional":true},{"name":"Amazon SageMaker","optional":true},{"name":"LangChain","optional":true},{"name":"LLMOps","optional":true},{"name":"MLFlow","optional":true},{"name":"NLP","optional":true}],"status":"live","first_seen_at":"2026-11-10T00:24:46Z","employer_posted_date":"2026-11-10","last_verified_at":"2026-11-10T04:04:14Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"Yalo\nHi! This is Yalo! We are on a mission to bring conversational commerce to the world...\nRemember how it used to be to interact with businesses that knew and understood you, that could recommend exactly what you needed, and that with a simple message could get you what you wanted??? Yep... neither do we. That is why at Yalo we are marrying the scale of digital commerce with the personalization and simplicity of conversations to help companies delight their users.\nWe know that traditional SAAS companies focus on first-world problems... we don't! Having started in Latin America, our roots are in Emerging Markets and therefore we care about bringing amazing experiences to a population that traditionally has been underserved, such as the small shop owner in Brazil who is ordering online for the first time.\nIf you're looking for a place to make things happen, learn fast, and impact emerging markets in a way that hasn't been done before, look no further.\nCome Join us in our mission of improving billions of lives through the power of conversational commerce!\nYour mission\nAs a Staff Data Scientist, you make our AI agents measurably better, and you prove it. You own how Yalo evaluates agents: evals, trace and error analysis, and the improvement loop that turns what we find in production into fixes to our agent skills, harness, CLI and customers' conversational flows. You work with Product, Engineering, Delivery and Customer Success. Your job is to show what moved the metric, not to guess.\nWhat are the responsibilities for this role?\nOwn the evals strategy for our agents: pre-release behavioral testing, production monitoring, and the judges that score conversations.\nRun continuous error analysis on agent traces at scale. Find failure patterns, attribute them to skills, prompts or releases, and rank improvement opportunities by impact.\nClose the loop. Turn findings into fixes and new evals, ship them, and measure the result with sound before-and-after and causal methods.\nImprove our agent skills, harness and CLI using trace evidence. Work through the owning teams' repos and review process.\nPartner with delivery and customer teams to take real customer flows from underperforming to reliably healthy, and make the process repeatable for others.\nCalibrate and validate LLM judges against human review. Measure their stability, report their limits, and set the bar for what counts as evidence at Yalo.\nWrite production-quality Python and TypeScript for evals and analysis tooling, with tests and CI, so it can live in a shared codebase.\nMentor data scientists and engineers on measurement, experimentation and evaluation practice.\nHelp shape quarterly and annual planning for agent quality.\nJob Requirements (Must have)\nBS or MS in a technical field (Computer Science, Statistics, Applied Math, Operations Research, Economics, etc.).\n8+ years delivering data science solutions with measurable business impact in cross-functional teams.\nHands-on experience evaluating LLM-based systems or agents: eval design, LLM-as-judge, rubric design, and judge calibration.\nExpert in Python and SQL. Comfortable in TypeScript, or ready to be.\nStrong production engineering habits: version control, code review, testing, CI/CD, and shipping tools others use.\nExperience with GenAI systems in production: prompt and tool design, agent frameworks (e.g., LangGraph), and LLM observability tooling (e.g., LangSmith).\nSkilled at turning messy trace and conversation data into clear findings for technical and non-technical audiences.\nStrong product and business sense. You frame quality work in terms of customer outcomes.\nNice to have:\nExperience with conversational AI, customer-facing agents or tool-using agents.\nNLP and clustering for failure-pattern discovery.\nCausal inference and experimentation skills: A/B tests, difference-in-differences, pre/post analysis, and the judgment to say what the data can't show.\nExposure to MLOps or LLMOps platforms (Vertex, SageMaker, MLflow, or similar) and big data tools.\nFamiliarity with AI safety and responsible AI practices.\nComfort in an ambiguous, fast-changing environment.\nExperience mentoring other data scientists.\nWhat do we offer? \nUnlimited PTO policy\nCompetitive rewards on the market range\nRemote working is available (-+3 hours CT)\nFlexible time (driven by results)\nStart-up environment\nInternational teamwork\nYou and nothing else limit your career here\nWe care,\nWe keep it simple,\nWe make it happen,\nWe strive for excellence. \nAt Yalo, we are dedicated to creating a workplace that embodies our core values: caring, initiative, excellence, and simplicity. We believe in the power of diversity and inclusivity, where everyone's unique perspectives, experiences, and talents contribute to our collective success. As we embrace and respect our differences, we strive to create something extraordinary for the benefit of all.\nWe are proud to be an Equal Opportunity Employer, providing equal opportunities to individuals regardless of race, color, religion, national or ethnic origin, gender, sexual orientation, gender identity or expression, age, disability, protected veteran status, or any other legally protected characteristic. Our commitment to fairness and equality is a fundamental pillar of our company.\nAt Yalo, we uphold a culture of excellence. We constantly challenge ourselves to go above and beyond, delivering remarkable results and driving innovation. We encourage each team member to take initiative and make things happen, empowering them to bring their best ideas forward and contribute to our shared goals.","description_format":"text","description_chars":5594,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["Unlimited PTO"],"hiring_locations":[{"name":"Argentina","iso":"AR","kind":"country"},{"name":"Brazil","iso":"BR","kind":"region"},{"name":"Colombia","iso":"CO","kind":"region"},{"name":"Mexico","iso":"MX","kind":"region"},{"name":"Anguilla","iso":"AI","kind":"region"},{"name":"Antigua and Barbuda","iso":"AG","kind":"region"},{"name":"Aruba","iso":"AW","kind":"region"},{"name":"Bahamas","iso":"BS","kind":"region"},{"name":"Barbados","iso":"BB","kind":"region"},{"name":"Belize","iso":"BZ","kind":"region"},{"name":"Bermuda","iso":"BM","kind":"region"},{"name":"Bolivia","iso":"BO","kind":"region"},{"name":"British Virgin Islands","iso":"VG","kind":"region"},{"name":"Bonaire, Sint Eustatius and Saba","iso":"BQ","kind":"region"},{"name":"Cayman Islands","iso":"KY","kind":"region"},{"name":"Chile","iso":"CL","kind":"region"},{"name":"Costa Rica","iso":"CR","kind":"region"},{"name":"Curaçao","iso":"CW","kind":"region"},{"name":"Dominica","iso":"DM","kind":"region"},{"name":"Dominican Republic","iso":"DO","kind":"region"},{"name":"Ecuador","iso":"EC","kind":"region"},{"name":"El Salvador","iso":"SV","kind":"region"},{"name":"Falkland Islands","iso":"FK","kind":"region"},{"name":"French Guiana","iso":"GF","kind":"region"},{"name":"Grenada","iso":"GD","kind":"region"},{"name":"Guadeloupe","iso":"GP","kind":"region"},{"name":"Guatemala","iso":"GT","kind":"region"},{"name":"Guyana","iso":"GY","kind":"region"},{"name":"Haiti","iso":"HT","kind":"region"},{"name":"Honduras","iso":"HN","kind":"region"},{"name":"Jamaica","iso":"JM","kind":"region"},{"name":"Martinique","iso":"MQ","kind":"region"},{"name":"Montserrat","iso":"MS","kind":"region"},{"name":"Nicaragua","iso":"NI","kind":"region"},{"name":"Panama","iso":"PA","kind":"region"},{"name":"Paraguay","iso":"PY","kind":"region"},{"name":"Peru","iso":"PE","kind":"region"},{"name":"Puerto Rico","iso":"PR","kind":"region"},{"name":"Saint Kitts and Nevis","iso":"KN","kind":"region"},{"name":"Saint Lucia","iso":"LC","kind":"region"},{"name":"Saint Vincent and the Grenadines","iso":"VC","kind":"region"},{"name":"Saint-Pierre and Miquelon","iso":"PM","kind":"region"},{"name":"Sint Maarten","iso":"SX","kind":"region"},{"name":"Suriname","iso":"SR","kind":"region"},{"name":"Trinidad and Tobago","iso":"TT","kind":"region"},{"name":"Turks and Caicos Islands","iso":"TC","kind":"region"},{"name":"Uruguay","iso":"UY","kind":"region"},{"name":"United States Virgin Islands","iso":"VI","kind":"region"},{"name":"Venezuela","iso":"VE","kind":"region"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Commerce","Conversational AI"],"lifecycle":[{"event":"open","at":"2026-11-10T04:04:14Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":36,"reasons":["conf:2","win:early"],"computed_at":"2026-11-10T06:30:43Z"},"pay":null,"html_url":"https://alion.io/job/yalo-staff-data-scientist-agent-improvement-loop","json_url":"https://alion.io/job/yalo-staff-data-scientist-agent-improvement-loop.json","meta":{"generated_at":"2026-11-10T06:30:43Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":15184,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-11T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":57969},"rest":"https://alion.io/mcp/rest/get_company?id=57969"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fyalo-staff-data-scientist-agent-improvement-loop"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fyalo-staff-data-scientist-agent-improvement-loop"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fyalo-staff-data-scientist-agent-improvement-loop"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/yalo-staff-data-scientist-agent-improvement-loop\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fyalo-staff-data-scientist-agent-improvement-loop"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4002078}},{"tool":"search_jobs","arguments":{"company":"yalo.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4002078},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}