{"id":2040833,"url":"https://alion.io/job/alpaca-senior-data-scientist-ai-evaluation","title":"Senior Data Scientist AI Evaluation","company":{"id":3826,"name":"Alpaca","domain":"alpaca.markets","url":"https://alion.io/company/alpaca","size_band":"501-1000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"B","score":73,"open_postings":9,"ghost_share":0,"stale_share":0.889,"repost_share":0,"time_to_fill_p50_days":61,"computed_at":"2026-10-13T05:45:00Z"}},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":null,"work_mode":"remote","remote_scope":"stated_regions","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"inferred","locations":[],"countries":[],"hiring_countries":["US","CA","MX","AG","AW","BS","BB","BZ","BM","VG","BQ","KY","CR","CW","DM","DO","SV","GL","GD","GP","GT","HT","HN","JM","MQ","MS","NI","PA","PR","KN","LC","PM","SX","TT","TC","VI"],"hiring_countries_total":36,"salary":null,"salary_estimate":{"min_usd":114000,"max_usd":296000,"period":"year","method":"global_role_cell_scaled_by_country","sample_n":2157},"experience_years_min":6,"visa_sponsorship":false,"relocation_package":false,"has_equity":true,"technologies":[{"name":"LLM","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-10-07T15:55:18Z","employer_posted_date":"2026-10-07","last_verified_at":"2026-10-13T06:40:16Z","board_verified":true,"closed_at":null,"days_open":5,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":5},"description":"Who We Are:\nAlpaca is a US-headquartered, global leader in agent-first brokerage infrastructure for stocks, ETFs, options, crypto, fixed income, 24/5 trading, and more.\nAmongst our subsidiaries, Alpaca is a licensed financial services company, serving hundreds of financial institutions across 40 countries with our institutional-grade APIs. This includes broker-dealers, investment advisors, wealth managers, hedge funds, and crypto exchanges, totalling over 10 million brokerage accounts.\nOur global team is a diverse group of experienced engineers, traders, and brokerage professionals who are working to achieve our mission of opening financial services to everyone on the planet. We're deeply committed to open-source contributions and fostering a vibrant community, continuously enhancing our award-winning, developer-friendly API and the robust infrastructure behind it.\nAlpaca is proudly backed by $400 million in funding from top-tier global investors including Portage Ventures, Spark Capital, Tribe Capital, Social Leverage, Horizons Ventures, Opera Tech Ventures, SBI Group, Derayah Financial, Unbound, Peak XV, Elefund, and Y Combinator.\nOur Team Members:\nWe're a dynamic team of 400+ globally distributed members who thrive working from our favorite places around the world, with teammates spanning the USA, Canada, Japan, Hungary, Nigeria, Brazil, the UK, and beyond!\nWe're searching for passionate individuals eager to contribute to Alpaca's rapid growth. If you align with our core values-Stay Curious, Have Empathy, and Be Accountable-and are ready to make a significant impact, we encourage you to apply.\nYour Role: We're looking for a Senior Data Scientist, AI Evaluation to design how Alpaca measures whether our models and agents are actually right. You'll be a senior individual contributor who turns ambiguous quality questions into ground truth, scoring methods, and eval loops that the company can trust-and uses those results to make the systems better. You'll build on an established data foundation, so the focus is raising quality and speeding up safe rollout. You'll own the quality bar, independent of the teams that build and optimize those systems.\nThis role is for someone who cares as much about whether an answer is correct as about whether a model can generate one. You'll partner with Product, Engineering, Analytics Engineering, and business stakeholders to define what \"good\" looks like, build the evaluations that test it, and close the loop so evals drive iteration. If you have a strong quantitative background, have shipped rigorous, measurable work (evaluation, experimentation, or model validation), and want ownership over a greenfield eval practice at a fast-growing brokerage-infrastructure company, this is the role.\nWhat You'll Do\nDesign AI evaluations: Define ground truth, metrics, and scoring methods for models and agents.\nBuild repeatable eval loops: Track quality over time and catch regressions before release.\nPartner on infrastructure: Work with engineering and analytics engineering to operationalize eval harnesses.\nDrive iteration: Translate eval results into actionable recommendations for system improvements.\nEstablish quality standards: Set evaluation guidelines, documentation, and review practices.\nMentor and align: Foster evaluation best practices and build a culture of measurable AI quality across the team.\nWhat We're Looking For\nTrack record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation).\nStrong statistical and ML foundation-you treat evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validate automated graders against human ground truth.\nProficiency in Python and SQL, with experience evaluating models in production environments.\nStrong judgment in defining quality metrics and ground truth for ambiguous outputs.\nExcellent communication and cross-functional collaboration skills to align technical teams and leadership.\nStrong problem-solving ability in fast-paced, greenfield environments.\n6-10 years in quantitative data science or ML, with focused experience in measurement or evaluation. A quantitative degree is a plus; equivalent industry experience is equally welcome.\nNice to Have\nHands-on LLM/agent evaluation in production, including eval harnesses, LLM-as-judge calibration, and CI regression gates.\nExperience evaluating text-to-SQL, analytics agents, or other systems where correctness is verifiable against data.\nBackground in fintech, brokerage, or other domains where a wrong answer has real business or risk consequences.\nFluency with AI tools in research and engineering workflows.\nHow We Take Care of You:\nCompetitive Salary & Stock Options\nHealth Benefits\nNew Hire Home-Office Setup: One-time USD $500\nMonthly Stipend: USD $150 per month via a Brex Card\nAlpaca is proud to be an equal opportunity workplace dedicated to pursuing and hiring a diverse workforce.\n\nRecruitment Privacy Policy","description_format":"text","description_chars":5004,"description_truncated":false,"requirements":{"experience_years_min":6,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Stock options"],"hiring_locations":[{"name":"United States","iso":"US","kind":"region"},{"name":"Canada","iso":"CA","kind":"region"},{"name":"Mexico","iso":"MX","kind":"region"},{"name":"Anguilla","iso":null,"kind":"region"},{"name":"Antigua and Barbuda","iso":"AG","kind":"region"},{"name":"Aruba","iso":"AW","kind":"region"},{"name":"Bahamas","iso":"BS","kind":"region"},{"name":"Barbados","iso":"BB","kind":"region"},{"name":"Belize","iso":"BZ","kind":"region"},{"name":"Bermuda","iso":"BM","kind":"region"},{"name":"British Virgin Islands","iso":"VG","kind":"region"},{"name":"Bonaire, Sint Eustatius and Saba","iso":"BQ","kind":"region"},{"name":"Cayman Islands","iso":"KY","kind":"region"},{"name":"Costa Rica","iso":"CR","kind":"region"},{"name":"Curaçao","iso":"CW","kind":"region"},{"name":"Dominica","iso":"DM","kind":"region"},{"name":"Dominican Republic","iso":"DO","kind":"region"},{"name":"El Salvador","iso":"SV","kind":"region"},{"name":"Greenland","iso":"GL","kind":"region"},{"name":"Grenada","iso":"GD","kind":"region"},{"name":"Guadeloupe","iso":"GP","kind":"region"},{"name":"Guatemala","iso":"GT","kind":"region"},{"name":"Haiti","iso":"HT","kind":"region"},{"name":"Honduras","iso":"HN","kind":"region"},{"name":"Jamaica","iso":"JM","kind":"region"},{"name":"Martinique","iso":"MQ","kind":"region"},{"name":"Montserrat","iso":"MS","kind":"region"},{"name":"Nicaragua","iso":"NI","kind":"region"},{"name":"Panama","iso":"PA","kind":"region"},{"name":"Puerto Rico","iso":"PR","kind":"region"},{"name":"Saint Kitts and Nevis","iso":"KN","kind":"region"},{"name":"Saint Lucia","iso":"LC","kind":"region"},{"name":"Saint Vincent and the Grenadines","iso":null,"kind":"region"},{"name":"Saint-Pierre and Miquelon","iso":"PM","kind":"region"},{"name":"Sint Maarten","iso":"SX","kind":"region"},{"name":"Trinidad and Tobago","iso":"TT","kind":"region"},{"name":"Turks and Caicos Islands","iso":"TC","kind":"region"},{"name":"United States Virgin Islands","iso":"VI","kind":"region"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Financial Software & Embedded Finance","Online Brokerage & Trading Platforms"],"lifecycle":[{"event":"open","at":"2026-10-07T17:50:05Z"}],"visa":[],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":5,"expected_fill_days":61,"reasons":["conf:2","velocity","win:early"],"computed_at":"2026-10-13T05:45:00Z"},"pay":null,"html_url":"https://alion.io/job/alpaca-senior-data-scientist-ai-evaluation","json_url":"https://alion.io/job/alpaca-senior-data-scientist-ai-evaluation.json","meta":{"generated_at":"2026-10-13T08:47:11Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":13050,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-14T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3826},"rest":"https://alion.io/mcp/rest/get_company?id=3826"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Falpaca-senior-data-scientist-ai-evaluation"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Falpaca-senior-data-scientist-ai-evaluation"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Falpaca-senior-data-scientist-ai-evaluation"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/alpaca-senior-data-scientist-ai-evaluation\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Falpaca-senior-data-scientist-ai-evaluation"}]}