{"id":3520430,"url":"https://alion.io/job/klaviyo-lead-product-data-scientist-agents","title":"Lead Product Data Scientist, Agents","company":{"id":6845,"name":"Klaviyo","domain":"klaviyo.com","url":"https://alion.io/company/klaviyo","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"B","score":83,"open_postings":22,"ghost_share":0,"stale_share":0.682,"repost_share":0,"time_to_fill_p50_days":57,"computed_at":"2026-11-01T06:10:25Z"}},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Boston, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":172000,"max":258000,"currency":"USD","period":"year","gross":null,"usd_annual":258000},"salary_estimate":null,"experience_years_min":10,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"dbt","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Python","optional":false},{"name":"SQL","optional":false},{"name":"WhatsApp","optional":false},{"name":"Arize Phoenix","optional":true},{"name":"Braintrust","optional":true},{"name":"Human-in-the-Loop","optional":true},{"name":"LangSmith","optional":true}],"status":"live","first_seen_at":"2026-10-30T11:44:42Z","employer_posted_date":"2026-10-30","last_verified_at":"2026-11-01T14:07:58Z","board_verified":true,"closed_at":null,"days_open":2,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":2},"description":"At Klaviyo, we value the unique backgrounds, experiences and perspectives each Klaviyo (we call ourselves Klaviyos) brings to our workplace each and every day. We believe everyone deserves a fair shot at success and appreciate the experiences each person brings beyond the traditional job requirements. If you’re a close but not exact match with the description, we hope you’ll still consider applying. Want to learn more about life at Klaviyo? Visit klaviyo.com/careers to see how we empower creators to own their own destiny.\nLead Product Data Scientist, Agents\nAbout the Team\nKlaviyo's Product Data Science team plays a critical role in shaping how we understand, measure, and improve product and business performance. We partner closely with Product, Engineering, Finance, and Go-to-Market teams to turn complex data into actionable insights that inform product strategy, customer experience, and long-term growth.\nAs Klaviyo continues to expand its AI-powered capabilities, intelligent agents are becoming central to how businesses engage with their customers. Composer, Klaviyo's AI marketing agent, helps marketers analyze, build, and execute their marketing strategies, while Customer Agent enables brands to engage shoppers and answer questions across channels including chat, email, SMS, and WhatsApp.\nThese products are evolving rapidly, creating new opportunities and challenges in understanding agent performance, measuring customer value, and connecting AI-driven experiences to meaningful business outcomes.\nOur team takes an AI-first approach to data science, leveraging emerging technologies and agentic tools to accelerate analytical workflows, improve decision-making, and focus our efforts on the complex questions that matter most to customers and the business.\nAbout the Role\nWe're looking for a Lead Product Data Scientist, Agents to own and advance Klaviyo's measurement and data science strategy across Composer and Customer Agent, combining deep technical expertise with strong product and business judgment to influence some of our most important AI investments.\nThis is a highly visible, hands-on technical leadership role at the intersection of data science, AI product development, and business strategy. You'll define how we measure the adoption, engagement, quality, cost, and business impact of our AI agents, establishing the analytical foundations that help teams understand what creates meaningful customer value.\nThis isn't just about building dashboards or tracking traditional product metrics. We're looking for someone who can tackle emerging measurement challenges, develop rigorous approaches to evaluating agent performance, and determine whether AI-powered experiences are genuinely improving customer outcomes. You'll combine experimentation, causal inference, and advanced analytics to help leaders make informed decisions in an environment where established measurement frameworks are still evolving.\nYou'll partner closely with Product, Engineering, Finance, and executive leadership to shape product strategy, evaluate pricing and monetization opportunities, and translate complex analytical findings into actionable recommendations. You'll also help define the technical direction of agent analytics at Klaviyo, mentor other data scientists, and establish the shared metrics, methodologies, and data foundations that enable teams to make confident, data-driven decisions.\nHow You'll Make a Difference\nOwn agent measurement end to end across Composer and Customer Agent, developing comprehensive measurement frameworks that connect adoption, engagement, agent performance, customer value, and revenue impact.\nDefine what successful agent performance looks like by partnering with Product and Engineering to establish meaningful measures of agent effectiveness, task completion, customer outcomes, failure rates, and the trade-offs between quality, cost, and business value.\nBuild scalable agent evaluation frameworks by combining conversation data, agent traces, tool-call outcomes, human evaluation, and LLM-based classification to assess quality, identify failure patterns, and understand how changes to models and agent behavior influence product performance.\nMeasure causal impact and incremental value through experimentation and advanced statistical methods, including quasi-experimental approaches such as difference-in-differences, matched comparisons, staged rollouts, and synthetic controls, to determine how AI agents influence customer behavior, retention, and revenue.\nConnect agent performance to business strategy by partnering with Finance and Product teams to evaluate usage-based pricing, packaging, credit economics, model costs, conversion, and the financial implications of product investments.\nEstablish trusted metrics and analytical standards by defining consistent measurement methodologies, governed data models, and evaluation practices that enable teams to make informed decisions as AI products and underlying technologies evolve.\nLead with an AI-first approach to analytics, leveraging coding agents, LLMs, and emerging tools to accelerate SQL development, data modeling, dashboard creation, conversation classification, and analytical workflows while maintaining rigorous validation and quality standards.\nBuild scalable, self-service analytics capabilities that empower product managers and cross-functional stakeholders to independently explore agent performance, evaluate customer outcomes, and identify opportunities for improvement.\nPartner with Engineering and Analytics Engineering to develop reliable instrumentation, data pipelines, and dbt models that capture agent interactions, support robust evaluation, and maintain data quality as new capabilities are introduced.\nLead the technical direction of agent analytics, reviewing analytical approaches, mentoring other data scientists, establishing best practices, and raising the bar for statistical rigor, measurement quality, and AI-native ways of working.\nInfluence product and executive decisions by translating complex analytical findings into clear, actionable recommendations, aligning cross-functional stakeholders on meaningful metrics, and communicating uncertainty and trade-offs effectively.\nShape the future of AI product measurement by continuously evaluating emerging methodologies, challenging existing assumptions, and prioritizing the analytical questions that matter most as Klaviyo's agents and AI strategy evolve.\nWho You Are\nYou have 10+ years of experience in data science, product analytics, applied statistics, or a related quantitative discipline, including recent hands-on experience measuring AI, machine learning, or LLM-powered products in production.\nYou bring deep expertise in product measurement, experimentation, statistical modeling, and causal inference, with the ability to select and defend rigorous analytical approaches when traditional randomized experiments aren't feasible.\nYou have experience evaluating AI systems in production, including designing evaluation frameworks, analyzing agent interactions or conversations, leveraging human labeling or LLM-as-judge methodologies, and understanding how model, prompt, and tool changes influence product outcomes.\nYou're highly proficient in SQL and Python or R, with experience developing production-quality analytical solutions and familiarity with dbt and modern analytics engineering practices.\nYou take an AI-first approach to your work, actively leveraging coding agents, LLMs, and emerging tools to accelerate analytical workflows while understanding their limitations and ensuring the accuracy and reliability of their outputs.\nYou've demonstrated technical leadership by setting direction on complex analytical initiatives, mentoring other data scientists, reviewing methodologies, and influencing technical standards across teams.\nYou combine statistical rigor with strong product and business intuition, understanding how customer engagement, AI product performance, usage-based pricing, and unit economics translate into meaningful customer and business outcomes.\nYou're comfortable tackling ambiguous, emerging problems where established metrics or methodologies may not yet exist, and can develop practical frameworks that balance analytical rigor with business needs.\nYou have a track record of influencing senior stakeholders, aligning cross-functional teams around shared definitions and priorities, and translating complex findings into clear recommendations for technical and non-technical audiences.\nYou're a compelling communicator who can explain sophisticated analytical concepts, challenge assumptions constructively, and communicate uncertainty and trade-offs with confidence.\nYou thrive in fast-moving product environments, adapt quickly as priorities evolve, and know how to balance analytical depth with the urgency and iteration required to build and improve AI-powered products.\nYou have a degree in Statistics, Mathematics, Computer Science, Economics, Finance, Engineering, or another quantitative discipline, or an MBA with a quantitative emphasis. Equivalent hands-on experience in applied data science is equally valued.\nNice to Have\nExperience measuring and optimizing AI agents, conversational AI, or LLM-powered products in a high-growth B2B SaaS or product-focused technology company.\nHands-on experience with agent observability and evaluation platforms such as Braintrust, LangSmith, or Arize, including working with agent traces, tool-call data, and evaluation pipelines.\nExperience with usage-based or consumption-based pricing, credit models, AI infrastructure costs, or LLM unit economics.\nExperience building governed metrics, semantic layers, metric trees, or standardized measurement frameworks used across multiple product teams.\nExperience applying classification, clustering, or other advanced analytical methods to large volumes of unstructured text, customer conversations, or agent interactions.\nExperience developing internal AI agents or AI-powered analytics tools that improve team productivity and decision-making.\nFamiliarity with marketing attribution, incremental measurement, or evaluating the impact of AI-driven customer engagement.\nBackground in marketing technology, ecommerce, CRM, customer-service software, or other data-driven SaaS products.\nWhy This Role Matters\nAI agents are central to Klaviyo's product strategy and the future of how businesses engage with their customers. As Composer and Customer Agent become increasingly sophisticated, understanding whether these products are delivering meaningful value, how effectively they perform, and where to invest next will be critical to their long-term success.\nIn this role, you'll help establish how Klaviyo measures the effectiveness and business impact of AI agents, developing the frameworks, methodologies, and analytical foundations that inform some of our most important product, pricing, and investment decisions. Your work will directly influence how we improve agent quality, understand customer outcomes, and connect AI innovation to measurable business value.\nThis is an opportunity to combine deep technical ownership with meaningful strategic influence, working alongside a highly collaborative, AI-first data science team to solve emerging measurement challenges and shape how Klaviyo builds, evaluates, and scales its next generation of intelligent products.\nMassachusetts Applicants:\n It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.\nOur salary range reflects the cost of labor across various U.S. geographic markets. The range displayed below reflects the minimum and maximum target salaries for the position across all our US locations. The base salary offered for this position is determined by several factors, including the applicant’s job-related sk...","description_format":"text","description_chars":15022,"description_truncated":true,"requirements":{"experience_years_min":10,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"master","optional":false},"security_clearance":false,"languages":[]},"benefits":["Equity"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Marketing Automation","Email Marketing","Customer Data Platforms (CDP) & Reverse ETL"],"lifecycle":[{"event":"open","at":"2026-10-30T11:49:48Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 55 · green card filings: 24","filings_12m":55,"filings_prev_12m":39,"green_card_filings_12m":24,"median_offered_wage_usd":152500,"route":null,"cap_exempt":false,"checked_at":"2026-10-29T16:27:21+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":6}],"liveness":{"score":63,"band":"ok","label":"Likely open","p_open":1,"p_active":0.632,"p_room":1,"age_days":1,"expected_fill_days":57,"reasons":["conf:0","stale_co","velocity","win:early","comp:brand"],"computed_at":"2026-11-01T06:10:25Z"},"pay":{"stated_usd_annual":258000,"is_top_pay":true},"html_url":"https://alion.io/job/klaviyo-lead-product-data-scientist-agents","json_url":"https://alion.io/job/klaviyo-lead-product-data-scientist-agents.json","meta":{"generated_at":"2026-11-01T17:35:33Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":26936,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-02T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":6845},"rest":"https://alion.io/mcp/rest/get_company?id=6845"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fklaviyo-lead-product-data-scientist-agents"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fklaviyo-lead-product-data-scientist-agents"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fklaviyo-lead-product-data-scientist-agents"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/klaviyo-lead-product-data-scientist-agents\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fklaviyo-lead-product-data-scientist-agents"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3520430}},{"tool":"search_jobs","arguments":{"company":"klaviyo.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3520430},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}