{"id":4003223,"url":"https://alion.io/job/workshop-ventures-technical-product-manager-pedagogy-ai-quality","title":"Technical Product Manager, Pedagogy & AI Quality","company":{"id":384097,"name":"Workshop Ventures","domain":"workshop.dev","url":"https://alion.io/company/work-shop-5","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":true,"listed_via":null,"ats_vendor":"Ashby","truth_index":null},"role":"Product","role_family":"Product","seniority":null,"employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Boston, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":130000,"max":150000,"currency":"USD","period":"year","gross":null,"usd_annual":150000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Google Docs","optional":false},{"name":"LLM","optional":false},{"name":"Arize Phoenix","optional":true},{"name":"Braintrust","optional":true},{"name":"Langfuse","optional":true},{"name":"LangSmith","optional":true}],"status":"live","first_seen_at":"2026-11-10T00:12:09Z","employer_posted_date":"2026-11-10","last_verified_at":"2026-11-18T06:24:32Z","board_verified":true,"closed_at":null,"days_open":8,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":8},"description":"About Medley Learning\nMedley is an instructional support tool for multilingual learners that helps students access grade-level content. Our browser extension layers individualized reading and writing scaffolds directly onto the digital materials schools already use, including curriculum platforms, Google Docs, websites, and PDFs. Supports are tailored to a student's English proficiency level and home language, so students get the help they need in real time while staying engaged with the same rigorous content as their peers. Medley is grounded in research-backed practices and is already being used by over 50 districts across the US.\nThe Opportunity\nWe're hiring a Technical Product Manager, Pedagogy & AI Quality to ensure that Medley’s AI-powered instructional scaffolds continue to meet our high bar for quality and user experience. You'll tighten our rubrics for what high-quality scaffolds look like for reading and writing support in the areas of both multilingual learning and special education, and across various subjects. You’ll expand upon our metrics, build upon our robust evals systems, and review processes that hold our AI to that bar.\nThis is a hybrid role at the intersection of pedagogy and AI product work. You'll write and test prompts, label and analyze model outputs, dig through traces and error reports, and partner closely with engineering to ship quality improvements. You will also learn and leverage the research around scaffolding high-quality instructional materials for multilingual learners at different proficiency levels, as well as accommodations and supports for students with receptive and expressive language disabilities and neurodivergent students.\nYou'll work closely with our head of product and engineers to ensure that Medley consistently serves teachers and students in our partner districts with scaffolds that meet the highest levels of quality. Over time, you’ll own the quality and usability of our scaffolds suite as it continues to grow to support more students with diverse needs, prioritizing product improvements and new features to better support teachers and drive student outcomes.\nWhat You'll Do\nDefine what \"excellent\" looks like. Translate research-based practice in literacy, special education, and multilingual learner instruction into clear, usable quality standards for Medley scaffolds, and set targets for AI output quality.\n\nTrack and monitor quality with a robust evals system. Build and maintain datasets, rubrics, and eval suites and conduct regular error analysis and trace review, combining human labeling, LLM-as-judge and rule-based evals.\n\nRefine prompts and models as needed. Write, test, and refine prompts; run model comparisons and regression tests, making changes as needed to ensure outputs are consistently meeting target metrics and passing quality thresholds.\n\nStay close to our users and represent their voice in product decisions. Regularly review signals from educators, data from support tickets, product analytics, and conduct user research with students and teachers to inform product decisions.\n\nPrioritize and ship AI quality and scaffold-related product improvements. Maintain a prioritized backlog of AI quality and scaffold UX issues; write clear specs, and work with engineering to scope, build and verify fixes, product updates and new features.\n\nWhat We're Looking For\nPedagogical expertise, especially in scaffolding literacy instruction for struggling readers, special education, and/or multilingual learner education, and ideally from classroom teaching, intervention, instructional coaching or curriculum design.\n\nHands-on experience with AI evals, including human labeling, LLM-as-judge rubric design, AI metrics targets definition, and error analysis of LLM outputs.\n\nAI product fluency: you use LLMs daily and have experience designing complex AI systems for products.\n\nExperience defining new features and collaborating with engineers to deliver them: you can clearly communicate a problem and solution, and work through technical tradeoffs and UX questions.\n\nStrong initiative, sharp judgment, and clear writing: you take action when you spot a problem, make swift decisions grounded in data and instincts, and communicate clearly to both technical and non-technical audiences.\n\nNice to Have\nGraduate degree and/or significant direct experience in special education, reading/literacy, ESL/ELD or bilingual education.\n\nFamiliarity with AI eval and observability tools (e.g., LangSmith, Braintrust, Arize, Langfuse) or annotation platforms.\n\nProficiency in a language other than English.\n\nPrior work at an edtech company, or on content quality, assessment design or psychometrics.\n\nFamiliarity with accessibility standards (e.g., WCAG) and assistive technology.\n\nWhy Join Medley\nMedley is building products for multilingual learners and students with disabilities, populations that together represent more than 20% of U.S. K-12 students and have historically been underserved by edtech.\nYou'll join early, work directly with founders and engineering leadership, own meaningful technical and design decisions, and help shape a product that improves access to learning for students nationwide.\nWe are committed to creating a diverse and inclusive work environment. We welcome applicants from all backgrounds and experiences.","description_format":"text","description_chars":5335,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[{"language":"English","level":"Advanced (C1)","optional":false}]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Venture Capital","Angel Networks & Startup Accelerators"],"lifecycle":[{"event":"open","at":"2026-11-10T05:04:15Z"}],"visa":[],"liveness":{"score":81,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.811,"p_room":1,"age_days":8,"expected_fill_days":27,"reasons":["conf:4","win:early"],"computed_at":"2026-11-18T05:48:20Z"},"pay":{"stated_usd_annual":150000,"is_top_pay":false},"html_url":"https://alion.io/job/workshop-ventures-technical-product-manager-pedagogy-ai-quality","json_url":"https://alion.io/job/workshop-ventures-technical-product-manager-pedagogy-ai-quality.json","meta":{"generated_at":"2026-11-18T08:49:12Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":191,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-19T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":384097},"rest":"https://alion.io/mcp/rest/get_company?id=384097"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fworkshop-ventures-technical-product-manager-pedagogy-ai-quality"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fworkshop-ventures-technical-product-manager-pedagogy-ai-quality"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fworkshop-ventures-technical-product-manager-pedagogy-ai-quality"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/workshop-ventures-technical-product-manager-pedagogy-ai-quality\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fworkshop-ventures-technical-product-manager-pedagogy-ai-quality"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4003223}},{"tool":"search_jobs","arguments":{"company":"workshop.dev","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4003223},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}