{"id":4190813,"url":"https://alion.io/job/bask-health-ai-evaluation-engineer","title":"AI Evaluation Engineer","company":{"id":66537,"name":"Bask Health","domain":"bask.health","url":"https://alion.io/company/bask-health","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workable","truth_index":{"grade":"A","score":88,"open_postings":21,"ghost_share":0,"stale_share":0.286,"repost_share":0,"time_to_fill_p50_days":171,"computed_at":"2026-11-14T05:58:10Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":["US"],"hiring_countries_total":1,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"CI/CD","optional":false},{"name":"Claude Code","optional":false},{"name":"Cursor","optional":false},{"name":"Function Calling","optional":false},{"name":"Git","optional":false},{"name":"JavaScript","optional":false},{"name":"Machine Learning","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Structured Outputs","optional":false},{"name":"TypeScript","optional":false},{"name":"AI Agents","optional":true},{"name":"Anthropic","optional":true},{"name":"AWS","optional":true},{"name":"DeepEval","optional":true},{"name":"Embeddings","optional":true},{"name":"HIPAA","optional":true},{"name":"Human-in-the-Loop","optional":true},{"name":"Jest","optional":true},{"name":"LangSmith","optional":true},{"name":"LLM","optional":true},{"name":"Next.js","optional":true},{"name":"Node JS","optional":true},{"name":"OpenAI","optional":true},{"name":"Playwright","optional":true},{"name":"Promptfoo","optional":true},{"name":"Pytest","optional":true},{"name":"Ragas","optional":true},{"name":"React.js","optional":true},{"name":"Semantic Search","optional":true},{"name":"Vitest","optional":true}],"status":"live","first_seen_at":"2026-11-12T19:17:00Z","employer_posted_date":"2026-11-12","last_verified_at":"2026-11-14T11:00:49Z","board_verified":true,"closed_at":null,"days_open":1,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":1},"description":"About Bask Health\nBask Health is a software platform that enables entrepreneurs, doctors, and healthcare businesses to launch, manage, and grow direct-to-consumer telehealth businesses.\nOur platform brings together patient experiences, clinical workflows, pharmacy operations, and commerce tools in one place.\nWe're an entrepreneurial, product-focused engineering team that values ownership, technical excellence, and building software that solves real problems. Engineers work closely with product teams and company leadership, take features from idea to production, and have a direct role in shaping the platform.\nEngineering at Bask is AI-first and builder-led. We use tools like Cursor and Claude Code throughout development while maintaining high standards for reliability, testing, and software quality.\nAbout the Role\nWe're looking for an AI Evaluation Engineer to help ensure that AI-powered features across Bask Health behave accurately, reliably, and consistently before and after they reach production.\nBask Health supports the technology behind direct-to-consumer telehealth businesses, connecting patient experiences, merchant operations, clinical workflows, pharmacy coordination, and commerce capabilities. As we introduce more AI-powered functionality, we need dependable ways to verify that these systems work as intended under realistic conditions.\nYou'll build and maintain evaluation workflows that test large language models, AI assistants, retrieval systems, and automated features against clearly defined product requirements.\nThis is a hands-on engineering role that combines AI evaluation, software testing, automation, and production quality assurance. You'll develop test scenarios, evaluation datasets, automated checks, and tools that help engineering teams identify problems before they affect users.\nYour work may include testing AI-generated responses, validating structured outputs, evaluating tool-calling behavior, checking retrieval accuracy, and identifying regressions caused by changes to prompts, models, or application code.\nYou'll collaborate with applied AI engineers, machine learning engineers, backend engineers, product managers, and quality-focused engineering teams to define expected behavior and translate those expectations into repeatable evaluations.\nYou'll also help establish processes for reviewing AI failures, documenting recurring issues, and determining whether a feature meets its acceptance criteria. You'll work with engineers to reproduce problems and verify that fixes address the underlying issue.\nBecause Bask operates in healthcare-related environments, you'll pay particular attention to privacy, permissions, unsupported medical claims, and AI behavior that could interfere with approved clinical or pharmacy workflows. Evaluation systems should help verify that AI features remain within their intended operational boundaries.\nThis position emphasizes practical testing, thoughtful engineering, and continuous improvement. You'll contribute to evaluation standards while independently implementing and maintaining the tools and tests that support AI quality.\nWe're looking for an engineer who enjoys finding difficult edge cases, understands how AI behavior differs from traditional deterministic software, and wants to help build products that users can depend on.\nWhat You'll Do\nDesign and implement evaluation workflows for AI-powered applications across Bask Health.\nBuild automated tests for large language model outputs and AI application behavior.\nDevelop evaluation datasets that represent realistic user requests and product workflows.\nCreate test scenarios for patient-facing, merchant-facing, and internal AI features.\nImplement checks for response accuracy, relevance, completeness, and consistency.\nEvaluate AI-generated content for unsupported claims and misleading information.\nTest structured model outputs for schema compliance and application compatibility.\nBuild evaluations for AI tool calling and authorized action execution.\nValidate that AI assistants respect application permissions and tenant boundaries.\nTest retrieval-augmented generation systems for grounding and information relevance.\nDevelop regression tests that identify unexpected changes in AI behavior.\nIntegrate evaluation checks into engineering development and release workflows.\nCreate automated comparison tests for prompts, model versions, and configuration changes.\nDevelop scripts and utilities that simplify AI evaluation execution.\nBuild reporting tools that summarize evaluation results and failure patterns.\nInvestigate failed evaluations and help engineers reproduce underlying issues.\nWork with product managers to define expected AI behavior and acceptance criteria.\nCollaborate with applied AI engineers to improve prompts and application logic.\nTest AI applications against unusual inputs, ambiguous requests, and difficult edge cases.\nDevelop evaluation cases for prompt injection and unauthorized tool-use attempts.\nSupport human review workflows for outputs that require subjective assessment.\nTrack recurring AI failures and help prioritize reliability improvements.\nMonitor production evaluation signals and investigate quality regressions.\nDocument test coverage, evaluation procedures, and known limitations.\nImprove evaluation efficiency, repeatability, and maintainability.\nParticipate in code reviews, engineering planning, and technical discussions.\nSupport controlled testing and rollout of new AI-powered features.\nOwn assigned AI evaluation projects from test design through implementation, reporting, and ongoing improvement.\nRequirements\n3+ years of professional experience in software engineering, AI engineering, test automation, quality engineering, or a closely related technical field.\nExperience developing automated tests for production software applications.\nStrong programming skills in Python, TypeScript, JavaScript, or comparable languages.\nFamiliarity with large language models and modern AI-powered applications.\nUnderstanding of software testing methodologies and test case design.\nExperience building automated testing scripts, frameworks, or validation tools.\nFamiliarity with API testing and backend application integrations.\nUnderstanding of prompt engineering and common AI model failure modes.\nAbility to evaluate AI-generated responses against expected requirements.\nExperience working with structured data and validating application outputs.\nFamiliarity with regression testing and software release processes.\nExperience debugging application failures and reproducing technical issues.\nUnderstanding of authentication, authorization, and secure application behavior.\nFamiliarity with databases and application data models.\nExperience working with Git and collaborative software development workflows.\nUnderstanding of continuous integration and automated testing pipelines.\nFamiliarity with logging, monitoring, and production troubleshooting.\nAbility to create clear, reproducible test cases.\nStrong analytical thinking and attention to detail.\nAbility to distinguish application defects from model behavior limitations.\nExperience documenting technical issues and communicating findings.\nFamiliarity with cloud-based application environments.\nAbility to collaborate effectively with engineers, product managers, and other stakeholders.\nStrong problem-solving skills and a quality-focused engineering mindset.\nAbility to independently own evaluation and test automation projects.\nNice to Have\nExperience evaluating large language models or generative AI applications.\nFamiliarity with OpenAI, Anthropic, or comparable model APIs.\nExperience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools.\nFamiliarity with LLM-as-a-judge evaluation techniques.\nExperience developing AI evaluation datasets and test benchmarks.\nUnderstanding of human evaluation and response labeling workflows.\nExperience testing retrieval-augmented generation systems.\nFamiliarity with embeddings, semantic search, and vector databases.\nExperience evaluating AI agents and tool-calling behavior.\nFamiliarity with prompt injection testing and AI security evaluations.\nExperience implementing AI regression tests in CI/CD pipelines.\nExperience building automated evaluation reports or dashboards.\nFamiliarity with Playwright, Jest, Vitest, Pytest, or comparable testing frameworks.\nExperience testing TypeScript or Node.js applications.\nFamiliarity with React or Next.js application testing.\nExperience working with AWS or serverless infrastructure.\nExperience evaluating AI features in healthcare or other sensitive environments.\nUnderstanding of HIPAA and healthcare data privacy considerations.\nExperience developing human-in-the-loop quality review processes.\nExperience using AI-assisted development tools such as Cursor or Claude Code.","description_format":"text","description_chars":8850,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Health Care","Telemedicine & Virtual Care","Pharmacies"],"lifecycle":[{"event":"open","at":"2026-11-12T19:17:00Z"}],"visa":[{"country":"US","licensed_sponsor":true,"adverts_offering_sponsorship_24m":0,"sanctioned_now":false,"evidence":"H-1B filings in 12 months: 3","filings_12m":3,"filings_prev_12m":2,"green_card_filings_12m":0,"median_offered_wage_usd":94167,"route":null,"cap_exempt":false,"checked_at":"2026-11-14T07:33:33+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":0}],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":1,"expected_fill_days":171,"reasons":["conf:2","velocity","win:early","comp:brand"],"computed_at":"2026-11-14T05:58:10Z"},"pay":null,"html_url":"https://alion.io/job/bask-health-ai-evaluation-engineer","json_url":"https://alion.io/job/bask-health-ai-evaluation-engineer.json","meta":{"generated_at":"2026-11-14T14:43:24Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":15777,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-15T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":66537},"rest":"https://alion.io/mcp/rest/get_company?id=66537"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fbask-health-ai-evaluation-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fbask-health-ai-evaluation-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fbask-health-ai-evaluation-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/bask-health-ai-evaluation-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fbask-health-ai-evaluation-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4190813}},{"tool":"search_jobs","arguments":{"company":"bask.health","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4190813},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}