{"id":4146082,"url":"https://alion.io/job/bask-health-senior-machine-learning-engineer-evaluation-reliability","title":"Senior Machine Learning Engineer, Evaluation & Reliability","company":{"id":66537,"name":"Bask Health","domain":"bask.health","url":"https://alion.io/company/bask-health","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workable","truth_index":{"grade":"A","score":91,"open_postings":21,"ghost_share":0,"stale_share":0.143,"repost_share":0,"time_to_fill_p50_days":171,"computed_at":"2026-11-11T05:49:22Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":["US"],"hiring_countries_total":1,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"CI/CD","optional":false},{"name":"Claude Code","optional":false},{"name":"Cursor","optional":false},{"name":"Function Calling","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Structured Outputs","optional":false},{"name":"Anthropic","optional":true},{"name":"AWS","optional":true},{"name":"DeepEval","optional":true},{"name":"Embeddings","optional":true},{"name":"Hallucination","optional":true},{"name":"HIPAA","optional":true},{"name":"Human-in-the-Loop","optional":true},{"name":"LangSmith","optional":true},{"name":"MLFlow","optional":true},{"name":"OpenAI","optional":true},{"name":"Ragas","optional":true},{"name":"Semantic Search","optional":true},{"name":"Weights & Biases","optional":true}],"status":"live","first_seen_at":"2026-11-12T07:11:28Z","employer_posted_date":"2026-11-12","last_verified_at":"2026-11-12T07:11:28Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"About Bask Health\nBask Health is a software platform that enables entrepreneurs, doctors, and healthcare businesses to launch, manage, and grow direct-to-consumer telehealth businesses.\nOur platform brings together patient experiences, clinical workflows, pharmacy operations, and commerce tools in one place.\nWe're an entrepreneurial, product-focused engineering team that values ownership, technical excellence, and building software that solves real problems. Engineers work closely with product teams and company leadership, take features from idea to production, and have a direct role in shaping the platform.\nEngineering at Bask is AI-first and builder-led. We use tools like Cursor and Claude Code throughout development while maintaining high standards for reliability, testing, and software quality.\nAbout the Role\nWe're looking for a Senior Machine Learning Engineer, Evaluation & Reliability to design, build, and maintain the systems that measure and improve the quality of Bask Health's AI-powered applications.\nAs Bask introduces more AI capabilities across patient experiences, merchant tools, clinical workflows, and internal operations, we need reliable ways to understand how these systems perform. You'll help establish the engineering infrastructure, evaluation methods, and monitoring practices that allow teams to ship AI features with confidence.\nThis role sits at the intersection of machine learning engineering, software infrastructure, automated testing, and production reliability. You'll work on evaluation pipelines, benchmark datasets, regression testing, model monitoring, and tools that help engineers identify failures before they affect users.\nYou'll develop systems that measure more than whether an AI model produces a response. You'll help determine whether that response is accurate, relevant, grounded in appropriate information, consistent with application requirements, and safe for its intended use.\nYou'll also investigate production issues, identify patterns in model failures, and collaborate with applied AI engineers to improve prompts, retrieval systems, model configurations, and application behavior.\nBecause Bask operates in healthcare-related workflows, reliability is especially important. You'll help establish safeguards for sensitive use cases, support human review where appropriate, and make sure AI systems are evaluated against clear quality expectations before deployment.\nThis is a hands-on senior engineering position with ownership over evaluation infrastructure and reliability practices. You'll contribute production code, design scalable testing systems, and help establish standards that other engineers can use across multiple AI applications.\nWe're looking for someone who enjoys turning complex machine learning behavior into measurable engineering systems and who understands that successful AI products depend on consistent performance, not just impressive demonstrations.\nWhat You'll Do\nDesign and build evaluation infrastructure for AI and machine learning applications across Bask Health.\nDevelop automated testing systems for large language model applications and intelligent workflows.\nEstablish measurable quality standards for production AI features.\nCreate evaluation pipelines that assess accuracy, relevance, completeness, consistency, and reliability.\nBuild and maintain representative datasets for testing AI application behavior.\nDevelop regression testing systems to detect quality changes when models, prompts, or application logic are updated.\nDesign evaluation methods for retrieval-augmented generation and knowledge-grounded AI systems.\nMeasure retrieval quality, grounding accuracy, and unsupported model outputs.\nBuild tools for identifying hallucinations, incorrect reasoning, and unreliable responses.\nDevelop testing approaches for AI agents, tool calling, and multi-step workflows.\nEvaluate whether AI systems follow application instructions and respect workflow boundaries.\nImplement automated checks for structured outputs, API interactions, and tool execution.\nBuild monitoring systems that track AI performance in production.\nDevelop dashboards and reporting tools for model quality, latency, errors, and operational reliability.\nInvestigate production AI failures and identify their underlying technical causes.\nCollaborate with applied AI engineers to improve prompts, retrieval pipelines, and model configurations.\nEstablish testing procedures for model upgrades and provider changes.\nDevelop benchmarking tools to compare models based on quality, performance, and cost.\nImplement evaluation processes for sensitive healthcare-related AI workflows.\nWork with product and clinical stakeholders to define appropriate acceptance criteria and escalation requirements.\nSupport human review and feedback collection for AI outputs that require additional validation.\nDesign reproducible experiments for evaluating AI system improvements.\nBuild reusable evaluation libraries, services, and internal engineering tools.\nIntegrate AI evaluation checks into development and deployment workflows.\nImprove observability through structured logging, tracing, and production diagnostics.\nMentor engineers on AI testing methodologies and reliability practices.\nContribute to architecture discussions involving AI infrastructure, testing, and monitoring.\nOwn evaluation and reliability improvements from technical design through deployment and ongoing maintenance.\nRequirements\n5+ years of professional experience in machine learning engineering, software engineering, or a closely related technical discipline.\nDemonstrated experience developing and operating machine learning or AI systems in production.\nStrong programming skills in Python and experience building production software services.\nExperience designing automated testing frameworks or evaluation pipelines for AI or machine learning applications.\nStrong understanding of machine learning evaluation methodologies and performance measurement.\nExperience developing reproducible experiments and benchmark datasets.\nFamiliarity with large language models and modern LLM application architectures.\nUnderstanding of prompt engineering, model inference, and AI application integration patterns.\nExperience evaluating model outputs for quality, accuracy, and reliability.\nFamiliarity with retrieval-augmented generation and retrieval evaluation methods.\nExperience working with structured datasets, data pipelines, and analytical workflows.\nStrong understanding of software testing, regression testing, and continuous integration.\nExperience building or maintaining backend services and APIs.\nFamiliarity with cloud infrastructure and production deployment environments.\nExperience with monitoring, logging, tracing, and production troubleshooting.\nUnderstanding of model latency, throughput, and operational performance metrics.\nAbility to investigate complex system failures and identify root causes.\nExperience designing metrics and reporting systems for technical performance.\nFamiliarity with version control, code review, and collaborative software development.\nUnderstanding of data quality and its impact on model evaluation.\nAbility to evaluate technical tradeoffs between quality, performance, reliability, and cost.\nStrong analytical skills and attention to detail.\nExperience collaborating with engineers, product managers, and technical stakeholders.\nAbility to independently own complex engineering initiatives.\nStrong communication skills and the ability to explain evaluation results clearly.\nNice to Have\nExperience building evaluation frameworks specifically for LLM-powered applications.\nFamiliarity with tools such as LangSmith, DeepEval, Ragas, or comparable evaluation platforms.\nExperience implementing LLM-as-a-judge evaluation methods and validating their limitations.\nExperience building automated hallucination detection or groundedness evaluation systems.\nFamiliarity with agent evaluation and multi-step workflow testing.\nExperience evaluating tool-calling accuracy and structured model outputs.\nExperience with retrieval metrics such as precision, recall, and ranking quality.\nFamiliarity with vector databases, embeddings, and semantic search.\nExperience with MLflow, Weights & Biases, or similar experiment tracking tools.\nExperience developing machine learning observability infrastructure.\nFamiliarity with model drift detection and data quality monitoring.\nExperience integrating evaluation checks into CI/CD pipelines.\nExperience with distributed processing or large-scale evaluation workloads.\nFamiliarity with AWS and cloud-based machine learning infrastructure.\nExperience building internal developer tools for AI engineering teams.\nExperience evaluating systems built with OpenAI, Anthropic, or comparable model providers.\nFamiliarity with healthcare, telehealth, or other sensitive-data applications.\nUnderstanding of HIPAA and healthcare data privacy considerations.\nExperience designing human-in-the-loop evaluation or review systems.\nExperience using AI-assisted development tools such as Cursor or Claude Code.","description_format":"text","description_chars":9101,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Health Care","Telemedicine & Virtual Care","Pharmacies"],"lifecycle":[{"event":"open","at":"2026-11-12T07:11:28Z"}],"visa":[{"country":"US","licensed_sponsor":true,"adverts_offering_sponsorship_24m":0,"sanctioned_now":false,"evidence":"H-1B filings in 12 months: 3","filings_12m":3,"filings_prev_12m":2,"green_card_filings_12m":0,"median_offered_wage_usd":94167,"route":null,"cap_exempt":false,"checked_at":"2026-11-12T07:36:15+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":0}],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":171,"reasons":["conf:3","win:early","comp:brand"],"computed_at":"2026-11-12T10:45:08Z"},"pay":null,"html_url":"https://alion.io/job/bask-health-senior-machine-learning-engineer-evaluation-reliability","json_url":"https://alion.io/job/bask-health-senior-machine-learning-engineer-evaluation-reliability.json","meta":{"generated_at":"2026-11-12T10:45:08Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":22050,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-13T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":66537},"rest":"https://alion.io/mcp/rest/get_company?id=66537"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fbask-health-senior-machine-learning-engineer-evaluation-reliability"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fbask-health-senior-machine-learning-engineer-evaluation-reliability"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fbask-health-senior-machine-learning-engineer-evaluation-reliability"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/bask-health-senior-machine-learning-engineer-evaluation-reliability\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fbask-health-senior-machine-learning-engineer-evaluation-reliability"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4146082}},{"tool":"search_jobs","arguments":{"company":"bask.health","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4146082},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}