{"id":4190822,"url":"https://alion.io/job/bask-health-machine-learning-engineer-evaluation-reliability","title":"Machine Learning Engineer, Evaluation & Reliability","company":{"id":66537,"name":"Bask Health","domain":"bask.health","url":"https://alion.io/company/bask-health","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workable","truth_index":{"grade":"A","score":88,"open_postings":21,"ghost_share":0,"stale_share":0.286,"repost_share":0,"time_to_fill_p50_days":171,"computed_at":"2026-11-14T05:58:10Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":["US"],"hiring_countries_total":1,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"CI/CD","optional":false},{"name":"Claude Code","optional":false},{"name":"Cursor","optional":false},{"name":"Git","optional":false},{"name":"Hallucination","optional":false},{"name":"Machine Learning","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"TypeScript","optional":false},{"name":"AI Agents","optional":true},{"name":"Anthropic","optional":true},{"name":"AWS","optional":true},{"name":"DeepEval","optional":true},{"name":"Embeddings","optional":true},{"name":"HIPAA","optional":true},{"name":"JavaScript","optional":true},{"name":"LangSmith","optional":true},{"name":"LLM","optional":true},{"name":"Node JS","optional":true},{"name":"OpenAI","optional":true},{"name":"Ragas","optional":true},{"name":"Semantic Search","optional":true}],"status":"live","first_seen_at":"2026-11-12T19:17:00Z","employer_posted_date":"2026-11-12","last_verified_at":"2026-11-14T11:00:49Z","board_verified":true,"closed_at":null,"days_open":1,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":1},"description":"About Bask Health\nBask Health is a software platform that enables entrepreneurs, doctors, and healthcare businesses to launch, manage, and grow direct-to-consumer telehealth businesses.\nOur platform brings together patient experiences, clinical workflows, pharmacy operations, and commerce tools in one place.\nWe're an entrepreneurial, product-focused engineering team that values ownership, technical excellence, and building software that solves real problems. Engineers work closely with product teams and company leadership, take features from idea to production, and have a direct role in shaping the platform.\nEngineering at Bask is AI-first and builder-led. We use tools like Cursor and Claude Code throughout development while maintaining high standards for reliability, testing, and software quality.\nAbout the Role\nWe're looking for a Machine Learning Engineer, Evaluation & Reliability to help ensure that AI-powered features across Bask Health are accurate, dependable, and ready for real-world use.\nBask Health uses software to connect patient experiences, merchant operations, clinical workflows, pharmacy coordination, and commerce capabilities. As AI becomes more integrated into these systems, we need reliable ways to measure model performance, identify failures, and verify that changes improve the user experience without introducing new risks.\nYou'll help develop the evaluation infrastructure, testing workflows, and performance measurements used to assess large language models, retrieval systems, AI assistants, and automated workflows.\nThis is a hands-on engineering role that combines machine learning evaluation, software development, data analysis, and production quality assurance. You'll work with model outputs, evaluation datasets, automated testing tools, and application telemetry to understand how AI systems perform under realistic conditions.\nYour work may include creating evaluation datasets, implementing automated benchmarks, measuring response quality, testing retrieval accuracy, analyzing model failures, and monitoring production behavior.\nYou'll collaborate with applied AI engineers, machine learning engineers, backend engineers, product managers, and other stakeholders to define what successful AI behavior looks like for different workflows. You'll help translate those expectations into measurable tests and actionable engineering improvements.\nBecause Bask operates in healthcare-related environments, AI evaluation must account for accuracy, privacy, appropriate permissions, and the consequences of unreliable responses. You'll help identify situations where AI systems produce unsupported information, expose restricted data, attempt unauthorized actions, or operate outside approved clinical and operational boundaries.\nThis position emphasizes practical implementation and continuous improvement. You'll contribute to evaluation methodologies and engineering standards while working with senior engineers on broader reliability strategies and more complex technical decisions.\nWe're looking for an engineer who enjoys investigating how AI systems behave, can turn ambiguous quality problems into measurable tests, and understands that reliable AI products require ongoing evaluation-not just successful demonstrations.\nWhat You'll Do\nBuild and maintain evaluation systems for AI-powered applications across Bask Health.\nDevelop automated testing frameworks for large language models and AI application workflows.\nCreate and maintain representative datasets for evaluating model behavior.\nImplement benchmarks that measure response accuracy, relevance, completeness, and consistency.\nDevelop evaluation pipelines for prompts, retrieval systems, and model configurations.\nTest AI assistants and agents against realistic user scenarios and workflow requirements.\nBuild automated regression tests to detect performance changes between releases.\nMeasure hallucination rates, unsupported claims, and other model quality failures.\nEvaluate retrieval-augmented generation systems for document relevance and answer grounding.\nDevelop tests for structured model outputs and schema compliance.\nEvaluate tool-calling accuracy and agent action selection.\nTest AI workflows for unauthorized actions, incorrect permissions, and unsafe behavior.\nImplement evaluation checks for sensitive patient and operational information.\nBuild dashboards and reports that communicate AI quality metrics to engineering teams.\nAnalyze evaluation results and identify recurring failure patterns.\nInvestigate production incidents involving inaccurate or unreliable AI outputs.\nCollaborate with applied AI engineers to improve prompts, retrieval logic, and application behavior.\nSupport model comparisons and controlled experiments.\nEvaluate tradeoffs between model quality, latency, reliability, and inference cost.\nDevelop automated quality checks for CI/CD and AI feature release workflows.\nImplement monitoring and alerting for production AI performance.\nBuild tools for reviewing, labeling, and analyzing model responses.\nHelp maintain evaluation datasets as products and user behavior evolve.\nDocument evaluation methodologies, test cases, and known model limitations.\nWork with product teams to define measurable acceptance criteria for AI features.\nCollaborate with senior engineers on evaluation standards and reliability improvements.\nParticipate in code reviews, technical discussions, and engineering planning.\nOwn assigned evaluation and reliability projects from implementation through deployment and ongoing improvement.\nRequirements\n3+ years of professional experience in machine learning engineering, software engineering, applied AI, or a closely related technical field.\nExperience building and maintaining production software or machine learning applications.\nStrong programming skills in Python, TypeScript, or comparable languages.\nFamiliarity with large language models and modern generative AI applications.\nExperience testing or evaluating machine learning models or AI-powered software.\nUnderstanding of machine learning evaluation metrics and experimental methodologies.\nExperience developing automated tests and validation pipelines.\nFamiliarity with prompt engineering and model output analysis.\nAbility to analyze structured and unstructured datasets.\nExperience writing scripts or services for data processing and evaluation.\nUnderstanding of model accuracy, precision, recall, and other performance measures where applicable.\nFamiliarity with API integrations and backend application development.\nExperience working with relational databases or analytical data systems.\nAbility to identify patterns in model failures and translate findings into engineering improvements.\nUnderstanding of regression testing and software quality assurance.\nFamiliarity with logging, monitoring, and production troubleshooting.\nExperience using Git and collaborative software development workflows.\nUnderstanding of the limitations and failure modes of large language models.\nAbility to design representative test cases and evaluation scenarios.\nFamiliarity with cloud-based development and deployment environments.\nExperience writing maintainable, well-tested production code.\nStrong analytical thinking and attention to detail.\nAbility to communicate technical evaluation results clearly.\nExperience collaborating with engineers, product managers, and cross-functional teams.\nAbility to independently own assigned technical projects.\nNice to Have\nExperience building evaluation frameworks for large language models.\nFamiliarity with OpenAI, Anthropic, or comparable model APIs.\nExperience with tools such as LangSmith, DeepEval, Ragas, or comparable AI evaluation frameworks.\nFamiliarity with LLM-as-a-judge evaluation techniques.\nExperience creating human-labeled evaluation datasets.\nUnderstanding of inter-rater agreement and evaluation consistency.\nExperience evaluating retrieval-augmented generation systems.\nFamiliarity with embeddings, semantic search, and vector databases.\nExperience evaluating AI agents and tool-calling workflows.\nFamiliarity with prompt injection testing and AI security evaluations.\nExperience implementing automated regression testing for AI applications.\nFamiliarity with experiment tracking and model comparison tools.\nExperience building dashboards for machine learning performance monitoring.\nUnderstanding of statistical analysis and experiment design.\nExperience working with Python data analysis libraries.\nFamiliarity with TypeScript and Node.js application development.\nExperience working with AWS or serverless infrastructure.\nExperience evaluating AI systems used in healthcare or other sensitive environments.\nUnderstanding of HIPAA and healthcare data privacy considerations.\nExperience using AI-assisted development tools such as Cursor or Claude Code.","description_format":"text","description_chars":8837,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Health Care","Telemedicine & Virtual Care","Pharmacies"],"lifecycle":[{"event":"open","at":"2026-11-12T19:17:00Z"}],"visa":[{"country":"US","licensed_sponsor":true,"adverts_offering_sponsorship_24m":0,"sanctioned_now":false,"evidence":"H-1B filings in 12 months: 3","filings_12m":3,"filings_prev_12m":2,"green_card_filings_12m":0,"median_offered_wage_usd":94167,"route":null,"cap_exempt":false,"checked_at":"2026-11-14T07:33:33+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":0}],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":1,"expected_fill_days":171,"reasons":["conf:2","velocity","win:early","comp:brand"],"computed_at":"2026-11-14T05:58:10Z"},"pay":null,"html_url":"https://alion.io/job/bask-health-machine-learning-engineer-evaluation-reliability","json_url":"https://alion.io/job/bask-health-machine-learning-engineer-evaluation-reliability.json","meta":{"generated_at":"2026-11-14T14:28:28Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":15355,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-15T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":66537},"rest":"https://alion.io/mcp/rest/get_company?id=66537"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fbask-health-machine-learning-engineer-evaluation-reliability"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fbask-health-machine-learning-engineer-evaluation-reliability"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fbask-health-machine-learning-engineer-evaluation-reliability"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/bask-health-machine-learning-engineer-evaluation-reliability\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fbask-health-machine-learning-engineer-evaluation-reliability"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4190822}},{"tool":"search_jobs","arguments":{"company":"bask.health","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4190822},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}