{"id":4190834,"url":"https://alion.io/job/bask-health-software-engineer-ai-quality-evaluation","title":"Software Engineer, AI Quality & Evaluation","company":{"id":66537,"name":"Bask Health","domain":"bask.health","url":"https://alion.io/company/bask-health","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workable","truth_index":{"grade":"A","score":88,"open_postings":21,"ghost_share":0,"stale_share":0.286,"repost_share":0,"time_to_fill_p50_days":171,"computed_at":"2026-11-14T05:58:10Z"}},"role":"Backend","role_family":"Backend","seniority":null,"employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":["US"],"hiring_countries_total":1,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"CI/CD","optional":false},{"name":"Claude Code","optional":false},{"name":"Cursor","optional":false},{"name":"Git","optional":false},{"name":"JavaScript","optional":false},{"name":"Machine Learning","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"TypeScript","optional":false},{"name":"Amazon EventBridge","optional":true},{"name":"Anthropic","optional":true},{"name":"AWS","optional":true},{"name":"AWS Lambda","optional":true},{"name":"DeepEval","optional":true},{"name":"Embeddings","optional":true},{"name":"HIPAA","optional":true},{"name":"LangSmith","optional":true},{"name":"LLM","optional":true},{"name":"Next.js","optional":true},{"name":"Node JS","optional":true},{"name":"OpenAI","optional":true},{"name":"Prisma","optional":true},{"name":"Promptfoo","optional":true},{"name":"Ragas","optional":true},{"name":"React.js","optional":true},{"name":"tRPC","optional":true}],"status":"live","first_seen_at":"2026-11-12T19:17:00Z","employer_posted_date":"2026-11-12","last_verified_at":"2026-11-14T11:00:49Z","board_verified":true,"closed_at":null,"days_open":1,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":1},"description":"About Bask Health\nBask Health is a software platform that enables entrepreneurs, doctors, and healthcare businesses to launch, manage, and grow direct-to-consumer telehealth businesses.\nOur platform brings together patient experiences, clinical workflows, pharmacy operations, and commerce tools in one place.\nWe're an entrepreneurial, product-focused engineering team that values ownership, technical excellence, and building software that solves real problems. Engineers work closely with product teams and company leadership, take features from idea to production, and have a direct role in shaping the platform.\nEngineering at Bask is AI-first and builder-led. We use tools like Cursor and Claude Code throughout development while maintaining high standards for reliability, testing, and software quality.\nAbout the Role\nWe're looking for a Software Engineer, AI Quality & Evaluation to build the engineering systems that help Bask Health test, evaluate, and improve AI-powered applications throughout their development and production lifecycles.\nBask Health supports direct-to-consumer telehealth businesses through a platform that connects patient experiences, merchant operations, clinical workflows, pharmacy coordination, and commerce tools. As we introduce more AI-powered capabilities across these areas, we need reliable software infrastructure that makes it easy for engineering teams to understand how those systems behave.\nYou'll help develop the internal tools, backend services, testing frameworks, and automation that support AI quality assurance across the platform.\nThis is a hands-on software engineering role focused on building reusable systems rather than evaluating individual models in isolation. You'll work with APIs, databases, automated testing infrastructure, model integrations, and developer tools to make AI evaluation more efficient, consistent, and accessible.\nYour work may include evaluation execution services, test dataset management tools, automated regression pipelines, result comparison systems, and interfaces that help engineers investigate AI failures.\nYou'll collaborate with applied AI engineers, machine learning engineers, AI evaluation specialists, and backend engineers to understand their testing needs and translate those requirements into dependable software capabilities.\nYou'll also help integrate AI evaluation into normal engineering workflows. This includes making it easier to run tests during development, compare model or prompt changes, inspect failures, and identify regressions before production releases.\nBecause Bask operates in healthcare-related environments, you'll help ensure evaluation systems handle sensitive information appropriately, respect access permissions, and support testing for privacy, safety, and approved operational boundaries.\nThis position emphasizes strong software engineering fundamentals, developer experience, and production reliability. You'll contribute to technical architecture, write maintainable code, improve infrastructure performance, and own services that other engineering teams rely on.\nWe're looking for an engineer who enjoys building tools for other developers, understands the challenges of testing nondeterministic AI systems, and wants to create reliable infrastructure that improves software quality across the organization.\nWhat You'll Do\nDesign, build, and maintain software infrastructure that supports AI quality and evaluation across Bask Health.\nDevelop backend services that execute and manage automated AI evaluations.\nBuild reusable testing tools for large language model applications.\nCreate APIs that allow engineering teams to submit evaluation jobs and retrieve results.\nDevelop systems for managing evaluation datasets, test cases, and expected outcomes.\nBuild automated workflows for comparing prompts, models, and application configurations.\nImplement regression testing infrastructure for AI-powered features.\nIntegrate AI evaluation checks into continuous integration and deployment pipelines.\nDevelop services that record and organize evaluation results over time.\nBuild tools that help engineers inspect AI responses, failures, and test histories.\nImplement structured result storage and comparison capabilities.\nDevelop internal dashboards or interfaces for reviewing evaluation outcomes.\nCreate reusable software libraries that simplify AI testing across multiple applications.\nBuild systems that support both deterministic checks and model-based evaluations.\nImplement automated testing for tool-calling behavior and structured model outputs.\nDevelop infrastructure for evaluating retrieval-augmented generation applications.\nBuild asynchronous processing systems for large evaluation workloads.\nImplement job scheduling, retries, timeouts, and error handling for evaluation services.\nDevelop monitoring and logging for evaluation infrastructure.\nOptimize evaluation execution speed, reliability, and infrastructure costs.\nWork with AI evaluation engineers to translate testing requirements into reusable software tools.\nCollaborate with machine learning engineers to support model and prompt comparison workflows.\nImplement appropriate access controls and secure handling of evaluation data.\nEnsure evaluation infrastructure respects tenant boundaries and data privacy requirements.\nInvestigate infrastructure failures and resolve issues affecting evaluation reliability.\nImprove developer documentation and internal tooling for AI testing.\nParticipate in code reviews, technical planning, and architecture discussions.\nOwn assigned AI quality engineering projects from design through deployment, monitoring, and continuous improvement.\nRequirements\n3+ years of professional software engineering experience, preferably in backend, platform, infrastructure, or developer tooling roles.\nExperience building and maintaining production backend services.\nStrong programming skills in TypeScript, JavaScript, Python, or comparable languages.\nExperience designing and developing APIs and service integrations.\nFamiliarity with automated testing frameworks and software quality practices.\nUnderstanding of continuous integration and deployment workflows.\nExperience building reusable software components or internal engineering tools.\nFamiliarity with large language models and modern AI application development.\nUnderstanding of the challenges involved in testing nondeterministic AI systems.\nExperience working with relational databases and structured application data.\nFamiliarity with asynchronous processing and background job systems.\nUnderstanding of distributed systems fundamentals and service reliability.\nExperience developing automated testing pipelines.\nFamiliarity with cloud-based application deployment.\nUnderstanding of authentication, authorization, and secure service communication.\nExperience implementing logging, monitoring, and production diagnostics.\nAbility to troubleshoot backend services and infrastructure issues.\nFamiliarity with configuration management and software release processes.\nExperience writing maintainable, well-tested production code.\nUnderstanding of API error handling, retries, and failure recovery.\nExperience working with Git and collaborative engineering workflows.\nAbility to translate technical requirements into reusable engineering systems.\nStrong analytical thinking and problem-solving skills.\nAbility to collaborate effectively with software engineers, AI engineers, and machine learning specialists.\nAbility to independently own engineering projects from technical design through deployment.\nNice to Have\nExperience building infrastructure for AI evaluation or machine learning testing.\nFamiliarity with OpenAI, Anthropic, or comparable model APIs.\nExperience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools.\nFamiliarity with LLM-as-a-judge evaluation approaches.\nExperience building evaluation execution services or automated benchmark systems.\nFamiliarity with AI regression testing and prompt comparison workflows.\nExperience developing internal developer platforms or SDKs.\nExperience building testing infrastructure for distributed applications.\nFamiliarity with TypeScript and Node.js backend development.\nExperience with React or Next.js for internal tooling interfaces.\nFamiliarity with tRPC, Prisma, Drizzle, or comparable development tools.\nExperience working with AWS Lambda, SQS, EventBridge, or related cloud services.\nFamiliarity with serverless infrastructure and infrastructure-as-code tools.\nExperience with SST or comparable application deployment frameworks.\nExperience building scalable asynchronous job processing systems.\nFamiliarity with embeddings, vector databases, and retrieval-augmented generation.\nExperience implementing model observability or AI request tracing.\nUnderstanding of multi-tenant SaaS architecture and data isolation.\nFamiliarity with HIPAA and healthcare data privacy considerations.\nExperience using AI-assisted development tools such as Cursor or Claude Code.","description_format":"text","description_chars":9024,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Health Care","Telemedicine & Virtual Care","Pharmacies"],"lifecycle":[{"event":"open","at":"2026-11-12T19:17:00Z"}],"visa":[{"country":"US","licensed_sponsor":true,"adverts_offering_sponsorship_24m":0,"sanctioned_now":false,"evidence":"H-1B filings in 12 months: 3","filings_12m":3,"filings_prev_12m":2,"green_card_filings_12m":0,"median_offered_wage_usd":94167,"route":null,"cap_exempt":false,"checked_at":"2026-11-14T07:33:33+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":1}],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":1,"expected_fill_days":171,"reasons":["conf:2","velocity","win:early","comp:brand"],"computed_at":"2026-11-14T05:58:10Z"},"pay":null,"html_url":"https://alion.io/job/bask-health-software-engineer-ai-quality-evaluation","json_url":"https://alion.io/job/bask-health-software-engineer-ai-quality-evaluation.json","meta":{"generated_at":"2026-11-14T15:56:02Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":17883,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-15T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":66537},"rest":"https://alion.io/mcp/rest/get_company?id=66537"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fbask-health-software-engineer-ai-quality-evaluation"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fbask-health-software-engineer-ai-quality-evaluation"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fbask-health-software-engineer-ai-quality-evaluation"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/bask-health-software-engineer-ai-quality-evaluation\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fbask-health-software-engineer-ai-quality-evaluation"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4190834}},{"tool":"search_jobs","arguments":{"company":"bask.health","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4190834},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}