{"id":1558371,"url":"https://alion.io/job/geisinger-director-ai-evaluation","title":"Director AI Evaluation","company":{"id":1774033,"name":"Geisinger","domain":"geisinger.org","url":"https://alion.io/company/geisinger","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":{"grade":"B","score":80,"open_postings":164,"ghost_share":0,"stale_share":0.988,"repost_share":0,"time_to_fill_p50_days":20,"computed_at":"2026-10-12T06:55:09Z"}},"role":"Leadership","role_family":"Leadership","seniority":"head","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":168000,"max_usd":316000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":1670},"experience_years_min":8,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Python","optional":false},{"name":"SQL","optional":false}],"status":"closed","first_seen_at":"2026-07-16T00:00:00Z","employer_posted_date":"2026-07-16","last_verified_at":"2026-10-03T08:32:19Z","board_verified":false,"closed_at":"2026-10-03T08:32:19Z","days_open":79,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":79},"description":"Location:\nWork from home (Pennsylvania)Shift:\nDays (United States of America)Scheduled Weekly Hours:\n40Worker Type:\nRegularExemption Status:\nYesJob Summary:\nThe Director of AI Evaluation owns how Geisinger defines, proves, and sustains quality across its entire AI portfolio - internally built models and vendor-provided systems alike. This is a hands-on technical leader who also manages the people who do the building and the proving: the data scientists who develop production machine learning and fine-tuned AI systems, and the senior analysts who evaluate them.Every high-value AI initiative - bought or built - needs a single, credible standard for what constitutes quality, who validates it, and how it stays good in production. The Director sets that standard, leads the team that enforces it, and reports findings to the VP of AI, executive leaders, and the board.\nThis role is a manager who develops a multidisciplinary team, a technical authority who defines evaluation method across the enterprise, and a quality owner who guides every major AI program toward evidence that withstands scrutiny.\nJob Duties:\nReports to the VP of AI; directly career-manages the data science line and matrix-manages the Senior Analysts, AI Evaluation.\nDetermines the quality standard for any high-value AI initiative at Geisinger - internally built or vendor-provided - from design through production.\nHolds bought systems to the same standard as built ones, generating local evidence on whether a tool works here for Geisinger's clinicians and patients rather than accepting vendor aggregate or cherry-picked results.\nOwns the methodology that holds initiatives to that standard: pre-production validation and live production monitoring.\nOwns the health of the data science team- attracting and retaining strong technical talent, developing careers, and keeping the bench deep, engaged, and growing - and leads and develops the evaluation team alongside it. \nProvides hands-on technical guidance to program teams as they design validation studies, equity audits, monitoring plans, and escalation playbooks.\nOwns the evaluation toolkit and reusable playbooks and templates that let each new program move faster than the last.\nTranslates program-specific failure modes into concrete, measurable production-monitoring metrics; defines what is measured and how, while the AI Platform team builds the backend.\nTracks AI System Performance - the single most important accuracy indicator for each system, against thresholds set to clinical tolerance.\nTracks User Adoption - engagement, override rates, and time-to-action - distinguishing genuine workflow misalignment and alarm fatigue from poor predictive value.\nConnects each AI to the Outcome it was deployed to improve (mortality, time-to-treatment, boarding time, denial rate, cost per case) against a pre-launch baseline over a use-case-appropriate horizon, holding both tangible returns and harder-to-quantify value in view.\nMonitors Equity - the maximum performance gap on the Pillar 1 metric across the subgroups that matter for the initiative, so disparate impact surfaces early.\nWork is typically performed in an office environment. Accountable for satisfying all job specific obligations and complying with all organization policies and procedures. The specific statements in this profile are not intended to be all-inclusive. They represent typical elements considered necessary to successfully perform the job.\nPosition Details:\nRequired Skills and Qualifications:\nPeople-leadership experience - managing, developing, and growing technical staff; building teams, not just leading projects.\nStrong foundation in experimental design and causal inference, with judgment about which method fits which situation.\nHands-on experience designing and running model evaluation studies in real production settings.\nExperience evaluating LLM or generative AI systems, or comparable experience with complex ML systems where ground truth is ambiguous or noisy.\nProven ability to translate ambiguous failure modes into concrete, defensible evaluation designs and monitoring metrics.\nStrong fluency in Python and SQL; working comfort with modern ML tooling and cloud-native data environments.\nExperience in evaluating fairness and equity in ML systems.\nClear written communication - the role produces evaluation memos and specifications that non-technical decision-makers rely on.\nHealthcare, clinical, or regulated-industry experience strongly preferred.\nEducation:\nBachelor's Degree-Related Field of Study (Required)Experience:\nMinimum of 8 years-Related work experience (Required), Minimum of 3 years-Managerial/Supervisory (Required)Certification(s) and License(s):\nSkills:\nOUR PURPOSE & VALUES: Everything we do is about caring for our patients, our members, our students, our Geisinger family and our communities.\nKINDNESS: We strive to treat everyone as we would hope to be treated ourselves. \nEXCELLENCE: We treasure colleagues who humbly strive for excellence. \nLEARNING: We share our knowledge with the best and brightest to better prepare the caregivers for tomorrow. \nINNOVATION: We constantly seek new and better ways to care for our patients, our members, our community, and the nation.\n SAFETY: We provide a safe environment for our patients and members and the Geisinger family. \nWe offer healthcare benefits for full time and part time positions from day one, including vision, dental and domestic partners. Perhaps just as important, we encourage an atmosphere of collaboration, cooperation and collegiality.\nWe know that a diverse workforce with unique experiences and backgrounds makes our team stronger. Our patients, members and community come from a wide variety of backgrounds, and it takes a diverse workforce to make better health easier for all. We are proud to be an affirmative action, equal opportunity employer and all qualified applicants will receive consideration for employment regardless to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or status as a protected veteran.","description_format":"text","description_chars":6088,"description_truncated":false,"requirements":{"experience_years_min":8,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["Equity"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Health Care","Hospitals & Health Systems","Health Insurance & Benefits"],"lifecycle":[{"event":"open","at":"2026-10-01T03:02:29Z"},{"event":"close","at":"2026-10-03T08:32:19Z"}],"visa":[],"liveness":null,"pay":null,"html_url":"https://alion.io/job/geisinger-director-ai-evaluation","json_url":"https://alion.io/job/geisinger-director-ai-evaluation.json","meta":{"generated_at":"2026-10-12T15:34:54Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":24475,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-13T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1774033},"rest":"https://alion.io/mcp/rest/get_company?id=1774033"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fgeisinger-director-ai-evaluation"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fgeisinger-director-ai-evaluation"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fgeisinger-director-ai-evaluation"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/geisinger-director-ai-evaluation\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fgeisinger-director-ai-evaluation"}]}