{"id":3648285,"url":"https://alion.io/job/workday-senior-ai-evaluation-engineer-cicd-platform","title":"Senior AI Evaluation Engineer (CICD & Platform)","company":{"id":15002,"name":"Workday","domain":"workday.com","url":"https://alion.io/company/workday","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":{"grade":"A","score":86,"open_postings":79,"ghost_share":0.025,"stale_share":0.443,"repost_share":0.076,"time_to_fill_p50_days":23,"computed_at":"2026-11-08T05:47:12Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Dublin, Ireland"],"countries":["IE"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":96000,"max":144000,"currency":"EUR","period":"year","gross":null,"usd_annual":161300},"salary_estimate":null,"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":true,"technologies":[{"name":"AI Agents","optional":false},{"name":"Braintrust","optional":false},{"name":"CI/CD","optional":false},{"name":"DeepEval","optional":false},{"name":"GitHub Actions","optional":false},{"name":"GitLab CI","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Platform Engineering","optional":false},{"name":"Promptfoo","optional":false},{"name":"Pydantic","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"Structured Outputs","optional":false},{"name":"Synthetic Data","optional":false}],"status":"live","first_seen_at":"2026-11-02T23:26:04Z","employer_posted_date":"2026-11-02","last_verified_at":"2026-11-08T15:05:02Z","board_verified":true,"closed_at":null,"days_open":5,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":5},"description":"Your work days are brighter here.\nWe’re obsessed with making hard work pay off, for our people, our customers, and the world around us. As a Fortune 500 company and a leading AI platform for managing people, money, and agents, we’re shaping the future of work so teams can reach their potential and focus on what matters most. The minute you join, you’ll feel it. Not just in the products we build, but in how we show up for each other. Our culture is rooted in integrity, empathy, and shared enthusiasm. We’re in this together, tackling big challenges with bold ideas and genuine care. We look for curious minds and courageous collaborators who bring sun-drenched optimism and drive. Whether you're building smarter solutions, supporting customers, or creating a space where everyone belongs, you’ll do meaningful work with Workmates who’ve got your back. In return, we’ll give you the trust to take risks, the tools to grow, the skills to develop and the support of a company invested in you for the long haul. So, if you want to inspire a brighter work day for everyone, including yourself, you’ve found a match in Workday, and we hope to be a match for you too.\nAbout the Team\nThe Forward Intelligence team owns the skills that power Workday's deployment agent-turning the deep configuration expertise and hard-earned wisdom of our Domain Experts and Deployment Consultants into production-grade, fully tested skills that the agent can execute lawfully in live customer tenants.\nWhat we own: The end-to-end skill engine and governance ecosystem, spanning skill knowledge extraction, promotion pipelines, evaluation frameworks, self-correcting drift detection, and developer authoring environments.\nHow we build: Automated testing harnesses and release-gating pipelines running at enterprise scale, ensuring deterministic execution, strict schema compliance, and zero regressions for skills executing in live customer tenants.\nAgentic engineering: We build evaluation tooling, synthetic datasets, and automated red-teaming pipelines that help engineers test how AI agents and Model Context Protocol (MCP) skills perform across complex tenant environments.\nHow we work: Close end-to-end collaboration across domain experts, AI platform engineers, and site reliability teams to transform domain knowledge into trustworthy, production-ready AI skills.\nAbout the Role\nWe are seeking a Senior AI Evaluation Engineer (CI/CD & Platform) to own the continuous evaluation architecture, deployment pipeline infrastructure, and release gating for our enterprise AI Skills platform. In this role, you will build both the MLOps/CI-CD deployment pipelines and the automated eval harnesses that ensure all production AI skills execute accurately, remain schema-compliant, and run safely in live customer environments.\nYou will bridge the gap between AI skill development and production platform engineering-leveraging modern evaluation frameworks (e.g., LangSmith, Braintrust, DeepEval) alongside CI/CD pipelines to evaluate agent trajectories, test schema contracts, and certify every skill before deployment.\nKey Responsibilities\nContinuous Evaluation:. Build tiered evaluation pipelines-separating instant static/schema checks for PR developer loops from comprehensive synthetic, multi-turn trajectory, and LLM-as-a-Judge eval suites for release promotion gates.\n CI/CD Quality Gates & MLOps Infrastructure: Architect and maintain CI/CD build pipelines (GitHub Actions, GitLab CI), skill registry versioning, and runner environments to enforce automated \"pass/fail\" release gates on every pull request.\nDeterministic Code & Schema Validation:. Develop synthetic data generation pipelines and automated red-teaming utilities that allow skill authors to instantly generate edge-case evaluation suites from raw API schemas and domain specifications.\nBenchmark Datasets & Regression Testing: Curate versioned golden datasets and execute continuous regression testing across base model upgrades, prompt changes, and security boundary checks.\nProduction Observability: Ingest real-world production traces and failure logs back into benchmark datasets to continuously close test coverage gaps.\nAbout You\nBasic Qualifications\n4+ years of professional software engineering experience in Python building production-grade systems-with hands-on strength in EITHER CI/CD pipeline engineering OR AI evaluation methodology, plus working familiarity with the other.\n2+ years designing, building, and operating test automation frameworks (pytest), CI/CD build pipelines (GitHub Actions, GitLab CI), or automated release gates.\n1+ years of hands-on experience evaluating or building Generative AI applications (LLM tool-calling, agent workflows, or structured output parsing).\nBachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.\nOther Qualifications\nAI Evaluation Tooling: Direct experience with modern evaluation and observability frameworks (e.g., LangSmith, Braintrust, DeepEval, Promptfoo, Phoenix).\nSchema Contracts & API Validation: Strong knowledge of schema validation libraries (JSON Schema, Pydantic, OpenAPI) and API contract testing.\nAgentic Standards & MCP: Familiarity with the Model Context Protocol (MCP), tool-calling specifications, or agent execution harnesses.\nStatic Code Analysis: Experience using schema validation frameworks (Pydantic, JSON Schema), linters, and test automation tooling to enforce agreed code contracts.\nAI Security & Governance: Understanding of core LLM failure modes (hallucinations, prompt injection, PII/PHI sanitization, RBAC boundaries).\nQuality & Testing Discipline: Strong engineering rigor around test coverage, dataset versioning, evaluator calibration metrics, and regression prevention in continuous deployment pipelines.\nTechnical Communication & Leadership: Demonstrated ability to own problem domains end-to-end, collaborate closely with platform and product teams, and write clear technical documentation and architecture designs.\nWorkday Pay Transparency Statement (For EU Locations Only)\nListed below is the base salary range applicable to this position. Workday pay ranges (and the precise pay offered to the successful candidate) are based on a number of objective criteria such as relevant experience and skills, and educational qualifications, level of responsibility, demands of the role, work location and business need. As a part of the total compensation package, this role may be eligible for the Workday Bonus Plan or a role-specific commission/bonus, as well as annual refresh stock grants awarded by Workday Inc. For more information regarding Workday’s comprehensive benefits, please click here.\nPrimary Location Base Pay Range: €96,000 EUR - €144,000 EUR IrelandOur Approach to Flexible Work\nWith Flex Work, we’re combining the best of both worlds: in-person time and remote. Our approach enables our teams to deepen connections, maintain a strong community, and do their best work. We know that flexibility can take shape in many ways, so rather than a number of required days in-office each week, we simply spend at least half (50%) of our time each quarter in the office or in the field with our customers, prospects, and partners (depending on role). This means you'll have the freedom to create a flexible schedule that caters to your business, team, and personal needs, while being intentional to make the most of time spent together. Those in our remote \"home office\" roles also have the opportunity to come together in our offices for important moments that matter.\nPursuant to applicable Fair Chance law, Workday will consider for employment qualified applicants with arrest and conviction records.\nWorkday is an Equal Opportunity Employer including individuals with disabilities and protected veterans.\nWorkday is committed to providing reasonable accommodations for qualified individuals during our application process, in order to perform one or more essential functions of their job, as well as regarding the use of AI tools for employment decision-making to any degree. Please see below for more details including how to request an accommodation as a qualified veteran, due to a disability or for religious reasons, or as otherwise provided under applicable law.\nWorkday prohibits taking adverse action against any candidate or employee for reporting a possible violation of this policy, requesting one or more work accommodations, exercising a privacy right, or cooperating in an investigation in accordance with applicable law. Any employee who retaliates against a candidate or employee for doing so may be subject to disciplinary action, up to and including termination of employment, to the fullest extent allowable under applicable law.\nIf you require a reasonable accommodation, you may email , as far in advance as possible.\nAre you being referred to one of our roles? If so, ask your connection at Workday about our Employee Referral process!\nAt Workday, we value our candidates’ privacy and data security. Workday will never ask candidates to apply to jobs through websites that are not Workday Careers.\nPlease be aware of sites that may ask for you to input your data in connection with a job posting that appears to be from Workday but is not.\nIn addition, Workday will never ask candidates to pay a recruiting fee, or pay for consulting or coaching services, in order to apply for a job at Workday.","description_format":"text","description_chars":9454,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["Flexible schedule","Home office"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["ERP & Business Management Software","HR & Payroll Software","Accounting & Tax Software"],"lifecycle":[{"event":"open","at":"2026-11-02T23:26:04Z"}],"visa":[{"country":"IE","licensed_sponsor":true,"adverts_offering_sponsorship_24m":0,"sanctioned_now":false,"evidence":"Irish employment permits in 12 months: 31","filings_12m":31,"filings_prev_12m":36,"green_card_filings_12m":null,"median_offered_wage_usd":null,"route":"Employment permits","cap_exempt":false,"checked_at":"2026-11-08T10:45:15+00:00","sources":["Department of Enterprise, Trade and Employment: employment permits issued to companies"],"filings_for_role_12m":0}],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":5,"expected_fill_days":23,"reasons":["conf:1","velocity","win:early","comp:brand"],"computed_at":"2026-11-08T05:47:12Z"},"pay":{"stated_usd_annual":161300,"is_top_pay":false},"html_url":"https://alion.io/job/workday-senior-ai-evaluation-engineer-cicd-platform","json_url":"https://alion.io/job/workday-senior-ai-evaluation-engineer-cicd-platform.json","meta":{"generated_at":"2026-11-08T17:45:28Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":37260,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-09T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":15002},"rest":"https://alion.io/mcp/rest/get_company?id=15002"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fworkday-senior-ai-evaluation-engineer-cicd-platform"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fworkday-senior-ai-evaluation-engineer-cicd-platform"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fworkday-senior-ai-evaluation-engineer-cicd-platform"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/workday-senior-ai-evaluation-engineer-cicd-platform\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fworkday-senior-ai-evaluation-engineer-cicd-platform"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3648285}},{"tool":"search_jobs","arguments":{"company":"workday.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3648285},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}