{"id":2895112,"url":"https://alion.io/job/dayforce-ai-engineer-sr","title":"AI Engineer Sr","company":{"id":175960,"name":"Dayforce","domain":"dayforce.com","url":"https://alion.io/company/dayforce","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Dayforce","truth_index":{"grade":"B","score":75,"open_postings":14,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-10-22T06:10:10Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Canada"],"countries":["CA"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":81000,"max_usd":140000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":65},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Embeddings","optional":false},{"name":"Function Calling","optional":false},{"name":"Human-in-the-Loop","optional":false},{"name":"Hybrid Search","optional":false},{"name":"LangChain","optional":false},{"name":"LangGraph","optional":false},{"name":"LLM","optional":false},{"name":"Multi-Agent Systems","optional":false},{"name":"OpenAI Agents SDK","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Reranking","optional":false},{"name":"Structured Outputs","optional":false}],"status":"live","first_seen_at":"2026-10-05T05:00:00Z","employer_posted_date":"2026-10-20","last_verified_at":"2026-10-22T04:10:39Z","board_verified":true,"closed_at":null,"days_open":17,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":17},"description":"About the opportunity\nWe are looking for an AI Engineer - Agentic Systems Evaluation to help define how we measure, test, and improve the quality of enterprise AI systems.\nAs AI evolves from conversational assistants and Retrieval-Augmented Generation (RAG) into tool using agents, multi-agent systems, and autonomous business workflows, evaluating only the final response is no longer enough.\nAn agent may reach the right answer while choosing the wrong tool, taking unnecessary steps, retrieving incorrect context, failing to escalate to a human, or violating business rules.\nThis role will build the evaluation frameworks needed to understand not only whether an AI system succeeded, but how it succeeded, how reliably it can repeat that outcome, and whether the architecture is appropriate for the problem.\nWhat you’ll get to do\nBuild Agentic Evaluation Frameworks\nDesign evaluation methodologies covering the complete AI execution lifecycle:\nIntent → Planning → Retrieval → Tool Use → Reasoning → Action → Business Outcome\nEvaluate systems across dimensions including:\ntask and business outcome accuracy\nplanning and decision quality\nretrieval quality and groundedness\ntool selection and execution\nagent routing and delegation\nhuman escalation decisions\nreliability and failure recovery\nsafety and policy compliance\nlatency, token consumption, and cost\nEvaluate Agentic Design Patterns\nDesign experiments and benchmarks that help engineering teams determine which architecture works best for a given problem.\nEvaluate patterns such as:\nSingle-agent vs. multi-agent systems\nSupervisor/router architectures\nPlanner–executor patterns\nSequential and parallel workflows\nTool-using agents\nHuman-in-the-loop workflows\nLong-running agents with state and memory\nMeasure whether additional agent complexity actually improves task success, reliability, and business outcomes enough to justify increased latency, cost, and operational complexity.\nAdvance RAG & Knowledge Evaluation\nBuild rigorous evaluation approaches for enterprise retrieval and knowledge systems, including:\nretrieval precision and relevance\ngroundedness and citation accuracy\nsource authority and freshness\nchunking, metadata, and indexing strategies\nsemantic vs. hybrid search and reranking\npermission-aware retrieval\nEvaluate how retrieval decisions ultimately impact downstream agent performance, rather than treating RAG evaluation as an isolated problem.\nBuild Automated Evaluation & Regression Testing\nDevelop scalable evaluation infrastructure including:\ngolden and synthetic datasets\nscenario and adversarial test suites\ndeterministic graders\nLLM-as-a-Judge evaluation\nhuman evaluation workflows\ntrace-based evaluation\nautomated regression testing\nIntegrate evaluations into AI development and release pipelines so changes to models, prompts, retrieval, tools, or agent architectures can be measured before reaching production.\nBuild Agent Trace & Failure Analysis\nAnalyze complete agent execution traces including planning, retrieved context, tool calls, handoffs, retries, exceptions, latency, and cost.\nDevelop failure taxonomies that distinguish between:\nModel | Retrieval | Planning | Tool | Routing | Memory | Integration | Policy | Orchestration failures\nTurn production failures and user feedback into measurable regression tests and engineering improvements.\nDefine Production AI Quality\nEstablish measurable quality standards and release criteria for AI systems.\nMetrics may include Task Success Rate, First-Pass Success Rate, Tool Selection Accuracy, Agent Routing Accuracy, Plan Execution Fidelity, Failure Recovery Rate, Human Escalation Accuracy, Business Outcome Accuracy, Cost / Latency per Successful Task\nHelp teams answer a fundamental question:\nIs this AI system reliable, safe, efficient, and valuable enough to operate in production?\nSkills and experience we value\nStrong experience with:\nPython and software engineering\nLLM application development\nAI/ML evaluation and experimentation\nautomated testing and data analysis\nRAG, embeddings, hybrid search, and reranking\ntool/function calling and structured outputs\nagent orchestration and multi-agent workflows\nAI observability and tracing\nAPIs and enterprise integrations\nExperience with platforms or frameworks such as Agents SDK, LangGraph/LangChain, Microsoft AI Foundry, Amazon Bedrock, or similar agent platforms is valuable.\nYou should be comfortable working with evaluation techniques such as offline/online evals, deterministic graders, model-based graders, human evaluation, synthetic datasets, adversarial testing.\nWhat would make you stand out\nYou don't stop when an agent successfully completes a task. You ask:\nDid it choose the right approach?\nDid it use the right tools and information?\nCan it succeed consistently?\nCan it recover when something fails?\nDid adding more agents improve the outcome?\nCould a simpler architecture achieve the same result?\nWhat did the successful outcome cost?\nYou turn those questions into measurable experiments that help engineering teams build better AI systems.\nWhy This Role Matters\nEnterprise AI is moving from systems that answer questions to systems that make decisions and perform work.\nThat changes how quality must be measured.\nThe next generation of AI evaluation must measure:\nWhat the system understood → what it retrieved → what it decided → what it did → and whether the business outcome was correct.\nThis role will help establish the engineering discipline required to make those systems measurable, reliable, and production-ready.","description_format":"text","description_chars":5535,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["HR & Payroll Software"],"lifecycle":[{"event":"open","at":"2026-10-20T04:23:24Z"}],"visa":[{"country":"CA","licensed_sponsor":true,"evidence":"Approved LMIA in the last two years","filings_12m":0,"filings_prev_12m":4,"green_card_filings_12m":null,"median_offered_wage_usd":null,"route":null,"cap_exempt":false,"checked_at":"2026-10-03T21:10:52+00:00","sources":["ESDC: employers issued a positive LMIA"],"filings_for_role_12m":0}],"liveness":{"score":52,"band":"ok","label":"Likely open","p_open":1,"p_active":0.581,"p_room":0.9,"age_days":17,"expected_fill_days":26,"reasons":["conf:1","stale_co","velocity","win:mid"],"computed_at":"2026-10-22T06:10:10Z"},"pay":null,"html_url":"https://alion.io/job/dayforce-ai-engineer-sr","json_url":"https://alion.io/job/dayforce-ai-engineer-sr.json","meta":{"generated_at":"2026-10-22T06:36:27Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":1656,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-23T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":175960},"rest":"https://alion.io/mcp/rest/get_company?id=175960"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fdayforce-ai-engineer-sr"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fdayforce-ai-engineer-sr"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fdayforce-ai-engineer-sr"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/dayforce-ai-engineer-sr\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fdayforce-ai-engineer-sr"}]}