{"id":480060,"url":"https://alion.io/job/caterpillar-manager-ai-evaluation-engineering","title":"Manager; AI Evaluation Engineering","company":{"id":176554,"name":"Caterpillar","domain":"caterpillar.com","url":"https://alion.io/company/caterpillar","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":{"grade":"B","score":84,"open_postings":32,"ghost_share":0,"stale_share":0.844,"repost_share":0,"time_to_fill_p50_days":12,"computed_at":"2026-10-17T05:46:00Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Chicago, United States","Peoria, United States","Westminster, United States","Irving, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":147760,"max":240110,"currency":"USD","period":"year","gross":null,"usd_annual":240110},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":true,"relocation_package":false,"has_equity":false,"technologies":[{"name":"A/B Testing","optional":false},{"name":"A2A","optional":false},{"name":"AI Agents","optional":false},{"name":"Amazon SageMaker","optional":false},{"name":"Arize Phoenix","optional":false},{"name":"AWS","optional":false},{"name":"AWS Bedrock","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"Cortex","optional":false},{"name":"DeepEval","optional":false},{"name":"Embeddings","optional":false},{"name":"Fine-tuning","optional":false},{"name":"Function Calling","optional":false},{"name":"GCP","optional":false},{"name":"Human-in-the-Loop","optional":false},{"name":"Incident Management","optional":false},{"name":"LangChain","optional":false},{"name":"Langfuse","optional":false},{"name":"LangGraph","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"LoRA","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Multimodal AI","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"RAG","optional":false},{"name":"Ragas","optional":false},{"name":"Snowflake","optional":false},{"name":"Synthetic Data","optional":false},{"name":"Tool Use","optional":false},{"name":"PEFT","optional":true},{"name":"Prometheus","optional":true}],"status":"live","first_seen_at":"2026-09-01T00:00:00Z","employer_posted_date":"2026-09-01","last_verified_at":"2026-10-17T06:11:28Z","board_verified":true,"closed_at":null,"days_open":46,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":46},"description":"Career Area:\nTechnology, Digital and DataJob Description:\nYour Work Shapes the World at Caterpillar Inc.\nWhen you join Caterpillar, you're joining a global team who cares not just about the work we do - but also about each other. We are the makers, problem solvers, and future world builders who are creating stronger, more sustainable communities. We don't just talk about progress and innovation here - we make it happen, with our customers, where we work and live. Together, we are building a better world, so we can all enjoy living in it.\nJob Summary:\nJoin the AI Engineering team of Cat Digital and take charge of leading a team dedicated to evaluating and validating our advanced generative AI solutions-including intelligent agents, digital assistants, and other innovative capabilities. You will shape the development of cutting-edge products that redefine how customers and dealers interact with technology through AI-driven experiences. \nWhat You Will Do:\nProviding strong technical support and clear direction to ensure the team is aligned with company goals and capable of evaluating advanced AI projects.\nOverseeing the performance of both individual team members and the team as a whole, fostering a culture of learning by identifying and addressing training and development needs. \nTaking ownership of the quality of AI engineering products, making sure all solutions are robust, reliable, and meet high standards. \nEstablishing and supervising the implementation of engineering best practices to maintain consistency and excellence in development processes. \nWhat You Will Have:\nProducts and Services: Knowledge of major products and services and product and service groups; ability to apply knowledge of product and service appropriately to diverse situations.\nSoftware Development: Knowledge of software development tools and activities; ability to produce software products or systems in line with product requirements.\nSoftware Development Life Cycle: Knowledge of software development life cycle; ability to use a structured methodology for delivering and managing new or enhanced software products to the marketplace.\nSoftware Quality Assurance and Testing: Knowledge of software quality assurance and testing; ability to apply appropriate processes, tools, and techniques for assuring a high level of quality in computer software products and systems.\nConsiderations For Top Candidates:\nProven experience leading software quality, test automation, validation, and AI evaluation teams, with the ability to scale processes, tools, metrics, and engineering practices across multiple products and enterprise initiatives.\nDemonstrated success delivering enterprise-scale software and GenAI solutions across hybrid cloud and embedded/edge environments, leveraging modern software engineering practices including CI/CD, automated testing, incident management, feature flags, and progressive deployment strategies (e.g., blue/green and canary deployments).\nDeep expertise in GenAI architecture and system design, including LLMs, SLMs, multimodal, speech, and real-time models; prompt engineering; agentic systems; tool use and orchestration; RAG architectures; vector databases, embeddings, and chunking strategies; fine-tuning techniques such as LoRA; and emerging standards including MCP and A2A.\nHands-on experience with AI development and evaluation ecosystems, including Azure, AWS, GCP, Azure AI Foundry, SageMaker, Bedrock, Snowflake Cortex, LangChain, LangGraph, Langfuse, Arize, LangSmith, Humanloop, Ragas, DeepEval, Phoenix, or similar technologies.\nDeep understanding of AI evaluation methodologies and the challenges of non-deterministic systems, including metric design, RAG quality assessment, output reliability, safety evaluation, A/B testing, human-in-the-loop validation, and balancing deterministic and probabilistic testing approaches.\nExpertise in modern AI observability and monitoring practices, including tracing, telemetry, evaluation data collection, root-cause analysis, and the use of AI technologies to enhance testing through automated test generation, synthetic data creation, scenario expansion, and LLM-assisted evaluation.\nSummary Pay Range:\n$147,760.00 - $240,110.00Compensation and benefits offered may vary depending on multiple individualized factors, job level, market location, job-relatedknowledge, skills, individual performance and experience. Please note that salary is only one component of total compensation at Caterpillar.\nBenefits:\nSubject to plan eligibility, terms, and guidelines. This is a summary list of benefits.\nMedical, dental, and vision benefits*\n\nPaid time off plan (Vacation, Holidays, Volunteer, etc.)*\n\n401(k) savings plans*\n\nHealth Savings Account (HSA)*\n\nFlexible Spending Accounts (FSAs)*\n\nHealth Lifestyle Programs*\n\nEmployee Assistance Program*\n\nVoluntary Benefits and Employee Discounts*\n\nCareer Development*\n\nIncentive bonus*\n\nDisability benefits\n\nLife Insurance\n\nParental leave\n\nAdoption benefits\n\nTuition Reimbursement\n\n* These benefits also apply to part-time employees\nVisa Sponsorship is not available for this position.Posting Dates:\nSeptember 1, 2026 - September 20, 2026Any offer of employment is conditioned upon the successful completion of a drug screen.\nCaterpillar is an Equal Opportunity Employer, Including Veterans and Individuals with Disabilities. Qualified applicants of any age are encouraged to apply.\nNot ready to apply? Join our Talent Community.","description_format":"text","description_chars":5457,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Life insurance","Parental leave"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Energy & Utilities","Manufacturing","Fossil Fuels","Industrial Machinery"],"lifecycle":[{"event":"open","at":"2026-09-06T16:02:27Z"},{"event":"open","at":"2026-10-14T02:00:51Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 343 · green card filings: 92","filings_12m":343,"filings_prev_12m":364,"green_card_filings_12m":92,"median_offered_wage_usd":114569,"route":null,"cap_exempt":false,"checked_at":"2026-10-15T11:24:14+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":164}],"liveness":{"score":4,"band":"cold","label":"Long shot","p_open":1,"p_active":0.134,"p_room":0.28,"age_days":46,"expected_fill_days":12,"reasons":["conf:4","stale_co","velocity","win:tail","crowd:brand"],"computed_at":"2026-10-17T05:46:00Z"},"pay":{"stated_usd_annual":240110,"is_top_pay":true},"html_url":"https://alion.io/job/caterpillar-manager-ai-evaluation-engineering","json_url":"https://alion.io/job/caterpillar-manager-ai-evaluation-engineering.json","meta":{"generated_at":"2026-10-17T07:09:38Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":9768,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":176554},"rest":"https://alion.io/mcp/rest/get_company?id=176554"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcaterpillar-manager-ai-evaluation-engineering"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcaterpillar-manager-ai-evaluation-engineering"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcaterpillar-manager-ai-evaluation-engineering"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/caterpillar-manager-ai-evaluation-engineering\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcaterpillar-manager-ai-evaluation-engineering"}]}