{"id":2942142,"url":"https://alion.io/job/peraton-aiml-operations-assurance-engineer","title":"AI/ML Operations & Assurance Engineer","company":{"id":722,"name":"Peraton","domain":"peraton.com","url":"https://alion.io/company/peraton","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"iCIMS","truth_index":{"grade":"A","score":100,"open_postings":11,"ghost_share":0,"stale_share":0,"repost_share":0.091,"time_to_fill_p50_days":10,"computed_at":"2026-10-24T06:02:35Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Chantilly, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":112000,"max":179000,"currency":"USD","period":"year","gross":null,"usd_annual":179000},"salary_estimate":null,"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"Amazon CloudWatch","optional":false},{"name":"AWS","optional":false},{"name":"AWS Bedrock","optional":false},{"name":"CI/CD","optional":false},{"name":"Function Calling","optional":false},{"name":"GitLab","optional":false},{"name":"Grafana","optional":false},{"name":"Kubernetes","optional":false},{"name":"LLM","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"OpenSearch","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Prometheus","optional":false},{"name":"RAG","optional":false},{"name":"ServiceNow","optional":false},{"name":"SRE","optional":false}],"status":"live","first_seen_at":"2026-10-20T17:37:14Z","employer_posted_date":"2026-10-20","last_verified_at":"2026-10-24T17:51:51Z","board_verified":true,"closed_at":null,"days_open":4,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":4},"description":"Responsibilities\nWe are seeking an AI/ML Operations & Assurance Engineer to monitor, evaluate, and maintain production AI-enabled applications, foundation-model integrations, and agentic workflows. This role focuses on operational reliability, workflow performance, evaluation and regression testing, configuration control, usage and cost monitoring, and the safe introduction of AI-related changes into production environments.\nThe ideal candidate combines cloud and application operations experience with an understanding of large language model (LLM) applications, agent behavior, and AI-specific operational risks. The engineer will collaborate with AI/ML developers, site reliability engineers, cybersecurity teams, and platform operations personnel to maintain reliable, auditable, and secure AI-enabled services.\nResponsibilities\nProduction Monitoring and Operations\nMonitor availability, latency, error rates, model and token usage, service costs, tool-call failures, and workflow completion metrics for production AI services.\n\nIntegrate AI service health and agent-workflow telemetry into observability dashboards, monitoring systems, and alerting processes.\n\nInvestigate failed, stalled, repetitive, or abnormal AI workflows and coordinate remediation with software, platform, and AI engineering teams.\n\nMaintain operational runbooks, escalation procedures, troubleshooting guidance, and recovery processes for AI-enabled applications.\n\nDistinguish infrastructure and application failures from issues associated with model behavior, prompts, retrieval sources, or agent orchestration.\n\nAI Evaluation and Operational Assurance\nDevelop and maintain repeatable evaluation scenarios for critical AI workflows, developer assistants, and model-service updates.\n\nAssess task completion, output quality, error patterns, inappropriate tool use, and other indicators of operational suitability.\n\nCoordinate regression testing before changes to models, prompts, tools, workflow logic, or retrieval sources are deployed to production.\n\nAnalyze user feedback and operational results to identify where workflows require refinement, additional controls, or restrictions on automated execution.\n\nDocument evaluation results, limitations, and recommendations to support evidence-based deployment decisions.\n\nChange and Configuration Management\nMaintain version control and change records for prompts, agent instructions, tool definitions, model selections, evaluation datasets, and workflow configurations.\n\nCoordinate model and service updates with AI engineering, cybersecurity, platform operations, and architecture stakeholders.\n\nSupport rollback to validated configurations when changes negatively affect reliability, performance, or workflow quality.\n\nEnsure AI-related changes follow established review, testing, approval, and deployment procedures.\n\nMonitor configuration drift and verify that production services remain aligned with approved configurations.\n\nSecurity, Audit, and Compliance Support\nCoordinate with cybersecurity and identity teams to verify access controls, tool permissions, logging, and auditability for production AI services.\n\nSupport investigations involving anomalous AI behavior, unauthorized actions, or potential data-handling concerns.\n\nCollect and maintain operational evidence supporting security assessments, continuous monitoring, and service reviews.\n\nVerify that production workflows use approved models, endpoints, tools, and data sources in accordance with established policies.\n\nSupport the implementation and validation of operational safeguards for AI-enabled functions.\n\nOperational Improvement and Reporting\nIdentify repetitive operational tasks and support activities suitable for controlled AI-assisted automation.\n\nMeasure the impact of deployed workflows on processing time, support workload, service quality, and operational efficiency.\n\nCollaborate with service delivery, Tier 2 support, SRE, and platform engineering teams to improve workflows based on operational demand.\n\nReport trends in service health, model usage, operating costs, evaluation results, and unresolved issues.\n\nRecommend improvements to monitoring, testing, configuration control, and operational procedures.\n\nLocation: This position is fully onsite in Chantilly, VA.\nQualifications\nRequired Qualifications\nActive TS/SCI clearance with CI Polygraph.\n\nBS/BA and 5 years of experience, an additional 4 years of experience may be considered in lieu of a degree.\nApproximately 5-8 years of experience in platform operations, site reliability engineering, cloud engineering, software operations, applied AI/ML operations, or production application support.\n\nExperience operating and troubleshooting production software applications or cloud services.\n\nExperience with monitoring, logging, metrics, incident response, and configuration or change management.\n\nExperience evaluating software behavior through repeatable test cases, regression testing, quality measures, or operational performance analysis.\n\nWorking knowledge of foundation-model or LLM-based applications, including prompts, model endpoints, tool calls, and retrieval-augmented applications.\n\nExperience with AWS or comparable cloud platforms.\n\nStrong analytical, troubleshooting, documentation, and communication skills.\n\nDesired Qualifications\nExperience with AWS Bedrock or comparable managed foundation-model services.\n\nFamiliarity with LLM or agent evaluation frameworks, automated evaluation harnesses, or AI application testing.\n\nExperience with OpenTelemetry, Amazon CloudWatch, OpenSearch, Prometheus, Grafana, or comparable observability technologies.\n\nFamiliarity with ServiceNow, IT service management, and incident-management processes.\n\nExperience with GitLab, CI/CD pipelines, and controlled configuration promotion.\n\nFamiliarity with Kubernetes and containerized application environments.\n\nExperience monitoring cloud utilization, model consumption, and service costs.\n\nKnowledge of risk management frameworks, continuous monitoring, or security evidence collection.\n\nExperience supporting classified, restricted, or highly regulated environments.\n\nPeraton Overview\nPeraton is a next-generation national security company that drives missions of consequence spanning the globe and extending to the farthest reaches of the galaxy. As the world’s leading mission capability integrator and transformative enterprise IT provider, we deliver trusted, highly differentiated solutions and technologies to protect our nation and allies. Peraton operates at the critical nexus between traditional and nontraditional threats across all domains: land, sea, space, air, and cyberspace. The company serves as a valued partner to essential government agencies and supports every branch of the U.S. armed forces. Each day, our employees do the can’t be done by solving the most daunting challenges facing our customers. Visit peraton.com to learn how we’re keeping people around the world safe and secure.\nTarget Salary Range\n$112,000 - $179,000. This represents the typical salary range for this position. Salary is determined by various factors, including but not limited to, the scope and responsibilities of the position, the individual’s experience, education, knowledge, skills, and competencies, as well as geographic location and business and contract considerations. Depending on the position, employees may be eligible for overtime, shift differential, and a discretionary bonus in addition to base pay.EEO\nEEO: Equal opportunity employer, including disability and protected veterans, or other characteristics protected by law.","description_format":"text","description_chars":7616,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"phd","optional":false},"security_clearance":true,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["National Security","Information Security","Homeland Security","Systems Integrators"],"lifecycle":[{"event":"open","at":"2026-10-20T17:37:14Z"}],"visa":[],"liveness":{"score":80,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.886,"p_room":0.9,"age_days":3,"expected_fill_days":10,"reasons":["conf:0","velocity","win:mid","comp:brand"],"computed_at":"2026-10-24T06:02:35Z"},"pay":{"stated_usd_annual":179000,"is_top_pay":false},"html_url":"https://alion.io/job/peraton-aiml-operations-assurance-engineer","json_url":"https://alion.io/job/peraton-aiml-operations-assurance-engineer.json","meta":{"generated_at":"2026-10-24T19:50:55Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":29006,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-25T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":722},"rest":"https://alion.io/mcp/rest/get_company?id=722"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fperaton-aiml-operations-assurance-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fperaton-aiml-operations-assurance-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fperaton-aiml-operations-assurance-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/peraton-aiml-operations-assurance-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fperaton-aiml-operations-assurance-engineer"}]}