{"id":2701530,"url":"https://alion.io/job/palo-alto-networks-principal-machine-learning-engineer","title":"Principal Machine Learning Engineer","company":{"id":13263,"name":"Palo Alto Networks","domain":"paloaltonetworks.com","url":"https://alion.io/company/paloaltontwks","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":{"grade":"B","score":78,"open_postings":244,"ghost_share":0,"stale_share":0.881,"repost_share":0,"time_to_fill_p50_days":37,"computed_at":"2026-10-30T05:48:23Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"lead","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":163200,"max":264000,"currency":"USD","period":"year","gross":null,"usd_annual":264000},"salary_estimate":null,"experience_years_min":7,"visa_sponsorship":false,"relocation_package":false,"has_equity":true,"technologies":[{"name":"AI Agents","optional":false},{"name":"AWS","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"Embeddings","optional":false},{"name":"Fine-tuning","optional":false},{"name":"Function Calling","optional":false},{"name":"GCP","optional":false},{"name":"LLM","optional":false},{"name":"LLM Guardrails","optional":false},{"name":"Machine Learning","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Together AI","optional":false},{"name":"Docker","optional":true},{"name":"Human-in-the-Loop","optional":true},{"name":"Kubernetes","optional":true},{"name":"LLM Evaluation","optional":true},{"name":"SQL","optional":true}],"status":"live","first_seen_at":"2026-10-01T00:00:00Z","employer_posted_date":"2026-10-01","last_verified_at":"2026-10-30T16:04:14Z","board_verified":true,"closed_at":null,"days_open":29,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":29},"description":"Our Mission\nAt Palo Alto Networks®, we’re united by a shared mission-to protect our digital way of life. We thrive at the intersection of innovation and impact, solving real-world problems with cutting-edge technology and bold thinking. Here, everyone has a voice, and every idea counts. If you’re ready to do the most meaningful work of your career alongside people who are just as passionate as you are, you’re in the right place.\nWho We Are\nIn order to be the cybersecurity partner of choice, we must trailblaze the path and shape the future of our industry. This is something our employees work at each day and is defined by our values: Disruption, Collaboration, Execution, Integrity, and Inclusion. We weave AI into the fabric of everything we do and use it to augment the impact every individual can have. If you are passionate about solving real-world problems and ideating beside the best and the brightest, we invite you to join us!\nWe believe collaboration thrives in person. That’s why most of our teams work from the office full time, with flexibility when it’s needed. This model supports real-time problem-solving, stronger relationships, and the kind of precision that drives great outcomes.Job Summary\nYour Career\nThe AI Canvas team is building our next-generation data exploration experience for cybersecurity. AI Canvas is designed to transform how security teams explore, understand, and act on complex security data by combining natural language interaction, real-time visualization, dashboards, and intelligent agents in a unified experience.\nWe are building beyond traditional dashboards and static user interfaces. AI Canvas is based on a modern agent- and skill-driven architecture, bringing together AI, data exploration, visualization, and collaboration. We are looking for engineers who are comfortable operating in ambiguity, excited by hard technical problems, and motivated to build reliable AI systems that security teams can trust.\nAs a Principal Machine Learning Engineer on the AI Canvas team, you will take significant technical ownership of the AI layer powering the platform. You will help define the architecture, build production-grade AI systems, and shape how intelligence is integrated throughout the product.\nYou will work across large language models (LLMs), retrieval-augmented generation (RAG), AI agents and assistants, agent harnesses, natural-language-to-query generation, evaluation systems, guardrails, and cloud-based AI infrastructure.\nThis is a highly technical and hands-on role. You will work closely with ML, backend, UI, product, and design teams to solve challenging AI problems in cybersecurity, where reliability, accuracy, scalability, latency, observability, and trust are critical.\nYour Impact\nProvide technical leadership for the architecture and development of the AI capabilities powering AI Canvas, from early design through production deployment and continuous improvement.\nDesign and build scalable, production-grade systems using LLMs, RAG, AI agents, assistants, tools, skills, and agent harnesses.\nArchitect intelligent workflows that enable users to explore complex security data through natural language, including natural-language-to-query generation, follow-up interactions, clarification, investigation, and troubleshooting.\nDefine and evolve the architecture for agentic AI systems, including orchestration, context management, tool invocation, memory, reasoning workflows, and multi-step task execution.\nBuild robust evaluation frameworks and harnesses for measuring AI quality, including correctness, relevance, reliability, regression detection, and end-to-end product behavior.\nEstablish evaluation methodologies using automated metrics, LLM-based evaluators, human evaluation, and representative production datasets.\nDesign and implement guardrails and safety mechanisms to improve reliability, reduce hallucinations, enforce system constraints, and ensure responsible behavior of AI-powered features.\nDrive improvements in model and system quality through prompt engineering, retrieval strategies, model selection, fine-tuning where appropriate, and systematic experimentation.\nBuild scalable RAG and knowledge-retrieval systems capable of grounding AI responses in large, complex, and evolving security datasets.\nPartner closely with backend and platform engineers to build reliable APIs, services, and infrastructure supporting AI workloads at production scale.\nEstablish engineering best practices around observability, debugging, tracing, versioning, reproducibility, testing, and monitoring of LLM and agent-based systems.\nEvaluate emerging models, frameworks, and AI infrastructure and determine when and how they should be incorporated into the product.\nBalance rapid experimentation with the engineering rigor required to operate mission-critical AI systems in production.\nMentor engineers, influence technical direction across teams, and raise the engineering bar for applied AI development.\nQualifications\nYour Experience\nBachelor’s degree in Computer Science, Machine Learning, Engineering, or a related technical field, or equivalent practical experience.\n7+ years of software engineering, machine learning engineering, or related industry experience, including significant experience building production systems.\nStrong experience designing and building machine learning or AI-powered applications at scale.\nHands-on experience building applications using large language models and generative AI technologies.\nExperience with one or more areas such as RAG, AI agents, AI assistants, tool-calling systems, agent orchestration, or LLM-based workflows.\nStrong proficiency in Python and experience developing production-grade backend or ML services.\nExperience designing evaluation systems for AI/ML applications, including offline evaluation, regression testing, quality measurement, and production monitoring.\nStrong understanding of modern ML and AI concepts, including embeddings, retrieval, ranking, prompt engineering, model inference, and experimentation.\nExperience building scalable systems on public cloud platforms such as GCP, AWS, or Azure.\nStrong software engineering fundamentals, including system design, distributed systems, APIs, testing, CI/CD, and observability.\nAbility to lead complex technical initiatives across multiple teams while remaining deeply hands-on.\nStrong communication skills and the ability to translate ambiguous product problems into clear technical architectures and execution plans.\n Preferred Experience\nMaster’s or PhD in Computer Science, Machine Learning, Artificial Intelligence, or a related technical field.\nDeep experience building and operating LLM-powered products or agentic AI platforms in production.\nExperience with AI development frameworks and tooling for model orchestration, agent systems, evaluation, tracing, or observability.\nExperience building natural-language-to-SQL or natural-language-to-query systems, semantic layers, or AI-powered data exploration products.\nExperience designing LLM evaluation harnesses, synthetic test generation, LLM-as-a-judge approaches, or human-in-the-loop evaluation workflows.\nExperience with vector databases, search and retrieval infrastructure, embeddings, and large-scale knowledge systems.\nExperience with containerization and orchestration technologies such as Docker and Kubernetes.\nExperience designing highly available, low-latency AI inference and backend services.\nFamiliarity with AI security, prompt injection defenses, data privacy, access control, and guardrails for enterprise AI applications.\nExperience in cybersecurity, security analytics, observability, or large-scale data platforms.\nContributions to open-source AI, ML, agent, or infrastructure projects.\nCompensation Disclosure\nThe compensation offered for this position will depend on qualifications, experience, and work location. For candidates who receive an offer at the posted level, the starting base salary (for non-sales roles) or base salary + commission target (for sales/com-missioned roles) is expected to be the annual range listed below. The offered compensation may also include restricted stock units and a bonus. A description of our employee benefits may be found here.\n$163,200.00 - $264,000.00/yrOur Commitment\nWe’re trailblazers that dream big, take risks, and challenge cybersecurity’s status quo. It’s simple: we can’t accomplish our mission without diverse teams innovating, together.\nWe are committed to providing reasonable accommodations for all qualified individuals with a disability. If you require assistance or accommodation due to a disability or special need, please contact us at .\nPalo Alto Networks is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.\nAll your information will be kept confidential according to EEO guidelines.\nIs role eligible for Immigration Sponsorship?: Yes","description_format":"text","description_chars":9368,"description_truncated":false,"requirements":{"experience_years_min":7,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":true,"languages":[]},"benefits":["Restricted stock units"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Threat Intelligence","Endpoint Security","Information Security","Security Operations"],"lifecycle":[{"event":"open","at":"2026-10-16T11:21:43Z"}],"visa":[],"liveness":{"score":45,"band":"ok","label":"Likely open","p_open":1,"p_active":0.606,"p_room":0.75,"age_days":29,"expected_fill_days":37,"reasons":["conf:0","stale_co","velocity","win:late","comp:brand"],"computed_at":"2026-10-30T05:48:23Z"},"pay":{"stated_usd_annual":264000,"is_top_pay":true},"html_url":"https://alion.io/job/palo-alto-networks-principal-machine-learning-engineer","json_url":"https://alion.io/job/palo-alto-networks-principal-machine-learning-engineer.json","meta":{"generated_at":"2026-10-30T17:30:21Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":6049,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-31T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":13263},"rest":"https://alion.io/mcp/rest/get_company?id=13263"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fpalo-alto-networks-principal-machine-learning-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fpalo-alto-networks-principal-machine-learning-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fpalo-alto-networks-principal-machine-learning-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/palo-alto-networks-principal-machine-learning-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fpalo-alto-networks-principal-machine-learning-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":2701530}},{"tool":"search_jobs","arguments":{"company":"paloaltonetworks.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":2701530},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}