{"id":1607495,"url":"https://alion.io/job/penguin-solutions-senior-principal-engineer","title":"Senior Principal Engineer","company":{"id":178085,"name":"Penguin Solutions","domain":"penguinsolutions.com","url":"https://alion.io/company/penguin-solutions","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"SuccessFactors","truth_index":null},"role":"Industrial Engineering","role_family":"Industrial Engineering","seniority":"lead","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":15,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"AutoGen","optional":false},{"name":"Context Engineering","optional":false},{"name":"Function Calling","optional":false},{"name":"HPC","optional":false},{"name":"Kubernetes","optional":false},{"name":"LangGraph","optional":false},{"name":"LLM","optional":false},{"name":"LLM Guardrails","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Semantic Kernel","optional":false},{"name":"Tool Use","optional":false},{"name":"Human-in-the-Loop","optional":true},{"name":"LangChain","optional":true},{"name":"Multi-Agent Systems","optional":true}],"status":"closed","first_seen_at":"2026-09-30T07:00:00Z","employer_posted_date":"2026-09-30","last_verified_at":"2026-10-12T08:56:18Z","board_verified":false,"closed_at":"2026-10-12T08:56:18Z","days_open":12,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":12},"description":"At Penguin Solutions (Nasdaq: PENG) - The AI Factory Platform Company - we’re building a team of innovators who thrive on collaboration, creativity, and the opportunity to help shape the future of AI. As part of the AI technology revolution, our teams design, build, deploy, and manage AI factories for enterprises, sovereign AI initiatives, and neocloud providers worldwide.\nHeadquartered in Silicon Valley, California, Penguin Solutions operates globally through a network of R&D, manufacturing, and sales locations. For nearly three decades, we have operated at the intersection of memory and AI/HPC infrastructure. That engineering expertise positions us to power the next generation of AI workloads, from training to inference and agentic AI at scale.\nPenguin Solutions brings together differentiated infrastructure software, advanced memory, compute systems, end-to-end services, and industry-leading partner solutions in a full-stack AI factory platform designed to help customers deploy and scale AI workloads with speed and precision.\nAt Penguin Solutions, we value ideas over hierarchy and believe in servant leadership, where leaders enable teams to do their best work. We empower employees to take ownership, drive innovation, and grow through challenging work, continuous learning, and exposure to advanced AI tools and technologies. With flexibility where it matters and a strong focus on outcomes, Penguin Solutions is a place to do your best work, grow your career, and make a meaningful impact.\nJob Overview\nWe are looking for a Senior Principal Engineer to design and deliver key architectural components of ClusterWareAI's next-generation AI Operational Agent. You will help build a secure, reliable, and measurable agentic AI platform capable of reasoning over enterprise infrastructure, orchestrating tools and workflows, and executing actions with appropriate levels of autonomy.\nThis is a hands-on technical leadership role for a senior technical expert who will design complex production AI systems, partner closely with engineering and technical leadership, mentor engineers, and contribute to the evolution of ClusterWareAI's AI platform architecture.\nResponsibilities\nDesign and implement key architectural components of the ClusterWareAI AI Operational Agent, including planning, reasoning, memory, state management, tool orchestration, and autonomous or human-approved execution.\nDesign scalable agentic workflows and enterprise AI architectures, including multi-agent approaches where appropriate.\nDesign knowledge management, RAG, context engineering, agent memory, and Model Context Protocol (MCP) capabilities for enterprise infrastructure automation.\nDesign reliable execution patterns for long-running and asynchronous agent workflows, including state management, retries, recovery, fallbacks, and idempotency.\nContribute to AI evaluation and observability frameworks covering task success, quality, reliability, latency, cost, tool execution, and failure modes.\nDesign and implement guardrails and secure execution patterns for autonomous operations, including appropriate controls for tool access, privileges, data protection, tenant isolation, and human approval.\nPartner with Product Management, AI, platform, and infrastructure engineering teams to translate customer and operational requirements into scalable AI capabilities.\nEvaluate model and architecture choices, including model selection, routing, inference performance, reliability, and cost tradeoffs.\nMentor engineers, contribute to technical standards and engineering best practices, and provide technical guidance on complex AI system design.\nEvaluate emerging AI technologies, frameworks, and platform capabilities and provide recommendations for adoption.\nRequired Qualifications\n10+ years of software engineering experience, including demonstrated experience designing and delivering complex production systems.\nDeep hands-on experience designing, building, and operating production-grade LLM or agentic AI systems involving planning, reasoning, tool use, orchestration, and autonomous or human-approved execution.\nStrong expertise in Python and production software engineering.\nStrong expertise in distributed systems and fault-tolerant architectures, including asynchronous workflows, state management, retries, recovery, failure handling, and idempotency.\nExperience with agent orchestration frameworks such as LangGraph, Semantic Kernel, AutoGen, or similar, and familiarity with Model Context Protocol (MCP) or equivalent tool-integration patterns.\nExperience with RAG architectures, vector databases, knowledge retrieval, context engineering, and agent memory.\nStrong understanding of enterprise software architecture, APIs, microservices, and cloud-native application development.\nHands-on experience with Kubernetes and cloud-native infrastructure, including deployment, APIs, automation, observability, and production operations.\nExperience designing or implementing AI evaluation and observability approaches with measurable quality, reliability, latency, and cost metrics.\nStrong understanding of AI/agent security, including prompt injection, tool security, excessive agency, privilege management, data protection, tenant isolation, and secure execution.\nExperience evaluating model selection, routing, inference performance, and cost tradeoffs in production AI systems.\nDemonstrated technical leadership, architecture experience, and ability to mentor engineers and influence complex technical decisions.\nPreferred Qualifications\n15+ years of software engineering experience, with significant experience in architecture or senior technical leadership roles.\nExperience building AI agents for infrastructure, cloud operations, SRE, Kubernetes, or enterprise IT operations.\nExperience with infrastructure management, observability, virtualization, storage, networking, or cloud platforms.\nExperience designing human-in-the-loop or human-on-the-loop systems and risk-based levels of agent autonomy.\nExperience with long-running agent workflows, persistent memory, multi-agent systems, and advanced context engineering.\nExperience operating AI systems in multi-tenant enterprise environments with strong security, auditability, and reliability requirements.\nExperience with GPU-based AI infrastructure, inference platforms, or AI workload orchestration.\nContributions to open-source AI agent, Kubernetes, AI infrastructure, or distributed-systems projects.\nPublished technical content, conference presentations, patents, or other demonstrated thought leadership in AI or distributed systems are a plus.\nLocation\nHybrid - Bangalore, India","description_format":"text","description_chars":6651,"description_truncated":false,"requirements":{"experience_years_min":15,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Continuous learning"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Education","LLM & Generative AI","High-Performance Computing"],"lifecycle":[{"event":"open","at":"2026-10-01T19:57:58Z"},{"event":"close","at":"2026-10-12T08:56:18Z"}],"visa":[],"liveness":null,"pay":null,"html_url":"https://alion.io/job/penguin-solutions-senior-principal-engineer","json_url":"https://alion.io/job/penguin-solutions-senior-principal-engineer.json","meta":{"generated_at":"2026-10-14T02:40:07Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":4078,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-15T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":178085},"rest":"https://alion.io/mcp/rest/get_company?id=178085"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fpenguin-solutions-senior-principal-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fpenguin-solutions-senior-principal-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fpenguin-solutions-senior-principal-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/penguin-solutions-senior-principal-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fpenguin-solutions-senior-principal-engineer"}]}