Principal Site Reliability Engineer
First seen by Alion on Sep 4, 2026.
Role Overview :
We're looking for a Principal Site Reliability Engineer to join our Site Reliability & Infrastructure Engineering team. We run entirely on the cloud, and this team owns production intelligence, reliability, and operational automation - moving from reactive incident response toward systems that observe, correlate, reason, and act. We're building the next generation of SRE: not dashboards and runbooks, but agents and automation that make our platform self-aware and self-healing at scale.
At Principal level, you won't just operate the platform - you'll define how we think about reliability across the entire engineering organization. You'll set the technical direction, establish the standards, and be the person other engineers turn to when the hardest infrastructure problems need to be solved.
What You'll Do :
- Define the technical vision and long-term architecture for ServiceTitan's reliability and infrastructure platform - setting the standard for how we operate at scale.
- Lead the design and delivery of major platform initiatives: Kubernetes infrastructure, observability frameworks, SLO programs, and incident management systems.
- Partner with Engineering Managers, Staff Engineers, and product engineering leadership to review architecture and infrastructure decisions before they ship - and hold the bar on non-functional requirements across the org.
- Identify systemic reliability risks before they become incidents, and drive remediation across teams with org-wide impact.
- Define and operationalize SLIs, SLOs, and error budgets across ServiceTitan's platform - not just within the SRE team, but as a standard every engineering team adopts.
- Drive adoption of reliability and observability best practices across engineering - through documentation, design reviews, and direct partnership with product teams.
- Design and build AI-assisted operational systems - agents that correlate production telemetry, diagnose failure patterns, recommend or execute remediation, and participate safely in deployment decisions.
- Partner with Infrastructure Engineering on deployment safety - progressive delivery, canary analysis, and AI-assisted promotion decisions.
- Mentor Staff and Senior SREs, raising the technical ceiling of the team through code reviews, architecture feedback, and direct pairing on complex problems.
- Contribute to the technical hiring bar - participate in system design interviews and help define what Principal-level looks like at ServiceTitan.
What You'll Bring :
- Kubernetes: deep, expert-level understanding of Kubernetes as a system - internals, failure modes, capacity planning, and large-scale cluster management.
- SRE principles: proven experience defining and operationalizing SLIs, SLOs, and error budgets across an engineering organization.
- Observability at scale: deep expertise with a modern observability stack (OpenTelemetry, Prometheus, Grafana, Datadog, or Elasticsearch).
- Cloud engineering & networking: expert grounding in AWS, GCP, or Azure - including networking fundamentals, security, and cost optimization at scale.
- Distributed systems depth: able to reason through complex failure modes and architect systems that degrade gracefully under pressure.
- Incident management: track record of leading major incident response, driving blameless postmortems, and implementing systemic fixes.
- CI/CD and delivery safety: deep experience with CI/CD systems, progressive delivery, and the ability to improve pipeline reliability and safety at org scale.
- AI in operations: experience building or deploying AI-assisted operational capabilities in production.
- 12+ years of relevant hands-on experience, with a clear track record of technical leadership on complex, cross-team infrastructure initiatives at product companies operating at scale.
About You :
- You've operated at Staff or Principal level before and know what it means to own a technical domain.
- You care about reliability the way a product engineer cares about user experience.
- You're drawn to the question of what operations looks like when AI agents handle the repetitive analysis and humans own the judgment calls.
Be Human With Us :
- Being human isn't about checking every box on a list. It's about the experiences we have, people we meet, and the perspectives we share. So, if you have the skills but are hesitant to apply because of your background, apply anyway. We need amazing people like you to help us challenge the conventional and think differently about the problems that we're solving.
Skills
DevOps, Kubernetes, AWS, Observability Services, Site Reliability, Azure, OS Deployment, IT Head, Grafana, Prometheus, CI/CD