Brent Chapman

A vice president joins an incident channel and posts a single, reasonable question: "What's the customer impact?" Every responder in the channel pauses. The person investigating the network layer stops to check whether they should be the one to answer. Th

You see someone declare a Sev-2 and you wonder: wait, why is that even an incident? Nothing is down. Customers aren't affected. But a manager needed to get their team's problem to the top of another team's priority queue, and the incident process was a re

Your company has probably invested significantly in what happens after an incident is identified: incident response tooling, trained incident commanders, communication protocols, on-call rotations. That investment matters. But what about the gap between w

Is it coddling when an on-call engineer takes the next morning off to recover after handling a production incident at 3 a.m., or is it a smart company managing a reliability risk? Here's what that night actually looks like. The engineer gets paged at 3 a.

It's 3 a.m. in California, where most of the dev team are still snug in their beds. The auth system has started rejecting valid credentials. Early bird East Coast customers are already trying (and failing) to log in for the day, and thousands of users in

Every fire department has a training program. Big-city departments have entire training divisions; even small volunteer departments that can't spare anyone full time still name a training officer. Not because training is the department's mission, but beca

In the middle of a major incident, a senior leader joins the response channel and posts something like this: "Let's try to get this resolved in the next 10 minutes, please!" They mean it as encouragement. Maybe they're feeling pressure from their own lead

Most of the time, simultaneous incidents are no big deal. Any company with enough engineers and enough services will have multiple incidents open at the same time; when I led incident management at Slack, we had a few thousand engineers, and half a dozen

Most companies' incident processes work well for routine incidents. An engineer gets paged, a few colleagues join the response, someone takes on the incident commander (IC) role, and the team works the problem together in a Slack channel. The IC can keep

Every large software development organization has made the same bargain: if nobody has to understand the whole system, we can build a more capable system than we otherwise could, even though it grows bigger and more complex. We break systems into componen