A post-incident review is worth writing only if it changes something

Most incident reviews end as documents nobody opens again. A one-page format, actions filed as ordinary issues and a thirty-day check make change likely.

Published

Updated

Reading time

9 min

A post-incident review is worth writing only if something in the system is different a month later. A faster alert, a safer default, a deleted dangerous command, a runbook step that exists because the command could not be made safe. If you cannot name the change before you start writing, the review will produce a well-formatted record of an outage and nothing else, and for a team of two to ten engineers with no reliability function, that is a poor use of an afternoon.

The advice on how to write a blameless review is easy to find and mostly sound: describe systems rather than people, keep a timeline, ask what made the failure easy. This article takes those as given and argues about the part that decides whether any of it matters, which is what happens to the action items. The format below is built around that, and it is short on purpose.

The best case for the long document#

The strongest objection to this position is that a review has value even when no action follows. The act of reconstructing the timeline teaches the people in the room things they did not know about their own system. A written record lets a new engineer, or you in a year, recognise a failure when it comes back. And some incidents have no fix: the dependency was down, the cause was a one-off, the risk is accepted.

All true, and none of it requires a long document. The learning happens in the conversation, not in the file. The record needs a summary, a timeline, the conditions that made the failure possible, and what was decided, which fits on a page. What the long document adds is the appearance of diligence, written for a manager or customer who will not read it. And the third case, where no action is warranted, is itself a decision worth writing down: "we considered X and accepted the risk because Y" is an action item with a different outcome. An honestly dropped action is a good result. A silently dropped one is the failure to prevent.

Where the actions die#

Actions fail in predictable ways, and each has a structural fix.

  • They are wishes. "Improve monitoring" has no owner, no date and no definition of done. It sounds responsible and cannot be started.
  • There are too many. Fifteen actions mean none will happen. The list has been written to demonstrate thoroughness, not to be done.
  • They live in the document. A table in a markdown file is invisible to the people who plan the sprint. It does not compete with feature work, so it loses by default without anyone deciding it should.
  • They change attention, not the system. "Be more careful with migrations" cannot be checked and does not survive a busy week.
  • Nobody looks again. Without a date on which someone asks what shipped, the review is closed the moment the meeting ends.

The format, the meeting and the tracking below each exist to remove one of those.

A page that points at changes#

Keep reviews as Markdown files in a docs/incidents/ folder in the main repository, named YYYY-MM-DD-symptom-in-words.md. They are diffable, searchable with ordinary tools and reviewed like any change. A short index file listing each review with severity and a one-line summary lets a new engineer search for a symptom. Add a "Related" line when a review resembles an earlier one: recurrence is the most important signal in the whole archive.

docs/incidents/TEMPLATE.mdmarkdown
# YYYY-MM-DD short symptom in plain words
 
Severity: 1 | 2   Duration: HH:MM   Author: name   Status: draft | reviewed | closed
 
## Summary
Two or three sentences. What users saw, how long, and the one-line cause.
 
## Impact
Who was affected, what they could not do, and any data effect. Use counts
from logs if you have them, and say "approximately" if you do not.
 
## Timeline (UTC)
- HH:MM  The first sign. Not when someone noticed, when it began.
- HH:MM  Detected, and by what (alert, user report, someone watching).
- HH:MM  First action taken, and the reasoning at the time.
- HH:MM  Mitigation applied.
- HH:MM  Confirmed resolved, and how we confirmed.
 
## What made this possible
Conditions, not culprits. Each item is a sentence about the system.
 
## What helped
A dashboard, a runbook step, a quick rollback: what not to remove in clean-up.
 
## Where we got lucky
What could have made this worse and did not.
 
## Actions
| Action | Type | Owner | Due | Issue |
| --- | --- | --- | --- | --- |

Three sections do most of the work. The first line of the timeline is when the failure began, not when it was noticed, because the gap between those two moments is the detection delay and nearly always the most useful number in the review. "What made this possible" replaces the root-cause heading: a single root cause is usually a tidy story, while real incidents combine a missing check, an alert people had learned to ignore and a deploy at a bad time. And "Where we got lucky" is where the next incident tends to be hiding, because luck is a safeguard that has not been tested yet.

The Actions table has a Type column for a reason. Sort every proposal into one of three kinds: detect sooner, limit the blast radius, recover faster. If one kind is empty, say so aloud and ask whether that is true. A review whose actions are all "recover faster" has accepted that the next incident will be just as long and just as visible.

Make the follow-up visible#

Every row in the Actions table is also an issue in the normal tracker, labelled incident-action, with the review linked from the issue and the issue from the row. Now the action competes with feature work in the place where prioritisation happens, and an owner and a due date are fields the tracker already has.

Then add the thing most teams skip: a date on which someone looks. A small script lists the incident actions that are still open after thirty days, so nothing depends on memory:

scripts/incident-actions.shbash
#!/usr/bin/env bash
# Lists open incident actions older than 30 days.
# Requires the GitHub CLI authenticated for the repository.
set -euo pipefail
 
# GNU date; on macOS use: date -v-30d +%Y-%m-%d
cutoff=$(date -u -d '30 days ago' +%Y-%m-%d)
 
gh issue list \
  --label incident-action \
  --state open \
  --limit 100 \
  --search "created:<${cutoff}" \
  --json number,title,assignees,createdAt \
  --template '{{range .}}#{{.number}} {{.title}} ({{.createdAt}}){{"\n"}}{{end}}'

gh issue list supports --label, --state, --limit, --search, --json and --template, and its default limit is 30, hence the explicit one; see the GitHub CLI manual. Run it from a weekly scheduled workflow or a calendar reminder, whichever your team will actually keep.

From incident to a checked action in about a month. The three action groups keep the list balanced.

The thirty-day check is a ten-minute conversation: of the actions from this review, which shipped, which were dropped, and why. The review is marked closed only when every action is either done or dropped with a reason written in the issue. That one rule is what separates a review that changed something from one that recorded something.

Writing an action that can land#

A good action is something an engineer can start without asking a question. Three properties:

  • It changes the system. "Add a lock timeout to the migration runner so it fails instead of blocking" is an action. "Be more careful with migrations" is not.
  • It is sized. If it takes more than a couple of days, commit to the first slice and file the rest as parked work reviewed quarterly.
  • It has a way to check. "The alert fires within five minutes when the queue age passes ten minutes, verified with a test message" can be checked. "Better alerting" cannot.

When you have a choice, prefer an action that removes a class of failure over one that adds a reminder. Automation, defaults and guard rails outlive memory. A runbook is better than nothing, but a runbook step that exists because a command is dangerous suggests the command should be made safer. And commit to at most three actions per review. The point of the limit is the same as the point of the issue: to force the choice of what is worth doing, at the moment when the incident is fresh enough to make that choice well.

Running the meeting so the actions are real#

The meeting exists for what a document cannot do: finding the facts nobody wrote down, and choosing actions. Thirty to forty-five minutes, in a fixed order.

  1. The author reads the summary and timeline aloud and the group corrects it. Corrections are where the new facts appear, so do not skip this.
  2. Each person involved says what they knew at each point. The prompt is "what did you see, and what did you think it meant?", never "why did you do that?".
  3. The group fills in "What made this possible" and "Where we got lucky" together, on screen.
  4. The group proposes actions, sorts them by type, and picks at most three.

If someone explains a person's choice as a character flaw, the facilitator redirects to the information available at that moment. People act sensibly on the information they have, and when the outcome was bad, the information or the tools were the gap. One more safeguard: if the person who was on call writes the whole draft alone, their view becomes the record, so have someone else read the timeline against the logs before the meeting. And keep personal data out of the file: use counts and anonymised examples, since the repository's access list is the audience.

Incidents that do not deserve a review#

The argument cuts both ways. If an incident would not change anything, do not write the review, and let severity decide the ceremony. A simple three-level scale is enough:

  • Level one: customer-visible loss of a core function, or any data loss. Full review and meeting, draft timeline within a day and review within a week, while memory is fresh.
  • Level two: degraded service or a close call. A written review and no meeting unless someone asks.
  • Level three: a blip that resolved itself with no user effect. A two-line entry in an incident log: what happened and what you would look at first next time.

A solo developer can skip the meeting and read the draft the next morning, keeping only the template and the thirty-day check. A team large enough that incidents routinely span several services and owners has outgrown a one-page format and will want a facilitator and probably a tool. Two guards keep the scale honest. Do not let every incident become level one, because meetings that feel like a tax stop being attended. And watch the archive: when it holds several reviews with the same theme, the actions are not addressing the cause. Skim it once a quarter and promote the repeat offender from an action to a project.

If the review will be seen by customers, write a separate document for them, derived from the internal one. Never publish the internal review unedited.

Share
All articles →

An agent does what the spec says, not what you meant. Scope, named files and an executable check turn a long unattended session into a reviewable diff.

12 min

Denormalising trades write complexity for read speed, and the bill arrives later. Four questions decide when a copy is worth it, and how to keep it honest.

11 min