Incident Post-Mortem

A payment page fails for two hours and the team restores it, but unless someone examines the failure and response, the same weakness may cause another outage, so a post-mortem creates a record that turns the event into a practical improvement plan. Google SRE describes a blameless postmortem culture and useful incident documentation, and Atlassian explains how blameless review encourages accurate reporting.

These are operating practices, not a guarantee that every recurrence can be prevented, and a blameless approach is not a ban on accountability, because it asks why the system allowed an error to create harm and what can be changed. Set criteria for which incidents need a formal review, since severity, customer harm, duration, security concerns or repeated near misses may all matter, and a near miss can deserve a review even if customers did not see it.

Hold the review soon enough that details can be recovered while giving responders time to rest and gather evidence, and invite frontline responders and affected business teams, who may know workarounds or customer consequences absent from system logs. Preserve confidentiality where customer data, security details or personnel information are involved, choose the audience deliberately, and for security incidents follow the organisation's separate reporting, evidence and legal procedures, because a general post-mortem does not replace them.

Write a timeline from alerts, logs, tickets and messages, marking uncertainty instead of filling gaps with confident guesses, and keep it separate from causal analysis and corrective actions so readers see what is known and what is proposed. Record the customer and business impact, meaning affected users, lost service, delayed work and any known financial consequences, and distinguish estimates from verified totals.

Explain how the incident was detected, because a customer complaint arriving before an internal alert points to a monitoring gap, and check whether the same event happened before, since repeat incidents can show that earlier actions were incomplete or ineffective. Describe the contributing conditions, not merely the person who made the last change, because weak tests, confusing procedures and missing checks can combine.

A single "root cause" may oversimplify a complex failure, so trace interactions and the reasons defences failed, and review response decisions with the information people had at the time, since hindsight can make an uncertain choice seem obviously wrong. List what worked as well as what failed, because a fast rollback or helpful support message can become a standard practice.

Use a small number of meaningful actions, each with an owner, due date and evidence of completion, and avoid "be more careful" as the only action, because training may help but a safer default or automatic check can be stronger. Prioritise changes by likely risk reduction and feasibility, since a long list no one funds is not a fix, and identify immediate containment separately from durable prevention, as restarting a service restored it but may not remove the trigger.

If an action depends on another team or vendor, get agreement on ownership rather than assigning it to an absent group, and record accepted residual risk when a full fix is not practical so that leaders know what remains exposed. Track action completion, but do not use closed tickets as the only measure; test whether the control actually works, and revisit the review after enough time to verify the main corrective actions and update the record if evidence changes.

Share relevant lessons across teams whose systems have similar failure modes, avoiding broadcasting sensitive detail without a need, and use plain language so owners can understand the business effect and decide on investment. The review is complete when the facts are clear enough, uncertainty is marked, owners accept the actions and follow-through is tracked.