Maintenance

Write Incident Runbooks for Common Failure Modes

Document concise response procedures for recurring operational failures and make them usable under pressure.

A runbook should help an engineer make safe decisions during an incident. It is not a long architecture document or a substitute for understanding the system.

Structure the response

Include symptoms, impact assessment, dashboards, safe diagnostic commands, likely causes, mitigation options, escalation contacts, and recovery verification. State which actions are read-only and which can change production state.

Prefer safe first steps

Start with confirming scope and collecting evidence. Avoid destructive cleanup, broad restarts, or database changes as default first actions. Include preconditions and rollback instructions for risky operations.

Keep instructions current

Runbooks should be reviewed after incidents and significant deployment changes. Test commands in a safe environment and remove steps that depend on obsolete tools.

Link to ownership

Each runbook needs an owning team and a way to reach the responsible on-call group. A document without an owner becomes stale quickly.

Checklist

  • Document symptoms and impact.
  • Separate diagnostics from mitigations.
  • Mark risky commands clearly.
  • Include recovery verification and escalation.
  • Review after incidents and architecture changes.
Share this page Help other developers discover this resource.