Maintenance

Create Alerts That Detect Symptoms Rather Than Noise

Alert on sustained user-impacting symptoms with clear ownership, severity, and response guidance.

An alert should prompt a person to take action. If a notification is merely interesting, it belongs on a dashboard rather than waking an on-call engineer.

Alert on symptoms

High error rates, sustained latency, unavailable critical workflows, and rapidly growing queue age are often more actionable than individual CPU spikes. Resource metrics still matter, but alert thresholds should reflect likely user impact or imminent exhaustion.

Use duration and burn rate

A short spike may not require intervention. Combine thresholds with duration or SLO burn-rate logic to reduce transient noise. Critical fast-burn conditions can have a shorter evaluation window.

Include response context

Every alert should identify the service, impact, severity, owner, dashboard, and first diagnostic steps. Route it to a maintained on-call path.

Review alert quality

Track false positives, missed incidents, acknowledgement time, and alerts that repeatedly produce no action. Tune or remove low-value rules.

Checklist

  • Alert on user-impacting symptoms.
  • Use sustained thresholds or burn rates.
  • Include owner and runbook links.
  • Test notification routing.
  • Review false positives and missed incidents.
Share this page Help other developers discover this resource.