Skip to Content
AlertingIncidents

Incidents

An incident represents a single open problem raised by a rule for a specific resource. Incidents open when a rule’s condition is breached, recover automatically when things return to normal, and can be muted to suppress notifications during known work.

The Incident Lifecycle

  1. Open — a rule’s condition is breached, an incident opens, and the rule’s channels are notified.
  2. Ongoing — repeated breaches while the incident is open increment its occurrence count rather than sending a new notification, so you are not flooded while a problem persists.
  3. Recovered — on the next clean run for that resource, the incident recovers automatically and a recovery notification is sent.

You are notified only on a state change — when the incident opens and when it recovers — never repeatedly in between.

Viewing Incidents

Navigate to Alerts → Incidents. Open incidents are listed first, then recovered ones. You can filter by state (all, open, or recovered).

Each incident shows:

  • The rule that raised it
  • The severity (critical or warning)
  • The resource it concerns
  • Its state (open or recovered)
  • When it opened
  • How many times it has occurred

Muting

Muting suppresses notifications for an incident without closing it — useful during known maintenance or while you work on a fix. Mute an open incident for:

  • 1 hour
  • 1 day
  • Indefinitely

A muted incident is clearly marked. You can unmute it at any time to restore notifications. Muting requires the alerting.write permission.

Permissions

  • alerting.read — view incidents.
  • alerting.write — mute and unmute incidents.

Next Steps

  • Rules — Tune what raises incidents
  • Channels — Where incident notifications go
  • In-Warehouse Log — Query the full history behind incidents in SQL
Last updated on