Incidents
An incident represents a single open problem raised by a rule for a specific resource. Incidents open when a rule’s condition is breached, recover automatically when things return to normal, and can be muted to suppress notifications during known work.
The Incident Lifecycle
- Open — a rule’s condition is breached, an incident opens, and the rule’s channels are notified.
- Ongoing — repeated breaches while the incident is open increment its occurrence count rather than sending a new notification, so you are not flooded while a problem persists.
- Recovered — on the next clean run for that resource, the incident recovers automatically and a recovery notification is sent.
You are notified only on a state change — when the incident opens and when it recovers — never repeatedly in between.
Viewing Incidents
Navigate to Alerts → Incidents. Open incidents are listed first, then recovered ones. You can filter by state (all, open, or recovered).
Each incident shows:
- The rule that raised it
- The severity (critical or warning)
- The resource it concerns
- Its state (open or recovered)
- When it opened
- How many times it has occurred
Muting
Muting suppresses notifications for an incident without closing it — useful during known maintenance or while you work on a fix. Mute an open incident for:
- 1 hour
- 1 day
- Indefinitely
A muted incident is clearly marked. You can unmute it at any time to restore notifications. Muting requires the alerting.write permission.
Permissions
alerting.read— view incidents.alerting.write— mute and unmute incidents.
Next Steps
- Rules — Tune what raises incidents
- Channels — Where incident notifications go
- In-Warehouse Log — Query the full history behind incidents in SQL