Observability
Alert Fatigue: How to Stop the Noise and Start Solving Incidents
Don't let your team ignore critical outages. Learn the strategies to combat alert fatigue and how Watch.dog helps you create high-signal notifications.
By Watch Dog TeamPublished May 12, 202511 min read
The Boy Who Cried Wolf
Symptom Log
noisy_slack.slack
[10:00] Alert: Latency spike (50ms above normal)
[10:01] Alert: Minor CSS 404.
[10:05] Alert: Disk at 80%.
# RESULT: The engineer is no longer reading the messages.If a developer receives 50 Slack notifications a day from a monitoring tool, they will eventually 'mute' the channel. This is Alert Fatigue. Then, when a real database failure happens, the crucial alert is buried under 49 'False Positives.'
A successful monitoring strategy doesn't just alert on failure; it alerts only when human intervention is required.
The High-Signal Fix
Use Watch.dog Advanced Thresholds. Configure your alerts to only trigger if a failure persists for more than 5 minutes or across 3 different global regions.
Fix Verification
clean_alerts.log
[INFO] Minor latency detected from NYC node. (Muted: Transient)
[CRITICAL] Error 500 detected in 4+ Regions (Triggering Escalation).
[SUCCESS] Team only notified when a systemic failure occurred.Priority-Based Alerting
Not all services are equal. Watch.dog allows you to categorize your monitors, ensuring that a 'Checkout Failure' wakes up the engineer via phone call, while a 'Dev-Site Failure' just sends a Slack ping.
Alert Hierarchy Model
| Severity | User Impact | Watch.dog Notification |
|---|---|---|
| Critical (P1) | Site DOWN for all users | Phone Call + SMS + Slack |
| Warning (P2) | Performance Degradation | Priority Slack Channel |
| Info (P3) | Background Task Error | Daily Summary Email |
In SRE, quiet is the ultimate sign of success.