Work / 03 · In production
Collapses alert storms into one triage queue.
- services
- 185+services
01
Problem, approach, result
Problem
- Monitoring that alerted on everything and explained nothing. One fault arrives as a storm of pages.
Approach
- Collapse correlated alerts into a single triage queue, across 185+ services. Built in month 2 of the internship.
Result
- In production.
“47 simultaneous alerts about the same thing isn't 47 problems.”
02
How it works
Forty pages about one fault collapse into one queue item.