May 5
Loading video player...
0 alerts firing. Everything green. WHY NOBODY KNEW: • Alerts on wrong endpoints — checking /health. app served /api. both different. • Thresholds set to never fire — error rate alert threshold: 101%. someone typo'd. • PagerDuty was muted — muted during deploy week. never unmuted. • Dashboards showing cache — metrics pipeline had been broken for 3 days. "AN UNTESTED ALERT IS JUST DECORATION" ───────────────────────────── Follow @devopswithkosa for weekly DevOps war stories. Real incidents. Real chaos. Real lessons. Subscribe for new episodes every week. ───────────────────────────── #DevOps #SoftwareEngineering #TechStories #CodeLife #SRE #Programming #DevOpsNightmares #TechTok #Monitoring #Grafana #PagerDuty #Observability #SRE #Alerting