Applicability
When to Use
✓When critical issues need guaranteed human attention
✓When different severity levels need different response teams
✓When SLAs require guaranteed response times
Overview
How It Works
The Alerting Cascade pattern routes alerts through increasing severity levels with automatic escalation. First, a Slack message notifies the team. If not acknowledged within 5 minutes, PagerDuty creates an incident for the on-call engineer. If still unacknowledged after 15 minutes, it escalates to the engineering manager.
This pattern ensures critical issues are never missed, even during off-hours. The escalation timer runs independently, checking acknowledgment status at each interval.
Implementation
Code Example
typescript
async function alertWithEscalation(alert) {
// Level 1: Slack notification
const msg = await slack.sendMessage({ channel: "#alerts", text: `${alert.severity}: ${alert.message}\nReact with :white_check_mark: to acknowledge` });
// Level 2: PagerDuty after 5 min if not acknowledged
await sleep(5 * 60 * 1000);
const reactions = await slack.getReactions({ channel: "#alerts", timestamp: msg.ts });
if (!reactions.some(r => r.name === "white_check_mark")) {
await pagerduty.createIncident({ title: alert.message, urgency: "high", service_id: ON_CALL_SERVICE });
// Level 3: Manager escalation after 15 more min
await sleep(15 * 60 * 1000);
const incident = await pagerduty.getIncident({ id: incidentId });
if (incident.status !== "acknowledged") {
await pagerduty.escalateIncident({ id: incidentId, escalation_level: 2 });
await slack.sendMessage({ channel: "#engineering-leads", text: `UNACKNOWLEDGED: ${alert.message}` });
}
}
}Quick Info
Categorymonitoring
ComplexityMedium