1
Choose a realistic failure
A slow dependency or a lost zone beats a made-up scenario.
⚠ Guide
An incident response drill is a rehearsal where you break something real, page the on-call team and run the full response, roles, channel, status page, while the stakes are zero. Unlike a tabletop exercise, it tests whether alerts fire and runbooks work, not just whether people can talk through them.
1
A slow dependency or a lost zone beats a made-up scenario.
2
Surprise drills train detection. Announced drills train the fix.
3
Let alerts fire through the normal path. Do not shortcut paging.
4
Incident channel, roles, status page draft, customer comms.
Alerts that go to someone who left. Runbooks with missing steps. Nobody sure who owns customer comms. These are cheap to fix after a drill and expensive to discover during an outage.
Monthly for on-call rotations, so every engineer practises at least once a quarter.
Mix both. Start announced so people learn the process, then add surprise drills to test detection.
Every fault Oh My GDaaS runs is aimed at infrastructure you have proved you own, through a DNS record, a scoped cloud role or a deployed agent. Every run has a blast-radius limit, a hard time limit and an abort button. We are not a booter, stresser or DDoS service and never will be.