Network
The slow dependency
Add latency to a service your checkout relies on. Watch timeouts, retries, queue depth, and whether the customer gets a useful answer.
⚠ Blog
Field notes from controlled chaos
Practical notes for finding weak points before customers find them for you. Start with a small failure, a clear hypothesis, and an abort condition.
Run your first GameDay
Pick one customer journey that matters, such as checkout, login, or data ingestion. Write down what should keep working when one dependency fails. That sentence is your hypothesis.
Choose the smallest fault that can test it. Add a hard time limit, a narrow blast radius, and an abort condition tied to a signal your team already watches. Brief everyone before the run, then record what the system did rather than what the diagram said it would do.
The useful result is not a dramatic outage. It is a specific gap you can fix, assign, and test again.
Three drills worth running
Cloud
The missing zone
Remove capacity in one availability zone. Confirm traffic moves, new instances start, and the remaining fleet does not collapse under retry load.
Data
The database failover
Trigger a controlled failover. Measure recovery time and check that connection pools, workers, and scheduled jobs reconnect without manual help.
External
The provider outage
Make a third-party API fail or return slowly. Confirm your fallback works and customers can still complete the parts of the journey you control.
What to write down
Capture the hypothesis, the exact fault, who could be affected, the signals you watched, the abort threshold, and the timeline. Record every surprise, including alerts that did not fire and dashboards nobody trusted.
Finish with owners and dates for the changes that matter. Then schedule the rerun. A GameDay becomes valuable when the lesson turns into a safer system, not when the report looks impressive.