Anti-Fragility and Learning from Failure
Designed for failure
Complex systems fail. Anti-fragile practice assumes failure is normal and engineers for graceful degradation: redundancy without shared fate, load shedding and admission control, timeouts, retries with exponential backoff and jitter, circuit breakers, and the ability to serve a degraded but useful experience. Capacity planning and non-abstract large system design thinking matter here: know what breaks first and how the system behaves at the limit.
Learning from incidents
Blameless postmortems are the core learning mechanism: a factual timeline, impact measured against the SLO and error budget, contributing factors, and - crucially - owned, dated, prioritised action items. ‘Root cause’ is usually a trap: incidents have multiple contributing factors, and stopping at ‘human error’ stops the analysis where it should begin. Ask what made the error easy, likely and undetected.
Chaos engineering and game days (failure injection, disaster recovery tests, Wheel of Misfortune) turn hypotheses about resilience into experiments, preferably in production with a controlled blast radius and an abort condition.
Traps
- A postmortem with no action items, or action items nobody owns.
- Blaming an individual, which guarantees the next incident is hidden.
- Confusing MTTR and MTBF improvements without asking which failure modes they actually cover.
- Chaos experiments without a hypothesis, a steady-state definition, or a kill switch.
Worked example
Trigger a formal postmortem whenever an incident consumes more than a set share of the error budget (e.g. 20%) or any customer-visible data loss occurs. Review action items in the next reliability planning cycle; track the percentage completed on time as a health metric for the learning culture itself.