The Hotfix That Bypassed the Pipeline
A payments service deploys via a pipeline that builds an immutable artifact, runs integration tests, and promotes the same artifact from staging to production. During a Friday evening outage, an on-call engineer edits a configuration file directly on the production host and restarts the process; service recovers in 8 minutes. The change is never recorded in version control. Two weeks later, a routine deploy from the pipeline overwrites the host, the outage returns at 2 a.m., and the responder cannot find why production 'worked' before. Investigation takes 3 hours. The team has 40 hosts, deploys 12 times a week, and has no configuration drift detection. Leadership wants a rule banning all manual production changes.
1. Assess whether a blanket ban on manual production changes is the right response. What would you put in place instead?
Read the worked answer
A blanket ban is the wrong lesson. The engineer's action was rational under pressure: the pipeline could not deliver a config-only fix in 8 minutes, so the only fast path was manual. Banning it removes the safety valve without removing the reason people reach for it, and it will simply push changes underground where they are even less visible.
Better design:
- Make the fast path legitimate. Support a config-only change that goes through the same artifact and promotion path but skips the slow integration stage, with automated approval. Target time-to-prod under 15 minutes for low-risk changes.
- Break-glass procedure. Allow direct changes only through an audited channel: a logged session, an automatic ticket, and a mandatory follow-up commit within 24 hours that reconciles the change into version control.
- Drift detection. With 40 hosts and no drift detection, the team is blind. Run periodic reconciliation that compares actual host state to declared state and alerts (or auto-remediates) on divergence. This alone would have caught the Friday edit within minutes and prevented the 3-hour archaeology.
- Immutability. Hosts should be replaced, not edited. If a config change requires a host edit, the process is broken.
- Blameless follow-up. The 3-hour investigation is the real cost; capture why the manual change was invisible and fix that, not the person.
2. The team deploys 12 times a week across 40 hosts. How would you reason about the risk of the overwrite recurring, and what measurable controls would you add?
Read the worked answer
The recurrence risk is high because the mechanism that caused it (undetected drift plus pipeline overwrite) is structural, not a one-off. Twelve deploys a week means roughly 24 opportunities in the two weeks before the second outage for the pipeline to clobber an unrecorded change. The probability of an unrecorded change existing at deploy time is what matters, and with no drift detection it is effectively unknown.
Controls, each measurable:
- Drift detection coverage: percentage of hosts reconciled against declared state, target 100%, checked at least hourly. Alert on any divergence.
- Mean time to detect drift: should be minutes, not the two weeks seen here.
- Unmanaged change count: number of production changes not traceable to a commit, target zero, reviewed weekly.
- Break-glass usage: count and duration of emergency changes; a rising trend signals the normal path is too slow.
- Deploy-time preflight: before overwriting a host, compare live state to expected state and refuse or warn if they differ. This is a cheap guard that directly prevents the observed failure.
- Post-incident metric: time to root cause during incidents; the 3 hours here is the number to drive down via better change history.
The reasoning is that you cannot ban your way out of a structural gap; you shrink the window in which an invisible change can exist and make the window itself observable.