The Template Edit That Took Down Four Stores
Northwind Retail runs 140 store networks plus one data center, all inside a single Meraki organization. Ninety stores are bound to a configuration template called Store-Standard; the remaining fifty were built ad hoc and never converted. On Tuesday a store manager reports that POS terminals on VLAN 40 lost their gateway, and within an hour three more stores in the same region report the identical symptom. The organization has four full admins, six network-limited admins, and a nightly Python job that reads device status with a service account API key. A firmware rollout was announced last week. In the incident call, nobody admits to changing anything, and two people suggest rolling back the firmware.
1. What do you check first, and in what order, to establish what actually changed and how far the blast radius extends?
Read the worked answer
Work from broadest scope to narrowest, because the symptom pattern (identical VLAN, four stores, one region) already argues against firmware and for a shared configuration object.
- Organization > Change log, filtered to the last 72 hours. This is the authoritative audit trail of dashboard and API writes, and it names the admin and the object touched. If the four affected stores share only one thing - the Store-Standard template - the change log will say so in one screen.
- Template binding view. Confirm which networks are bound and whether any were bound or unbound recently. Four stores failing out of ninety bound stores is a clue: check whether those four are on the same template revision, or whether the other 86 simply had not yet picked up the change.
- Per-device event log on one broken MX, filtered to configuration events, to confirm the local device saw a config push and at what timestamp.
- Firmware: check Organization > Firmware upgrades for the rollout schedule. If the upgrade window has not started, firmware is eliminated and you can say so in the incident channel with evidence.
Only after those four do you touch anything. The instinct to roll back firmware is the most common wrong move here - it is slow, it is high-risk, and it treats a symptom the change log can explain in two minutes.
2. The change log shows an org-wide admin edited the VLAN gateway on the Store-Standard template at 02:14. What is the safest remediation path, and how do you treat the fifty unmanaged stores?
Read the worked answer
Remediate at the template, not per network. Fix the VLAN 40 gateway value in the template and let the push propagate. Editing four networks individually would leave them divergent from the template, and the next template push would silently overwrite that local fix - the classic double-fault.
If propagation is slow or partial, verify per network rather than re-pushing repeatedly. If a store is truly wedged, unbind it, fix it, and rebind - but record that as a deliberate exception, because unbinding is the action that most often becomes permanent by accident.
The fifty ad-hoc stores are not the incident, but they are the finding. Do not convert them during an outage. Instead: sample three or four, diff their VLAN configuration against Store-Standard, and produce a conversion plan with a test store, a named owner, and a rollback step. Converting fifty production stores in the middle of an incident converts one outage into a portfolio of them. Frame the conversion as the follow-up remediation item with a date, so it does not evaporate when the incident closes.
3. What controls would have contained this to one store, and what does the org's admin structure tell you about the real gap?
Read the worked answer
Containment is not one control, it is a stack:
- Blast radius: a template change is inherently org-wide for bound networks, so the only containment is the scope of who can edit templates. Template write access should sit with a small number of people, not every full admin.
- Role design: four full admins plus six network-limited admins is a reasonable ratio, but the incident proves the full-admin set is too broad. Introduce a read-only tier for the people who only need visibility, and reserve full admin for the two or three who do design work. The departing-staff API key is a related smell - keys inherit the creating admin's privileges and outlive their employment unless explicitly revoked.
- Change timing and review: 02:14 edits are almost never urgent. A change window plus a second pair of eyes on template edits would have caught a wrong gateway value before it shipped.
- Detection, not just prevention: a webhook or a scheduled API poll comparing template revision against expected config would have surfaced the drift in minutes rather than at the first manager complaint. Prevention reduces incidents; detection reduces their duration, and you need both.
The real gap is that template write access was treated as a normal admin permission rather than a privileged one.