Rolling Out a POS Release Across 2,300 Stores

Use expand–contract, and keep the tax logic change in a separate step from the schema change. Combining them is what made the last release unrecoverable.

Sequence:

1. Expand: add the new column as nullable, no table rewrite, no default backfill. Deploy. Old terminals are unaffected because nothing reads it.
2. Dual write: new code writes both old and new representations and reads the old one. The tax service sits behind a feature flag.
3. Backfill in batches, monitored, with the job restartable.
4. Switch reads to the new column behind a flag, verify, then make it authoritative.
5. Stop writing the old representation.
6. Contract: drop the old column in a later release, only after the last old terminal version is retired plus a retention window.

Offline protection. Terminals generate a client-side transaction UUID; the server enforces a unique constraint, so replay is idempotent rather than additive - the specific failure that double-posted loyalty points.

Tax reproducibility. Stamp the tax rate version onto each transaction and retain the rate table historically, so any sale can be recomputed with the logic in force at the time.

Rollout. Big-bang is the wrong shape for 2,300 stores with a history of version-mixing failures. Pilot with roughly 40 stores, including the stores that failed last time, then wave by terminal hardware version and connectivity quality. Run old and new versions concurrently by design - expand–contract exists to make that safe, and the objection to running two versions is an objection to a schema design, not to phasing.

Rollback in under 30 minutes is only credible if rollback is a flag flip plus redeploy of the previous application version, with no destructive migration having run. That is why the column drop is deferred to a separate later release: after a drop, rollback is not 30 minutes, it is a restore.

Triggers:

• Sync error rate above 0.5% in a store cohort.
• Any confirmed duplicate transaction or double-posted loyalty entry.
• Tax variance beyond 1 in 10,000 sampled transactions.
• Local queue depth growing across two consecutive checks.

Degradation budget. Track degraded minutes per store against the two-hour limit; the pilot cohort definition should exclude stores already close to it.

Offline protection. The terminal is the system of record until it syncs, so the design must make replay harmless:

• Client-generated transaction UUID, unique constraint server-side, so a retried sync is rejected as a duplicate rather than posted again.
• The transaction carries the tax rate version in force at the moment of sale; the server validates the version against the transaction timestamp and recomputes where needed.
• Rejections go to a reconciliation queue with an owner - never dropped silently, because a dropped sale is worse than a duplicate.
• Terminal version negotiation on sync, so a terminal two versions behind is refused cleanly rather than half-parsed.

First 72 hours, monitor:

• Sync success rate and p95 queue age by store.
• Duplicates blocked at the constraint, and - separately - duplicates that got through; the second number must be zero.
• Tax version mismatches and their resolution time.
• Loyalty postings reconciled against sales totals daily.
• Degraded minutes per store versus the two-hour commitment.

Report once daily to store operations and finance, with the numbers, not a status adjective. The previous incident was discovered by customers; the point of the dashboard is to find the next one first.