Dispense Confirmation at 4G Stores: A Consistency Trade-off
A national pharmacy chain operates 1,400 stores against a single central prescription database in one cloud region. Peak load is 380 dispense transactions per minute. Two requirements dominate the design: the pharmacist must see dispense confirmation within 2 seconds at the 95th percentile, and a controlled-substance prescription must never be dispensed twice within 24 hours. Most stores have fibre, but 6% run on 4G backup links with 600–900 ms round-trip latency. The current design uses a synchronous two-phase commit to the central database. Measured p95 confirmation is 4.1 s, and 4G stores time out roughly 30 times a day, each time forcing a pharmacist to use a manual override that is logged but never reconciled. The operations director proposes local caching with a 15-minute reconciliation window. The compliance officer says any window longer than zero is unacceptable. The chief pharmacist notes that 0.4% of dispenses are controlled substances.
1. Quantify the exposure created by the proposed 15-minute reconciliation window and compare it with the exposure the system already carries. What does the comparison imply for the decision?
Read the worked answer
Sizing the window. At 380 dispenses/min and 0.4% controlled, roughly 1.5 controlled dispenses occur per minute, so a 15-minute window contains about 23 controlled dispenses. That is the population at risk, not the expected number of double dispenses: a duplicate also requires the same prescription to be presented and accepted at a second location inside the window. Without the re-presentation rate, 23 is an upper bound and quoting it as a risk figure would be misleading.
Sizing the baseline. The current design already produces about 30 uncontrolled events per day - overrides made when the central check could not be reached. Those are unmeasured, unreconciled, and unbounded in duration. The compliance officer's 'zero window' describes the design intent, not the operating reality.
Implication. The decision is not zero versus 15 minutes; it is which mechanism yields fewer double dispenses. Compare measured duplicate-detection counts under both designs during a pilot, including overrides as exposure. Present that to compliance as a risk argument, escalated with data, rather than accepting or dismissing the absolute position.
2. Design a target architecture that satisfies both the 2-second p95 and the no-double-dispense rule. Explain its behaviour during a network partition at a 4G store.
Read the worked answer
Split by risk class. Only 0.4% of volume is controlled, about 1.5/min - small enough to spend latency on.
- Controlled substances: lease-based reservation. When the prescription enters the store queue, the store requests a 24-hour exclusive lease on the prescription ID from the central ledger. This is a single idempotent round trip keyed on prescription ID plus store ID. Dispense commits locally against a held lease, and no other store can obtain that lease.
- Everything else: asynchronous write with a durable local outbox and a client-generated idempotency key; confirmation renders optimistically against local state.
Why latency now fits. Worst-case 4G RTT is ~900 ms. A single round trip plus local commit stays inside 2 s; two-phase commit needs three or more round trips plus lock waits, which is exactly the 4.1 s observed.
Partition behaviour. If the store holds the lease, it dispenses. If it cannot reach the ledger, it cannot obtain a lease and must not dispense a controlled item - the prescription is deferred or routed to a connected store. The pharmacist override remains but becomes an explicit, alerted, post-hoc-reviewed path rather than a silent timeout side effect.
3. What fitness functions and evidence would you put in place to prove the design satisfies both requirements before national rollout?
Read the worked answer
Latency fitness function. p95 and p99 dispense-confirmation time, segmented by connectivity class, with a pass threshold of 2 s at p95 for every segment. Segmenting matters: an aggregate figure would hide the 4G population the design is meant to fix.
Correctness fitness functions.
- Count of controlled dispenses completed without a valid lease; target zero, alerting on any occurrence.
- Lease-acquisition failure rate and the resulting deferral rate per store.
- A nightly replay of the central ledger over a rolling 24 hours that searches for duplicate prescription IDs; this is the only direct test of the actual requirement.
Evidence before national rollout. Pilot on the 4G stores first, because they are the failure population and the cheapest place to be wrong. Inject partitions during the pilot and record what the terminals did. Produce a monthly reconciliation report, including overrides, signed by the chief pharmacist and accepted by compliance.
Decision rule agreed in advance. If lease deferrals exceed an agreed threshold in a store, the design is unworkable there and the store moves to a different connectivity tier - decided by data, not by argument at go-live.