Permit Renewals Collapse at 09:00 on 1 March
A local council runs an online permit system for residents' parking, bulky waste collection and hall hire. Renewals open at 09:00 on 1 March. Last year, with roughly 1,200 users arriving within the first four minutes, the single web server's CPU hit 100%, all 12 PHP-FPM workers were busy waiting on MySQL, and the site returned 503s for 19 minutes. During that window the payment gateway callback fired twice for 89 transactions, so 89 residents were charged twice, and the retry button created 340 duplicate bookings. The current stack is one 4 vCPU, 8 GB VM running Apache, PHP 8.2 and MySQL 8 on the same host, with no queue, no CDN and a 60-second PHP max_execution_time. The council has 45,000 residents and a budget for one additional VM and a managed database. The mayor wants the same page to stay up on 1 March next year and to meet the accessibility obligations that apply to public-sector websites.
1. Diagnose the 09:00 failure and propose what to do with one extra VM and a managed database.
Read the worked answer
The numbers tell the story. Twelve PHP-FPM workers, each able to hold a worker for up to 60 s (max_execution_time), cap throughput at 12 requests per 60 s, about 0.2 per second, no matter how many users arrive. At 1,200 users in four minutes you need roughly 5 requests per second sustained. The server is not short of CPU in the abstract; it is short of concurrency, and every worker is waiting on MySQL on the same host, so CPU is spent context switching while the database waits on disk. The 503s were the correct response from a saturated server; the real failure is that nothing absorbed the queue.
With one extra VM and a managed database:
- Move MySQL to the managed service, removing shared-host contention and giving managed backup and failover.
- Put the two web VMs behind the existing load balancer, both stateless (sessions in Redis or the database, not on disk).
- Size PHP-FPM workers by RAM, not CPU: roughly (available RAM minus OS) divided by average worker size.
- Cache static assets and the permit-type list; most of the 1,200 arrivals hit the same few pages.
- Queue payment callbacks and anything that need not answer synchronously.
- Cheapest control: a virtual waiting room admitting users at a rate the system can serve. That is a design decision, not a server one, and it fits the budget.
Do not simply add vCPU; the bottleneck moves with it.
2. The payment gateway callback fired twice for 89 transactions and the retry button created 340 duplicate bookings. Explain the mechanism and how you would design against both.
Read the worked answer
These are two duplicate paths and are often confused.
Gateway callback: the gateway posts the result to the council endpoint; if that endpoint is slow because all workers are busy, the gateway times out and retries. The first callback may still be running, so both are processed and both charge. Retrying is correct behaviour by the gateway; the endpoint must be safe to call more than once.
Retry button: the resident sees a 503 or a spinner, presses again, and the first request may still complete, creating two bookings for one intent.
Design:
- Idempotency key. Generate a key when the booking is created (booking reference plus attempt number) and require the gateway to echo it. The handler inserts with a UNIQUE constraint on the gateway transaction id, or does a select-for-update then insert in one transaction; a duplicate hits the constraint and returns 200 as already processed without charging. A unique constraint on (resident, permit_type, period) prevents the second booking row.
- Make retry idempotent: resubmit the same key so the server returns the existing result rather than creating a new record.
- Shorten the synchronous path: acknowledge the callback, write to a queue and return 200 in milliseconds.
- Reconcile daily against the gateway settlement file and refund the 89 automatically rather than waiting for complaints.
The principle: any endpoint an external system or a user can call twice must produce the same result twice.
3. Set out the testing you would require before 1 March, including accessibility, and state what would stop go-live.
Read the worked answer
Load test shaped like the real event:
- Ramp to 1,500 virtual users over four minutes starting at 09:00, not a steady 1,500; the arrival pattern is the problem.
- Measure p95 response time and error rate, not averages, which hide the queue.
- Test the worst case: duplicate payment callback, retry pressed, and a database failover mid-test.
- Use a staging environment with production-like data volume; 100 rows proves nothing about a 45,000-resident table.
- Repeat with the waiting room enabled to prove the queue drains and nobody waits indefinitely.
Accessibility (public-sector sites must meet WCAG 2.2 AA):
- Complete a renewal keyboard-only, with visible focus.
- Screen-reader test of the form and error messages: errors identified in text and linked to fields, not only coloured red.
- Contrast, text resize to 200%, and no time limit on the form or a means to extend it. A waiting room with a countdown can breach this, so check it.
- Test with an assistive-technology user, not only an automated checker.
Stop-go criteria agreed in advance: any p95 above the agreed threshold at peak, any duplicate charge in testing, any accessibility failure on the renewal path, or no tested failover. If 1 March cannot be met with those passed, move the date, not the criteria; a second year of double charges costs more than a two-week delay.