LFCA: Burstable Instance Degrades Every Morning at Ten

Burstable Instance Degrades Every Morning at Ten

A team lifts a Java order-intake service from a 4 vCPU / 8 GiB on-premises VM to a single AWS t3.medium (2 vCPU, 4 GiB) in one Availability Zone, leaving PostgreSQL on a t3.small in the same subnet. CloudWatch shows the instance at 90-98% CPU between 08:00 and 19:00, idle overnight, plus a reconciliation batch at 02:00 that pins both vCPUs for about an hour. p95 latency is 210 ms at 08:00 and 900 ms by 10:10, then stays elevated until evening; p99 shows 4-second spikes. CPUCreditBalance reads 0 from about 10:10 onward and CPUSurplusCreditBalance is 0; the credit option is the default standard. The JVM runs with -Xmx3g and the AMI includes a 2 GiB swap file on gp3 EBS. Developers insist "the old 4 vCPU box was fine, so it must be the network". Finance approved about $75 per month per app instance and wants the smallest change that removes the unpredictability.

1. Using credit accounting, explain why performance is normal at 08:00 and degraded by 10:10. Show the arithmetic and name the two metrics you would pull to prove it.

Read the worked answer

A t3.medium earns 20% of a vCPU per vCPU, so its baseline is 2 × 20% = 0.4 vCPU, i.e. 24 CPU credits per hour, with a maximum balance of 576.

Demand: 95% of 2 vCPU ≈ 1.9 vCPU = 114 vCPU-minutes per hour. Net: 24 earned − 114 consumed = −90 credits per hour while under load.

Overnight 19:00-08:00 is 13 idle hours → about 312 credits earned, less roughly 120 burned by the 02:00 batch → about 190 credits at 08:00. At −90/hour that lasts a little over two hours, which puts exhaustion at roughly 10:10 - exactly when latency steps up. After the balance hits zero, the instance is hard-capped at 0.4 vCPU regardless of demand, so request queues grow and p95 triples.

Proof metrics: CPUCreditBalance (the sawtooth that reaches zero) and CPUSurplusCreditBalance (stays zero, confirming standard mode is dropping demand rather than accruing a bill). Pair them with CPUUtilization to show demand is unchanged while throughput falls.

2. The team proposes switching the instance to the unlimited credit option. Evaluate that against two alternatives, with cost estimates, and recommend one.

Read the worked answer

Unlimited credits: stops the cliff immediately. The shortfall is about 90 credits/hour ≈ 1.5 vCPU of surplus; at roughly $0.05 per surplus vCPU-hour, 15 hours a day for 22 working days is roughly 495 vCPU-hours, about $25 per month on top of the ~$30 instance cost. It is a legitimate stop-gap, but it buys variable cost, leaves the design dependent on surplus capacity, and does nothing about memory - 3 GiB heap plus OS on 4 GiB forces swapping to EBS, which is the more likely cause of the 4-second p99 spikes.

Larger burstable (e.g. t3.large): more credits per hour and more RAM, but still a burstable model; the same arithmetic repeats at a higher baseline.

Fixed-performance instance (m5.large, 2 vCPU / 8 GiB, ~$70/month; m6g.large Graviton ~$55-60 if the JVM and agents support arm64): predictable CPU, doubles RAM, fits the $75 budget.

Recommendation: enable unlimited today to stop user-visible degradation, then move to m6g.large (or m5.large if arm64 is a problem) after confirming with vmstat/sar swap-in rates that memory pressure is real. Validate with a one-week canary rather than a big-bang cutover.

3. What else in this design will cause a future incident, and what is the cheapest fix for each?

Read the worked answer

The database is also burstable. t3.small has the same 24-credits-per-hour-per-vCPU model. A nightly batch plus daytime traffic can exhaust it, and a throttled database degrades every client at once. Cheapest fix: monitor its CPUCreditBalance now, and either move to a fixed-performance class or schedule the reconciliation batch off-peak.
Single Availability Zone, single instance. One AZ event or one instance failure is a full outage with no replica. Cheapest meaningful mitigation: automated snapshots plus a documented, rehearsed restore into another AZ - not "we have backups", but "we restored in under an hour last quarter".
Swap on network-attached storage as a design element. Swapping to EBS turns memory pressure into latency spikes. Fix the instance size rather than tuning swap.
No load baseline. The "it's the network" claim is unfalsifiable without per-endpoint timing. Cheapest fix is structured request logs with upstream response times, which also settles future arguments in minutes.