Retry Amplification at the Timeout Boundary
Checkout calls inventory over HTTP. Checkout's client timeout is 300 ms, with 2 retries and 100 ms backoff. Inventory has 20 pods across 3 AZs; its p99 is 120 ms. The team wants to test whether inventory latency causes checkout failure. A proposed fault injects 400 ms latency into 30% of inventory pods in a single AZ for 10 minutes. Checkout sends 1,200 rps; inventory is also used by search, which has a 2 s timeout. The platform team can target pods by label, namespace, and node; the cluster has a canary namespace with 2 inventory pods. The team debates using 400 ms vs 800 ms, and whether to target 30% of pods or 30% of traffic. They also plan to keep retries enabled to 'be realistic.' The experiment must not affect search.
1. Design the fault injection: fault type, target, magnitude, duration, and blast radius. Justify each choice.
Read the worked answer
Use a network latency fault on checkout pods' egress to the inventory service IP/port, not on inventory pods. If latency is injected on inventory pods, search also sees it because it shares the same service. Target 10% of checkout pods (for example 2 of 20) in one AZ for 10 minutes. Start at 200 ms, which is below the 300 ms timeout, to observe degradation; then move to 400 ms to test the timeout boundary. 800 ms is not the first choice because every attempt times out and the experiment becomes an outage test. Keep the blast radius to 2 checkout pods and one AZ, exclude search by namespace/label, and abort if checkout 5xx >1% or p99 >1 s. This isolates the caller behavior and protects unrelated tenants.
2. Explain retry amplification and whether keeping retries enabled is the right experimental control.
Read the worked answer
With a 300 ms timeout and 2 retries, checkout makes up to 3 attempts per request. At 400 ms inventory latency, every attempt exceeds the timeout, so each request holds a thread for roughly 900 ms plus backoff. At 1,200 rps this can exhaust the checkout thread pool and turn a latency fault into a cascading outage. Keeping retries enabled is realistic but confounds the variable: you cannot tell whether failures come from latency or retry amplification.
Better sequencing: run A with retries disabled to measure the pure latency effect, then run B with retries enabled to measure amplification, or test 200 ms first. If only one run is possible, keep retries but set abort conditions on thread-pool saturation, queue depth, and retry counts. The decision should be explicit, not hidden inside 'realistic' defaults.