Order sync: retry storms, duplicated POSTs, pagination and token refresh
OrderSync is a Python 3.11 service that pulls orders from a vendor's REST API and posts shipment confirmations back. It uses one global requests.get()/requests.post() call per operation, with no timeout, no Session, and a hand-written retry loop of five immediate retries on any exception. The vendor allows 100 requests per minute per token and returns 429 with Retry-After: 30; after the last deployment the service got 429 five times in a row and then a 15-minute IP block. During the same run, 37 shipment confirmations were duplicated: the client retried after ReadTimeout, but the vendor had already created the shipment. Pagination uses Link: <https://api.vendor.test/orders?page=2&per_page=100>; rel="next" and X-Total-Count: 25000. Access tokens expire after 3,600 seconds; four worker threads share one token and all refresh at once, causing 401 storms. Logs show single requests hanging for 60+ seconds. The team wants a resilient client without changing the vendor API.
1. Design the timeout, session and retry configuration. Explain why the current retry loop caused the IP block, and which methods should and should not be retried automatically.
Read the worked answer
First, every request needs a timeout; without one, a hung socket can block a worker for the OS TCP timeout, often minutes. Use a connect/read tuple, e.g. timeout=(3.05, 30). Second, use one requests.Session per thread, or per process, and mount an HTTPAdapter with urllib3.util.retry.Retry so connection errors and retryable statuses are handled consistently.
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
retry = Retry(
total=5, connect=5, read=2, status=5,
backoff_factor=0.5,
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=frozenset(['GET', 'HEAD', 'PUT', 'DELETE']),
respect_retry_after_header=True,
raise_on_status=False,
)
session.mount('https://', HTTPAdapter(max_retries=retry))
The old loop retried immediately, five times, on every exception including 429. That does not respect Retry-After; it amplifies load and triggers the IP block. respect_retry_after_header=True makes urllib3 wait for the server's Retry-After value, whether it is a number of seconds or an HTTP-date. backoff_factor adds exponential waits for other statuses. Do not put POST in allowed_methods: a read timeout on POST is ambiguous, so application-level idempotency is required. Also call response.raise_for_status() for non-retryable 4xx and log the request id and Retry-After header.
2. How do you eliminate the 37 duplicate shipment confirmations? Give the protocol and the client-side checks, and explain why a `ReadTimeout` is ambiguous.
Read the worked answer
A ReadTimeout means the request may have reached the server and been committed; the response was lost. Retrying the same POST without a deduplication mechanism creates a second shipment. The fix is an idempotency key generated once per logical shipment, persisted before the first attempt, and reused on every retry with the identical body:
key = shipment.idempotency_key # UUID created once, stored locally
resp = session.post(url, json=payload,
headers={'Idempotency-Key': key},
timeout=(3.05, 30))
The vendor must deduplicate on that key for a documented window, commonly 24 hours, and return the original response or a 409 with the existing id. If the vendor supports client-assigned IDs, PUT /shipments/{client_id} is even cleaner because PUT is idempotent by definition. Do not generate the key inside the retry helper; that defeats the purpose. Do not retry non-idempotent POSTs in the adapter. On ambiguous failure, either retry with the same key or reconcile with GET /shipments?external_id=... before creating again. Add a unique constraint locally on (order_id, shipment_seq) so two workers cannot create two logical shipments in the first place. Test by injecting a read timeout after the server commits.
3. Handle pagination and shared-token refresh for 25,000 orders without hammering the API. What termination condition should the paginator use, and how do four threads avoid a refresh storm?
Read the worked answer
Follow the Link header rather than computing page=n. Parse it with a small helper, find rel='next', and stop when it is absent; that survives changes to page size and cursor-based pagination. Use the same session and timeout/retry policy for each page. 25,000 records at per_page=100 is 250 requests; at 100 requests per minute that is at least 2.5 minutes of wall time before retries, so the job must be resumable, not a single 30-second request. Persist the last successful page URL or cursor and the count of processed records; on restart, resume from there. Respect X-Total-Count for progress, but do not treat it as a termination condition - it can change during the run.
For tokens: store access_token and an absolute expires_at, using a monotonic clock or UTC with skew. Before a request, if expires_at - now < 60, refresh. Protect refresh with a single threading.Lock and double-check the token after acquiring it, so four threads do not all refresh. On a 401, refresh once, retry that one request once, and if it fails again surface an auth error rather than looping. Do not refresh on every 401 from every thread. Log token expiry and 401 counts; a refresh storm usually means the expiry check is missing or the clock skew is wrong.