Inventory Script That Returns a Different Number Every Run
You maintain a Python inventory script for 4,200 wireless access points behind a vendor REST API. Each call returns at most 500 objects plus a nextCursor field, and the API allows 100 requests per minute per token - a token your monitoring job also uses. Last night's run stored 3,700 devices, the log shows 12 HTTP 429 responses the script ignored, and 61 MAC addresses were stored twice. Rerunning produced a different total. The code loops page=1..20 with requests.get(url, params={'page': page}), no delay, and writes each response body to the database before checking response.status_code. Inventory accuracy is audited next week.
1. Two different defects explain the missing ~500 devices and the duplicates. Identify both and explain how each is produced by the code as written.
Read the worked answer
Defect 1 - writing before validating status. The body is persisted before status_code is inspected, so a 429 error payload (or a truncated body) is written as if it were device data, and the page it should have delivered is never retried. 12 rate-limit rejections, one of which cost a full 500-record page, matches the ~500 gap. Sharing the token with the monitoring job makes 429s a normal operating condition, not an anomaly.
Defect 2 - offset pagination over a moving dataset. The client asks for page=N, but the API's contract is a cursor. Any page-number scheme is positional: if records are added or removed between page 1 and page 2, everything shifts and objects land on two pages or on none. That is exactly the duplicate-MAC symptom (61 rows written twice) plus smaller gaps. It also explains non-reproducibility - the shift depends on how much changed during the run.
The two interact: every 429 that is ignored is a lost window during which the dataset can move, which widens both the gap and the overlap.
2. Rewrite the retrieval loop as pseudocode so the run is correct, polite to the rate limit, and resumable after a crash.
Read the worked answer
cursor = load_checkpoint() or None
while True:
resp = get(url, params={'limit':500, 'cursor':cursor}, timeout=30)
if resp.status_code == 429:
sleep(int(resp.headers.get('Retry-After', backoff())))
continue # never advance cursor on failure
if resp.status_code >= 500:
retry with exponential backoff + jitter, capped
resp.raise_for_status() # validate BEFORE parsing or writing
body = resp.json()
upsert_devices(body['items'], key='mac') # idempotent write
save_checkpoint(body['nextCursor']) # commit cursor with the data
cursor = body['nextCursor']
if not cursor:
break
throttle_to(rate_budget_below_100_per_min)
Key judgements: the status check moves ahead of the write so a bad body can never be persisted; the cursor only advances after a successful transactional write, so a crash resumes from a consistent point; the upsert keyed on MAC makes a replayed page harmless; and the token budget is split with the monitoring job, since a per-token limit is a shared resource. Add a final reconciliation that compares the stored count against the API's reported total, because cursor pagination can still drift.
3. A teammate argues: 'Cursor pagination is more code. Just use page numbers and set limit=500, it worked in the lab.' How do you respond?
Read the worked answer
The lab is the problem: with a static dataset, page numbers and cursors are indistinguishable, so the lab cannot produce the failure. Production is the opposite of a lab - the monitoring job is mutating the same token budget and the controller is adding and removing APs continuously.
The decisive point is contract ownership. Cursor is what the API documents and therefore what the vendor tests; page is undocumented behaviour that can vanish in a minor version without appearing in a release note. nextCursor also encodes a stable sort position, which is the only thing that survives concurrent modification.
Concede the real cost: cursors are opaque, so you cannot jump to page 40, and a long run still sees the dataset drift. Mitigate with a resumable checkpoint and a post-run diff against a lastUpdated filter, not by reverting to offsets. If the API genuinely offers no cursor, then pagination must be paired with an explicit sort key and a dedupe key, and that fact must be written down.