Nightly User Sync Hits 429 and Duplicates
Your team runs a nightly Python job that reconciles an Okta org of 92,000 active users and 140 groups into a data warehouse. The job uses the Okta Python SDK, calls list_users() with the default page size, and follows next-page links sequentially. Around 01:00 it starts receiving HTTP 429 responses; the run logs show 38,000 users processed, then retries from the beginning because the script catches the exception and restarts the whole job. The next morning the warehouse has duplicate rows keyed by email. A separate onboarding batch creates 2,500 users one API call at a time; about 60 fail with 409 Conflict because rehires already exist under the same login, and operators rerun the entire batch. The org's rate limits are shared with interactive admin traffic and the vendor's SCIM connector. You need to make the sync correct, resumable, and safe to rerun without increasing admin friction.
1. What is the most likely reason for the 429s, and how should pagination and retry be changed? Be specific about Okta rate-limit headers and page size.
Read the worked answer
The 429s are almost certainly org-level rate limiting, not a pagination bug: the nightly job, the admin UI, and the SCIM connector draw from the same org limits, and a sequential full scan of 92,000 users at the default page size is a long, bursty request stream.
Set an explicit limit=200, the Users API maximum, and follow the Link header with rel=next rather than constructing offsets. Persist the next link before processing each page. On a 429, read X-Rate-Limit-Limit, X-Rate-Limit-Remaining, and X-Rate-Limit-Reset; sleep until the reset time plus jitter, then retry with exponential backoff and a cap. Do not restart the whole job after an exception; resume from the last committed link. For the nightly delta, avoid a full scan every night: use the System Log API for lifecycle events since a watermark, and reserve full scans for weekly reconciliation. Limit concurrency to one or two requests and add a jittered start so the job does not collide with the SCIM connector.
2. Design a safe idempotent approach for the 2,500-user onboarding batch, including handling 409 Conflict and externalId.
Read the worked answer
Treat each create as a deterministic upsert. Use the HR worker ID as externalId, not email, because email is mutable and rehires reuse it. Before creating, search by externalId and by login as a fallback, then decide: absent -> create with activate=false; present and active -> update the profile; present and deprovisioned -> reactivate and update. On a 409, do not retry blindly; fetch the conflicting user, compare the immutable HR ID, and either link or update that record.
Keep a per-record state table with states such as pending, created, linked, and failed so an operator rerun skips successes. If the endpoint supports an idempotency key, send a stable key derived from the HR event; otherwise rely on externalId plus the state table. Keep concurrency low, check rate-limit headers between batches, and activate only after the profile is correct. Emit an audit event for every link or reactivation decision so identity operations can explain why an account was reused.
3. How would you make the nightly reconciliation resumable and detect duplicates in the warehouse?
Read the worked answer
Checkpoint after every page by storing the exact next link, the timestamp, and the last Okta user ID processed; a resumed run starts from that link, not from page one. In the warehouse, make the Okta user ID the natural key and externalId the HR key; upsert on those keys so retries cannot create duplicate rows. Email should be an attribute, never the dedupe key.
Add a reconciliation report that compares counts by status and flags two Okta IDs sharing one HR ID. Run a full scan weekly in a low-traffic window and a System Log-based delta nightly. Alert on 429 rate, 5xx rate, checkpoint age, and duplicate HR IDs. Capture Okta request IDs from error responses so support can correlate failures with org-level events.