Nightly CSV Sync, API Rate Limits, and Terraform Drift
A retail company has 31,000 employees and 12,000 contractors. HR exports a full CSV to an SFTP server every night at 01:00. An Okta Workflows flow picks up the file and calls the Users API to create or update profiles, then assigns groups. The org is hitting API rate limits: 429 responses peak at 1,200 per hour, and the flow retries immediately, causing a cascade. Some users are created twice because the flow searches by email, and email changes after marriage. The company also uses Terraform to manage 180 app assignments and group rules; a recent manual change in the admin console was overwritten by the next Terraform apply. The team wants near-real-time terminations from HR and a reliable nightly sync without duplicates.
1. Diagnose the duplicate users and the 429 cascade. What changes make the nightly sync idempotent and rate-limit-safe?
Read the worked answer
The duplicates come from using email as the search key. When an email changes, the search misses the existing user and the flow creates a new one. Use an immutable HR identifier, such as employeeNumber, as the Okta externalId and search key. Make the flow create users inactive, search by externalId, and only activate after a successful match or create. Never search by display name or email for identity decisions.
The 429 cascade comes from immediate retries and uncontrolled concurrency. On a 429, read Retry-After or X-Rate-Limit-Reset and back off exponentially with jitter. Cap concurrency, process in batches, and do not retry the whole file because one record failed. Reduce the nightly volume by sending only deltas from HR, or by comparing the CSV to Okta before calling the API.
For the existing duplicates, export users with duplicate employeeNumber or externalId, identify the account that applications reference, and deactivate or merge the other record under change control. Add a pre-flight check that fails the flow if the CSV contains duplicate employeeNumbers or missing required attributes.
2. HR only produces a nightly CSV, but the business wants near-real-time terminations. What architecture would you propose, and what tradeoffs would you state?
Read the worked answer
A nightly file cannot deliver near-real-time termination. State that clearly rather than promising a 5-minute SLA from a 01:00 batch. Propose an event push from HR: when HR terminates a worker, it calls a webhook or API endpoint that triggers an Okta Workflows flow to deactivate the user and remove assignments immediately. If HR cannot push events, ask for a termination-only delta file every 5 minutes, separate from the full nightly file, and process only those records. Keep the nightly full sync as reconciliation, not as the primary termination path.
Tradeoffs: event push adds an integration dependency and requires HR to guarantee delivery and retries. A 5-minute delta file is simpler but still has a window. Both need idempotency, because the same termination event may arrive twice. Also shorten application session lifetimes for high-risk apps so a missed event does not mean indefinite access.
Measure the result with a timestamp comparison: HR termination time, Okta deactivation event time, and app access removal time. Report the p95, not just the average, and alert when the gap exceeds the agreed SLA.
3. How should Terraform and manual admin changes be governed so the next apply does not silently overwrite an emergency fix?
Read the worked answer
Declare Terraform the source of truth for the 180 app assignments and group rules, and prohibit manual changes to those objects except through a documented break-glass process. Put Terraform state in remote storage with locking, and run plan in CI so every change is reviewed before apply. Use separate workspaces or state files for production and non-production, and enable prevent_destroy on critical assignments.
For existing resources created manually, use terraform import rather than recreating them. After import, run plan and reconcile differences deliberately. If an emergency manual change is required, the operator must open a follow-up change to codify it in Terraform within an agreed window; otherwise the next apply will revert it. Add drift detection that runs plan on a schedule and alerts when the plan is non-empty.
Do not put everything in Terraform just because it is available. Keep high-churn user lifecycle in Okta Workflows or the HR integration, and keep Terraform for stable configuration such as app assignments, group rules, and policy settings. This boundary prevents a user termination from waiting on a Terraform apply.