Bulk Device Tag Automation Hits API Limits
A team runs a nightly PowerShell script against the Workspace ONE UEM REST API to assign tags to Windows devices based on AD group membership. The script reads 6,000 devices, computes 1,200 tag changes, and applies them one call at a time. It authenticates with a shared admin username and password stored in the scheduled task. When the change volume grew to 4,500, the API returned HTTP 429 and intermittent 400 errors; about half the changes applied, no retry logic exists, and the run takes over six hours. A second run the next night produces duplicate tag assignments and sometimes removes tags added manually by the service desk. The security team requires that no interactive admin credentials be stored on the server, and that all changes be attributable to a named service identity. The script must finish within a four-hour maintenance window.
1. Redesign the automation for reliability, rate-limit handling, and a four-hour completion target. What changes matter most?
Read the worked answer
Measure before rewriting. Log per-call latency and count calls by endpoint. A six-hour run for 4,500 changes suggests serial calls plus retry-free failures, not raw API slowness. If the average call is two seconds, serial execution cannot meet four hours at 4,500 changes, so concurrency and batching are required.
Design changes.
- Replace one-call-per-change with bulk or batch endpoints where the API supports them; otherwise use bounded concurrency of four to eight workers, not unbounded parallelism.
- Honor HTTP 429 with exponential backoff and jitter, and read
Retry-Afterwhen present. Treat 400 as a data error and log the device, payload, and response; do not retry it blindly. - Make each operation idempotent: read current tags, compute add/remove deltas, and apply only the delta. A second run should produce zero changes.
- Checkpoint progress to a local database or JSON file after each batch so a failed run resumes rather than restarts.
- Use an incremental lookup: cache device and tag IDs, and only refresh what changed since the last run.
- Add a dry-run mode that outputs the planned delta count before applying anything.
Four-hour target. Estimate throughput from the measured per-call latency. If concurrency of six with batching still exceeds four hours, split the workload by organization group and run two workers against disjoint device sets. Do not simply increase concurrency without monitoring for 429s; that trades one failure mode for another.
2. How would you satisfy the security requirement for credential handling and attributable changes?
Read the worked answer
Remove the stored admin username and password entirely. Use OAuth 2.0 client credentials with a dedicated API service account that has only the permissions required to read devices and manage tags. Store the client secret in a secrets vault or, better, use certificate-based client authentication so no reusable password exists on the server. Grant the service account a named identity that maps to a real owner in the identity system, and record that owner in the runbook.
Attribution. The API audit log should show the service account as the actor for every change. Add a correlation ID per run and include it in script logs alongside the device ID, tag ID, action, and result. If the API supports custom headers or change reasons, pass the run ID so a tag change can be traced back to a specific execution. Forward logs to the SIEM and retain them for the audit period.
Least privilege. The service account should not be a full administrator. Test with a role that can manage tags and read device records but cannot wipe devices or change compliance policy. If the API does not offer a narrowly scoped role, document the gap and add compensating monitoring for actions outside the script's expected scope.
Secret rotation. Automate rotation and alert on failed authentication so an expired secret does not silently stop the nightly run. The scheduled task should run under a managed service account, not a personal login.
3. How do you make tag changes idempotent and protect manually applied service-desk tags?
Read the worked answer
Namespace the automation. Define a tag prefix or tag group that the script owns, for example auto-ad-*. The script may add, remove, or reconcile only tags in that namespace. Any tag outside it is treated as manually managed and is never removed. This is the simplest control that prevents the script from deleting service-desk work.
Compute a desired state. For each device, build the desired set of owned tags from AD group membership, then compare it with the current set returned by the API. Apply add and remove operations only for the difference. If the current state already matches, make no call. This makes reruns safe and eliminates duplicate assignments.
Handle conflicts explicitly. If a service-desk tag and an owned tag express the same intent, do not let the script silently override the manual tag. Log a conflict and route it to a review queue, or define precedence in the runbook. If the API allows multiple tags with the same name, deduplicate by tag ID, not by display name.
Verification. After the run, re-read a sample of 5% of changed devices and confirm the final state. Compare the count of planned changes with applied changes and fail the run if they differ. Store the before-and-after state for each device so a bad run can be reversed without guessing. Finally, keep the dry-run output as the change record for the maintenance window.