Change-Alert Bot That Dies After an Hour
An internal Flask service polls DNA Center every five minutes, diffs the device list against the previous poll, and posts change alerts into a Webex space. It authenticates to DNA Center once at startup, stores X-Auth-Token in a module-level variable, and reuses it for the life of the process. After roughly an hour, every poll logs HTTP 401 and the poll thread dies; the Webex side keeps working. Separately, when an engineer triggers a config push through the service, the code POSTs to the intent API, receives HTTP 202 with a taskId in the body, and immediately writes 'push complete' to the audit log. Neither behaviour reproduces on demand.
1. Diagnose the one-hour failure and describe how token handling should be designed instead.
Read the worked answer
DNA Center authentication tokens are time-limited (on the order of an hour), and the service treats a startup value as permanent state. Nothing in the code observes the 401 - the thread raises on the first rejected call and stays dead, so the service looks 'up' while silently alerting on nothing. That partial-failure mode is the real defect; token expiry is merely the trigger.
Design instead:
- Store the token with its acquisition timestamp and refresh proactively before the expected expiry.
- Also refresh reactively: on a 401, re-authenticate once and replay the request, so clock skew or a shorter server-side TTL does not break the poll.
- Guard refreshes with a single-flight lock. Several threads polling concurrently must not each trigger their own login; that turns recovery into a thundering herd and can itself trip rate limits.
- Fail loudly: if re-authentication fails, emit an alert through the Webex path (which is independent and still healthy) rather than dying quietly.
- Never log the token; log its age and the refresh outcome instead.
The asymmetry in the symptoms - Webex fine, DNA Center dead - is the clue that these are two independent credential lives, not one outage.
2. Explain why HTTP 202 with a taskId is not 'push complete', and what the audit log should record instead.
Read the worked answer
202 Accepted means the request was queued, not executed. The intent API is asynchronous because pushing config to a device can take tens of seconds or fail on the device - the API cannot hold the connection. The body's taskId is an identifier for the job, nothing more. A 2xx here proves only that DNA Center understood the request.
The audit log should record: task submitted (with taskId, target device, timestamp), then the outcome of polling the task endpoint - task state, progress, isError, and any failure reason - and finally a verification read of the device's actual configuration where the platform supports it. Interim states ('in progress', 'pending') must not be flattened to success.
Polling needs a bounded budget: an interval, a maximum wait, and a terminal-state check, otherwise a stuck task silently becomes 'complete' when the process exits. Also decide what a timeout means - it means unknown, not failed, and it must be escalated to a human rather than guessed.
This is the same class of bug as the token issue: treating a transport-level response as a business-level outcome.
3. A colleague proposes replacing raw `requests` calls with the vendor's Python SDK. What factors decide whether that is a good trade here?
Read the worked answer
The SDK is worth it if it buys you the three things the code currently gets wrong by hand: token lifecycle management, async task polling helpers, and pagination/cursor handling. If those are genuinely encapsulated and tested, it removes exactly the code most likely to break.
The costs are real and must be weighed, not assumed:
- Maintenance lag. SDKs trail platform releases. If you depend on a new endpoint the SDK has not wrapped, you will mix SDK and raw HTTP anyway - which is fine, but the abstraction must not force you back to square one.
- Version pinning. The SDK becomes another dependency in
requirements.txt; an unpinned upgrade is a silent production change. Pin it and test the upgrade. - Debuggability. SDK abstractions hide status codes and headers, which is precisely the information you needed to diagnose both bugs in this case study. Confirm the SDK surfaces errors with enough detail.
Decision rule: adopt the SDK for authentication and task polling, keep a thin transport layer you control so you can drop to raw HTTP, and write a test that proves a 401 triggers a refresh. If the SDK cannot express that, it is not solving your actual problem.