On-Call Bot with a Leaked Token and a Trusting Button
An on-call Webex bot serves 40 spaces across three regions. It is a single Flask app with the bot token committed in config.py, a static webhook secret, and an Adaptive Card with an Acknowledge button that posts back to the same URL. Anyone in a space can press the button, and the handler trusts the personId in the card action payload. During an incident last month, the bot posted the same alert 14 times in 90 seconds because the monitoring system retried after a 5-second timeout. The bot also sends a new message for every status change, hitting Webex rate limits and getting 429s. The team wants to add Resolve and Assign to me buttons and support 200 spaces. The security team requires that actions be attributable and that no secret be in source. The app runs on a single VM with no autoscaling.
1. Redesign authentication and secret handling. Compare the current bot token with a service app and scoped OAuth, and describe token refresh for 200 spaces.
Read the worked answer
Move the token out of source and into a secret manager (Vault, AWS Secrets Manager) injected at runtime; rotate the exposed token immediately and audit its usage. Prefer a Webex service app with scoped OAuth rather than a personal bot token: the app authenticates as itself, gets short-lived access tokens via client credentials, and is not tied to a departing employee.
Request only the scopes needed (messages write, rooms read, membership read), not admin scopes. Store the client secret in the secret manager, not in config.py or the image. For the webhook, use a per-endpoint secret and verify the signature on every request; a static shared secret in source is a forgery vector. For 200 spaces, keep a durable mapping of space ID to tenant/region; do not hardcode 40. Add token refresh with skew (refresh at 80% of lifetime) and a single-flight lock so 200 spaces do not trigger 200 refreshes. Log token fingerprints, never tokens. Add a startup check that fails closed if the secret is missing rather than falling back to a default.
2. The bot posted the same alert 14 times and creates a new message for every status change. Design message lifecycle and rate-limit handling.
Read the worked answer
Assign each alert a deterministic fingerprint (space ID + alert rule + resource + state) and keep a mapping to the Webex message ID. On a status change, update that message with a PUT/PATCH instead of posting a new one; only create a new message when the state transitions to a new incident. This reduces volume and makes the space readable.
Serialize per space: a queue with one worker per space, or a partitioned queue, prevents 14 concurrent posts for the same alert. Deduplicate at ingress by fingerprint within a window (for example 10 minutes) and collapse repeats into a count. Honor 429 with Retry-After and exponential backoff with jitter; do not retry a 400. The 5-second timeout causing 14 retries means the upstream retries too aggressively. Return 202 after enqueueing, and make the upstream retry idempotent via the fingerprint. Add a circuit breaker per space so one broken space does not stall the rest. Persist message IDs and state in a database, not memory, so a restart does not re-post.
3. Secure the new Resolve and Assign to me card actions. Explain how to authenticate the actor, authorize the action, and make it idempotent.
Read the worked answer
A card action arrives from the client, so treat every field as untrusted. The personId in the payload can be forged or replayed. Verify the request signature, then call the Webex API to resolve the actor from the authenticated context, or use a signed short-lived nonce embedded in the card that is bound to the message ID and intended action. Check that the actor is a member of the space and, for Resolve or Assign, that they hold the required role. Do not rely on the card's displayed name.
Make the action idempotent: store (message_id, action, actor_id, nonce) with a unique constraint, so a double-click or retry does not assign twice. Record an audit event with who, what, when, and previous state. Return a fast acknowledgement and update the card asynchronously. For Assign to me, resolve 'me' server-side from the verified actor, never from a client-supplied ID. Expire nonces (for example 15 minutes) and rotate the signing key. With 200 spaces, rate-limit card actions per user to prevent a scripted flood.