Public Webhook Receiver With Three Problems and One Deadline
A Flask app receives configuration-change webhooks from a SaaS platform and opens a ticket for each event. It runs on a public VM behind nginx: nginx terminates TLS and proxies to gunicorn in a container running as root. The webhook signing secret is a string literal in app.py, and the handler parses JSON and then compares a signature header against the reparsed body. The TLS certificate expired three days ago; someone 'fixed' the alert by adding an exception to the expiring-certificate monitor. The platform retries any non-2xx response for 24 hours, and users report three identical tickets per real change. requirements.txt lists flask with no version.
1. Rank the issues by what you would fix first and justify the ordering on risk rather than tidiness.
Read the worked answer
Rank by blast radius times likelihood, not by how easy the fix is:
- Expired TLS plus a muted monitor. The endpoint is public and the certificate is already invalid, so any client sending credentials or event payloads is exposed, and the monitor suppression means you would not learn about the next expiry either. Restore the alert first, then renew. Muting a monitor is the most damaging single act in this story because it destroys your ability to see the next failure.
- Secret in source. Anyone with repository access has the signing key, and it is in git history permanently - deleting the line does not revoke it. Rotate the secret, move it to environment or a secret store, then scrub. Rotation must precede the code change, or you ship a fix that still trusts a compromised key.
- Root container. Privilege escalation amplifier, not an entry point. Drop to a non-root user and read-only filesystem; it is a small change with a good payoff.
- Signature verification correctness and unpinned Flask. These are correctness and supply-chain risks; real, but not exposed on a public endpoint the way the first two are.
The trap is starting with the elegant code fix because it is interesting. Start with the exposure.
2. The current signature check parses the JSON and then compares against the re-serialised body. Explain why this cannot work and what the correct verification sequence is.
Read the worked answer
The signature is computed by the sender over the exact bytes it transmitted. Re-serialising parsed JSON does not reproduce those bytes: key order changes, whitespace changes, Unicode escaping changes, and number formatting changes (float vs int, exponent forms). Any of these yields a different digest, so the comparison fails for genuine requests - or, worse, the code is loosened until it accepts everything.
Correct sequence:
- Read the raw request body once, before any parsing (
request.get_data()in Flask, cached so the handler can still parse afterwards). - Compute the HMAC of those raw bytes using the shared secret and the algorithm the platform specifies.
- Compare in constant time (
hmac.compare_digest) to avoid leaking the digest byte-by-byte through timing. - Reject with 401 and no side effects if it does not match; only then parse and process.
- Reject stale timestamps if the platform provides one, otherwise a captured request replays forever.
Verifying after side effects is equivalent to not verifying. Also bind the secret to the specific webhook subscription, so rotating one integration does not break another.
3. Users see three tickets per real change. What is causing it, and how should the fix relate to the platform's retry behaviour?
Read the worked answer
Three deliveries means the app returned non-2xx twice. Because the handler acts first and responds afterwards, a slow or failing ticket-system call leaves the platform with no acknowledgement, so it retries on its schedule - and each retry passes signature verification and creates another ticket. The retries are correct behaviour; the handler is the defect. Returning 500 after the side effect is already committed is the classic dual-write problem.
The fix has two parts that must both be present:
- Idempotency. Key the side effect on a stable identifier from the event payload (event id or delivery id), stored uniquely in the ticket system or a dedupe table. A replayed delivery then updates nothing rather than creating a second ticket. Consider the race where two retries arrive concurrently - the uniqueness constraint must be enforced by the datastore, not by a read-then-write check in Python.
- Fast acknowledgement. Persist the event to a durable queue, return 2xx immediately, and process asynchronously. The platform's 24-hour retry window is a safety net, not a work queue.
Do not 'fix' this by disabling retries on the platform - that trades duplicate tickets for silently lost changes.