429s, Worker Restarts, and a Breaking v2
An Order API runs on CloudHub in us-east-1 as a single 0.1 vCore worker, registered in API Manager through autodiscovery with a rate-limiting policy of 100 requests per minute and Client ID Enforcement. At 02:00 a nightly batch client sends about 400 requests per minute and receives 429s; no other client is affected. The same night CloudHub restarts the worker twice, and p99 latency for interactive clients climbs from 600 ms to 3.8 s. The team must also ship v2 of the API, which removes a field from the response, while four consumers still need v1 for at least 30 days.
1. Explain the 429s and choose the fix.
Read the worked answer
Mechanism: the rate-limiting policy is evaluated per client application (per contract/SLA tier), not as one global counter for the API. The batch client is exceeding its quota; that is why interactive clients are unaffected. The API itself is not saturated at that point.
Wrong fix: raising the API-wide limit so the batch succeeds. That removes the protection for every client and hides the real sizing question.
Right fix: create a separate SLA tier with a limit matching the batch contract (for example 600/min), then request/approve a contract for the batch client's credentials on that tier. Verify the batch really sends the expected client_id and client_secret, otherwise it may be inheriting a default tier.
Diagnostic note: 429 responses are generated by the policy in the gateway before the flow executes. They will not appear in the application log. Confirm them in API Manager analytics and the monitoring dashboard, not by tailing CloudHub logs.
2. The worker restarted twice and p99 tripled. What do you inspect, and what do you change?
Read the worked answer
Inspect, in this order:
- CloudHub logs (not just application logs) for the restart reason - out-of-memory, failed health check, or a platform/worker event.
- The monitoring charts for heap, CPU, and thread count across the 02:00 window.
- The flow's own behaviour under batch load: blocking retries, large payload buffers, chatty logging.
Likely reading: 0.1 vCore gives a small heap and a small request thread pool. A 6x traffic spike plus long-running synchronous work causes thread starvation (latency climbs) and heap pressure (restarts on health-check failure).
Changes that actually address it:
- Increase the worker to 0.2 vCore and/or add a second worker so the load is spread and a restart does not take the API fully offline.
- Set HTTP response timeouts on outbound calls so threads are never held indefinitely.
- Move the nightly batch off the synchronous path (VM queue or a separate batch job) rather than scaling the API to absorb a burst it was never sized for.
- Reduce logging to INFO in production.
The judgement: restarts plus latency under a known, scheduled burst is a sizing and threading problem, not an error-handling problem.
3. How do you deliver v2 (a removed field) without breaking the four v1 consumers for 30 days?
Read the worked answer
Do not redeploy over the existing application - that replaces v1 instantly.
Pattern:
- Keep the v1 application running unchanged.
- Deploy v2 as a separate CloudHub application (and a distinct API instance in API Manager, or a new major asset version in Exchange).
- Put a routing layer in front: an API proxy application, or a load balancer/Flex Gateway route, keyed on a path version (
/v1,/v2) or a version header. Both backends register in API Manager so policies, SLA tiers, and analytics apply consistently. - Publish a deprecation notice and sunset date on v1 in Exchange, and notify the four consumers with the migration steps.
- Before decommissioning, use API Manager analytics to prove that v1 traffic has actually reached zero - do not rely on the consumer's word.
CloudHub's zero-downtime rolling update helps only when you already run multiple workers on the same version; it does nothing for a breaking change between versions.