90-Second Report Route Returns 504 Through API Gateway

Diagnosis. The gateway has a bounded upstream read timeout, and the client connection is torn down at ~30 s while the function keeps running. The function eventually succeeds, but nobody is listening - the work is done and paid for, the result is discarded. This is a synchronous-call-through-a-proxy problem: a 60-95 s workload cannot be delivered over a request/response path designed for short calls.

Why the obvious knobs fail.

• Raising the function timeout does nothing; it is already 120 s and not the constraint.
• Raising gateway timeouts runs into the service's own ceiling and, even at the maximum, leaves a fragile path: any client-side or DNS/CDN hop with its own timeout will break first, and every retry keeps a long-held connection plus an ATP session alive.
• Increasing memory is irrelevant - the workload is I/O- and database-bound, not CPU-bound.
• Retries from partners make it worse: each 504 spawns another 95-second query against ATP, which explains the session spike. Retries are being applied to a non-idempotent, expensive, long-running operation.

Judgement. The right answer is a protocol change, not a tuning change: the client's need is "give me the report", not "hold a socket for 95 seconds".

Asynchronous job pattern.

1. POST /reports validates the request, writes a job record (jobId, requester, parameters, status PENDING) to ATP or Object Storage, and returns 202 Accepted with Location: /reports/{jobId} and a body containing the job id. Response time becomes milliseconds; the gateway timeout stops being relevant.
2. A separate worker performs the long aggregation. It should be triggered off the request path - an OCI Events rule on the job-creation event or a queue/stream - so the heavy query is decoupled from client traffic and can be rate-limited independently.
3. On completion the worker writes the report to Object Storage and updates the job status, publishing an OCI Events event so downstream notification can be sent.
4. GET /reports/{jobId} returns 202 with status/progress while running, 303 See Other (or a body with a pre-authenticated URL) when done, and a documented error payload on failure. The pre-authenticated URL should have a short, explicit expiry and a single object scope.
5. Add Idempotency-Key handling on POST /reports so a partner retry does not create a second 95-second job - for a known key, return the existing job id.

Client contract. Partners poll with backoff (or receive a webhook) and must treat 202 as "in progress", not as an error. Publish this in the API description.

Why this is better operationally. Worker concurrency becomes a deliberate throttle on ATP load, retries happen inside the platform where they can be made idempotent, and a gateway or CDN timeout can never destroy completed work.

Attribution first. A 429 can be produced at three distinct layers: the API Gateway (usage plan / rate limit on the deployment or route), OCI Functions (per-function concurrency and invocation throttling), or the partner's own downstream dependency. The evidence that separates them:

• Gateway-generated 429s appear with gateway access-log entries and gateway metrics, before any function invocation.
• Function-generated TooManyRequests appears in function logs and implies the invocation was accepted and then rejected for concurrency - exactly what the logs show here.
• Third-party 429s would be a body relayed from the backend.

Since the usage plan allows 1,000 rps and the observed load was ~350 rps, the gateway was not the limiter; the function's concurrency ceiling was. Raising the plan would have changed nothing.

Fixes.

1. Make the effective capacity explicit: set the function's concurrency/provisioned capacity (or move the work to the async worker) so report-fn cannot be the hidden limiter for a route whose contract promises 1,000 rps. Add alerting on function throttles.
2. Keep per-client usage plans for fairness, but key them on a reliable client identity (an API key or JWT claim, not source IP shared behind a partner's NAT), and verify the plan is attached to the route being called.
3. Return Retry-After and document 429 as retryable with backoff, while 504-on-a-long-report is replaced by the 202 pattern so partners stop retrying expensive work.
4. Load-test at plan capacity before publishing the number - a usage plan is a promise the backend must be able to keep.