Large Object Uploads Fail with OOM and PAR Expiry in Archive Service

Why it fails. A single PutObject is not designed for multi-gigabyte payloads: the service caps a single put at 50 GiB, but the practical failure arrives much earlier because the request is one long-lived HTTP exchange. The SDK buffers, retries and request signing all operate on one monolithic stream, so a transient network hiccup or a 15-minute idle window kills the whole transfer and there is no resume point; memory pressure comes from the buffering and retry machinery holding the stream. The observed 4 GB failure threshold is a symptom of the client, not of a bucket quota.

Design: multipart upload.

1. Use the SDK's upload manager (UploadManager.upload) which performs CreateMultipartUpload → UploadPart (in parallel) → CommitMultipartUpload automatically.
2. Part size math. Minimum part size is 10 MiB (last part excepted) and a maximum of 10,000 parts per upload. For 12 GB: 12 GiB / 10,000 ≈ 1.2 MiB, so the minimum size forces at least ~8 MiB; choose 64-128 MiB parts - 12 GiB at 128 MiB is ~96 parts, well inside the limit, with room for future growth.
3. Heap math. Concurrent parts × part size must fit the heap: 4 concurrent × 128 MiB ≈ 512 MiB of buffers, comfortable in 6 GB. Streaming each part from a file channel range avoids loading the file.
4. Resilience. Each part retries independently, so a dropped connection costs one part, not the transfer. Set sensible per-request timeouts and retry with backoff.

Result. The 12 GB case becomes ~96 small, independently retryable requests, and heap usage is bounded and predictable instead of proportional to file size.

What is wrong. A pre-authenticated request (PAR) is a bearer credential with a fixed expiry. Issuing one with a 10-minute expiry for a 20 GB browser upload guarantees failure: at typical consumer bandwidth a 20 GB transfer takes far longer than 10 minutes, so the PUT is rejected with 403 partway through - exactly the reported symptom. The credential expired mid-transfer, not at creation.

Fix.

1. Size the expiry to the realistic worst-case transfer time plus margin, and keep it as short as that allows - the trade-off is exposure window versus completion. Compute it, don't guess: 20 GB at a documented minimum partner bandwidth determines the floor.
2. Prefer a multipart PAR flow: issue a PAR that permits creating and committing a multipart upload, sign each part (or use a PAR scoped to the multipart upload), so a single expiry does not invalidate a long transfer and parts can be resumed.
3. CORS. A browser PUT to Object Storage is a cross-origin request; the bucket needs a CORS rule allowing the partner origins, the PUT/POST/HEAD methods and the headers the SDK sends (including Content-Type and any checksum header). Missing OPTIONS handling is what produces the CORS errors, and those often mask the 403 rather than being a separate bug. Scope origins narrowly; do not use * on a bucket holding dumps.
4. Treat PARs as secrets. Never log the full URL, never embed it in a page that is cached or shared, and generate one per upload session with the narrowest object prefix possible.

Judgement. For 20 GB, the browser should not be a single PUT at all - multipart with per-part retry is what makes the flow survive a flaky partner connection.

Cause. A multipart upload creates parts in the bucket before the commit. If the client crashes, the PAR expires, the network drops, or an exception is thrown between CreateMultipartUpload and CommitMultipartUpload, the uploaded parts remain as an incomplete multipart upload. They are billed as stored objects but are invisible to a normal list of the bucket's objects - which is why cost climbs with no visible artefacts. The current PutObject flow can also leave orphaned state after retries.

Controls.

1. Lifecycle policy on the bucket to abort incomplete multipart uploads after N days (e.g. 1-3, longer than the longest legitimate transfer). This is the systemic fix: it bounds cost regardless of client bugs.
2. Explicit cleanup in the service: wrap the upload in try/finally and call AbortMultipartUpload on failure so the normal error path does not rely on the lifecycle sweep. Log the upload id.
3. Reconciliation job: list in-progress multipart uploads periodically, alert on age/count, and correlate against expected nightly jobs. A sudden jump is an early signal that transfers are failing silently.
4. Verification before commit: check part count and each part's ETag/checksum, then verify the object's size and checksum after commit; record the result in the job log so "upload succeeded" means the bytes are correct, not merely that the HTTP call returned 200.
5. Alert on cost and on the OOM/timeout errors themselves - the accumulating storage was a lagging indicator of a broken upload path that had been failing nightly for some time.

Ordering. Enable the lifecycle rule and the abort-on-failure path together: the rule cleans history, the code stops creating new orphans.