Object Storage Retention: Legal Holds, Versions, and Cost Reduction
A Swiss insurance company stores 120 TB of claim documents in an Exoscale Object Storage bucket in CH-DK-2. Versioning is enabled. About 30% of objects have five or more versions, and 8% are subject to litigation holds that must not be deleted for up to ten years. Monthly GET traffic is 2 million requests, concentrated in the first 30 days after upload; older documents are rarely read but must remain retrievable within 24 hours. The current bill is dominated by storage, not requests. The company wants a 25% storage-cost reduction without weakening retention. They also found 14 TB of incomplete multipart uploads and many delete markers from a failed cleanup script. The compliance team requires evidence that held documents cannot be purged by lifecycle rules, and the security team wants server-side encryption and least-privilege bucket policies. The team uses the S3-compatible API and can change application code only for new uploads.
1. Design a lifecycle policy that reduces cost while preserving held documents.
Read the worked answer
Use lifecycle rules scoped by prefix or tag, not a bucket-wide delete rule. For current versions, transition objects to an Infrequent Access storage class after 30 days if the bucket supports that class and the retrieval SLA remains within 24 hours. For noncurrent versions, expire them after 90 days, but exclude objects tagged legal-hold=true. Abort incomplete multipart uploads after 7 days and expire expired delete markers where safe.
The critical judgement is the interaction between lifecycle and legal hold. A tag-based exclusion is only trustworthy if the tag cannot be removed by ordinary users. Add a bucket policy that denies s3:DeleteObjectVersion and denies removal of the legal-hold tag for principals outside a break-glass role. If the bucket supports Object Lock, use it for held documents; do not rely on lifecycle filters alone for a ten-year legal obligation. Test the policy against a sample of held objects and confirm that lifecycle evaluation leaves them in place. Also ensure new uploads set the tag or lock at write time, because the application cannot be changed for old objects.
2. How should the 14 TB of incomplete multipart uploads and delete markers be handled safely?
Read the worked answer
Treat the 14 TB as two separate problems. Incomplete multipart uploads are invisible to normal object listing but are billable. Inventory them with ListMultipartUploads, identify uploads older than seven days, and abort them in batches. Then add a lifecycle AbortIncompleteMultipartUpload rule so the problem does not recur. Validate on a small prefix first because an active upload that is aborted will fail and must be retried by the application.
Delete markers are different. A delete marker does not itself store object data, but removing the wrong marker can resurrect an object or make a noncurrent version current. Do not bulk-delete markers by prefix. List object versions, and only remove a marker when there are no noncurrent versions behind it and the object is not held. For held documents, leave all versions and markers untouched. Use a script that reports what would change before it changes anything, and require a second approver for any marker deletion. The goal is cost cleanup without altering the evidential record.
3. What access and encryption controls would you require for this bucket?
Read the worked answer
Require TLS on every request with a bucket policy condition denying aws:SecureTransport=false. Enable server-side encryption with the bucket's managed key or a customer-provided key if the compliance team requires key custody. Enforce encryption on upload with a policy that rejects PutObject requests without the required encryption header. Use separate IAM roles for writers, readers, and lifecycle administration; no role should have wildcard actions on the bucket.
Scope writer roles to PutObject and multipart actions on the claim prefix only. Reader roles should be granted GetObject and ListBucket on the same prefix. Deny DeleteObjectVersion except for a break-glass role, and log use of that role. If cross-zone replication is not required, keep data in CH-DK-2 to preserve residency. Do not enable replication to another country for convenience. Review bucket policies for public access and block public ACLs. These controls support the claim that held documents cannot be purged and that access is least-privilege.