Deletion Gaps in a 4.8 PB Object Store
Lakeside Retail operates 4.8 PB in object storage. A lifecycle rule is supposed to delete raw clickstream objects after 400 days. The monthly deletion report shows 2.3% of eligible objects remain, some for more than 600 days. Investigation finds bucket versioning is enabled, object lock is applied to a subset for litigation, and an analytics team copies objects into a 'temp' prefix that has no lifecycle rule. The privacy office requires evidence that personal data is deleted on schedule, and storage costs are rising. The platform team proposes disabling versioning and object lock globally to make deletion work. The analytics team says its temp copies are needed for a quarterly model refresh and cannot be deleted. No one has reconciled the deletion report against the actual object inventory.
1. Evaluate the platform team's proposal and explain why deletion is failing. What controls should be added?
Read the worked answer
Disabling versioning and object lock globally is the wrong fix. Object lock may be preserving data under a legal hold; removing it could destroy evidence and violate the hold. Versioning explains part of the gap because deleting a current object can leave noncurrent versions that the lifecycle rule does not expire. The temp prefix is a separate copy outside the rule, so the original deletion succeeds while personal data persists elsewhere. Incomplete multipart uploads may also linger. Add targeted lifecycle rules for noncurrent versions, abort incomplete multipart uploads, and require every prefix to have an owner and a lifecycle policy, including temp and analytics copies. Keep object lock only where legal has authorized it, and record each locked object in an exception register. Do not globally disable versioning; instead set a retention period for noncurrent versions that is consistent with the records schedule. Reconcile the deletion report to an independent inventory by prefix and version state, so the 2.3% residual is explained by category rather than treated as noise. The analytics team must move its refresh to a governed dataset with its own retention period, not an unmanaged copy.
2. What evidence would you provide to the privacy office to demonstrate deletion, and how should exceptions be governed?
Read the worked answer
Evidence should show what was eligible, what was deleted, what remained, and why. Provide the lifecycle rule configuration and version history, deletion job logs with object counts and timestamps, an independent inventory reconciliation by prefix and version state, and a list of residual objects mapped to a lawful exception. For legal holds, show the hold identifier, authorizing matter, scope, and release date. For analytics copies, show the approved retention basis and expiry. Include a sample test that creates a dummy object, waits for the rule, and verifies that all versions are removed. Exceptions should be time-bound, approved by privacy or legal, recorded in a register, and reviewed monthly; an expired exception should automatically block further copying or trigger deletion. The privacy office should receive a signed attestation that deletion controls operated, plus a remediation plan for any residual category. This is stronger than a raw deletion count because it distinguishes successful deletion from data that was never in scope or is lawfully retained.