The Bucket That Grew Without Bounds
An application writes JSON event records to a single S3 bucket, one object per event, using a key scheme of events/YYYY/MM/DD/HH/<uuid>.json. Volume is ~40 million objects per month, average 2 KB each. The team also runs a nightly Spark job that lists the entire events/ prefix to find the previous day's files. Over six months, the nightly job's runtime has grown from 8 minutes to over 3 hours, and S3 request costs have climbed sharply. Lifecycle rules currently transition objects to Glacier Instant Retrieval after 30 days and expire them after 400 days. The team wants the nightly job back under 15 minutes and the request costs reduced, without losing the ability to query any event from the last 13 months.
1. Diagnose the two cost/performance problems and propose fixes that preserve 13-month queryability.
Read the worked answer
Two separate problems, two separate fixes.
1. The nightly job's LIST is the runtime killer. Listing events/ walks ~40M objects/month × 6 months ≈ hundreds of millions of keys. Each LIST returns up to 1,000 keys, so the job makes hundreds of thousands of paginated calls. That is the 3-hour runtime and a large chunk of the request cost. The key scheme is actually good (date-partitioned, high cardinality), so the fix is to stop listing the whole prefix: list only events/YYYY/MM/DD/ for the previous day. That's ~1.3M objects/day, still large, so better: have the writer emit a manifest, or use S3 Inventory (a daily manifest of all objects) and query the inventory for the target date instead of listing live. Inventory also moves the cost off the LIST API.
2. Object count and small-object overhead. 40M × 2 KB objects means per-request costs dominate over storage. Consider batching events into larger objects (e.g., hourly Parquet/JSONL files) to cut object count by orders of magnitude - this reduces both PUT and GET/LIST costs and speeds any scan. If the app can't batch at write time, a compaction job can roll daily small objects into one file.
Preserving 13-month queryability: the 400-day expiration already covers 13 months, so keep it. But Glacier Instant Retrieval after 30 days is a poor fit if the team queries older events interactively - IR has millisecond retrieval but a retrieval fee per GB and a 90-day minimum storage duration. If queries are rare, IR is fine; if frequent, keep a warmer class or move to Intelligent-Tiering. Confirm the access pattern before tuning the class.
2. A teammate suggests simply adding a lifecycle rule to delete objects older than 30 days to cut costs. Assess this against the requirements.
Read the worked answer
It violates a hard requirement. The team needs 13 months of queryability; deleting at 30 days destroys 12 months of that. This is the kind of move that looks like a cost win and is actually a data-loss incident - and with versioning off (not stated, but common), it's unrecoverable.
Even setting the requirement aside, it treats the wrong cost driver. The dominant cost here is requests, driven by object count and the nightly LIST, not storage of 2 KB objects. Deleting recent data removes the queryable data while barely denting request costs, because new events keep arriving at 40M/month.
The right levers, in order of impact: (1) fix the LIST pattern (targeted prefix or S3 Inventory), (2) reduce object count via batching/compaction, (3) tune the storage class to the actual retrieval pattern. Deletion only enters the picture at the 400-day boundary the business already agreed to - and even then, confirm with the data owner, because "queryable for 13 months" is a business promise, not an engineering default.