Choosing storage for a write-heavy logging pipeline
A team needs to store application logs from 200 ECS instances. Each instance writes about 50 GB per day, giving roughly 10 TB/day. Logs must be queryable for 30 days (about 300 TB hot), then retained for 1 year at the lowest possible cost. Queries are ad hoc and scan large time ranges; the team wants SQL-like querying without managing servers. They are considering Object Storage Service (OSS), ApsaraDB for RDS, and a data warehouse option, and must also decide on a storage class for the cold tier. Budget pressure is high, and the cold tier is read rarely (a few times a year).
1. Recommend a storage and query architecture across the hot (30-day) and cold (1-year) tiers, justifying each choice against the workload.
Read the worked answer
Split the problem into hot queryable storage and cold archival storage, because the access patterns and cost profiles differ sharply.
Hot tier (30 days, ~300 TB, ad hoc large-range SQL):
- Do not use RDS. RDS is a transactional OLTP database; 10 TB/day of append-heavy logs and large analytical scans would overwhelm it and cost far too much.
- Use OSS as the landing/durable store for raw logs (cheap, virtually unlimited, durable), and
- Query with a serverless/warehouse analytics layer over that data (e.g., a data warehouse / analytical engine that reads from OSS), giving SQL-like querying without managing servers.
Cold tier (1 year, rarely read):
- Keep it in OSS using an archive/low-frequency storage class (e.g., Archive or Cold Archive), which trades retrieval latency for much lower storage cost - appropriate for a few reads per year.
- Use lifecycle rules to automatically transition objects from Standard to the cold class after 30 days, and to expire them at one year.
This separates cheap durable storage from compute-on-demand query, matching cost to access frequency.
2. The team worries that moving cold data to an archival storage class will break occasional queries. How should they handle rare reads of archived logs, and what lifecycle design supports it?
Read the worked answer
Archival classes are cheap precisely because retrieval is not instant, so the design must account for restore latency rather than assume immediate reads.
Handling rare reads:
- Expect a restore step. Reading archived objects typically requires initiating a restore and waiting for it to complete before the object is accessible; plan for that latency in any investigation workflow.
- Keep a short recent window hot. Because most investigations touch recent logs, keep the last 30 days in a queryable class and only archive older data. This means the common case never hits the cold tier.
- Automate tiering with lifecycle rules. A rule transitions objects to the archival class after 30 days and expires them at 365 days, so no manual movement is needed and cost stays bounded.
- Document the restore procedure so on-call staff know archived reads are not instant.
The lifecycle rule plus a documented restore step lets the team keep costs low without being surprised when a rare query needs old data.