Data Management and Storage Architecture
Data Management and Storage Architecture
This domain covers selecting storage by access pattern, planning lifecycle and retention, and protecting data against deletion as well as disaster. Items often hinge on arithmetic: a large dataset with rare access, a retention obligation, and a budget that rules out the obvious choice.
Classify, then choose
- Block storage for single-instance databases, file storage for shared POSIX workloads, object storage for unstructured data at scale, managed engines when you do not want to operate the database.
- Durability, availability, and backup are three different properties. Object durability of eleven nines does not protect you when an application deletes the bucket.
- Consistency model shapes your code: strong consistency simplifies logic; eventual consistency demands idempotent writes and read-your-writes handling.
Lifecycle, partitioning, and cost
- Lifecycle rules move objects hot to infrequent to archive and then delete. Retrieval fees and minimum storage durations (30, 90, 365 days) are where budgets actually break.
- Partition or cluster large tables by date or tenant. Unpartitioned full scans are the single most common warehouse cost driver.
- Keep one authoritative copy. Replicas exist for latency and DR, not as a second backup.
Protecting the data
- Backups must be automated, versioned, stored in a separate account, immutable, and restore-tested. An untested backup is a hypothesis.
- Object versioning plus retention lock gives WORM behaviour for regulated records.
- Data residency: pin the storage location, and remember replication, CDN caches, and vendor support access can all move data across borders.
Worked example
200 TB of compliance logs, queried roughly twice a year, retained seven years, RTO 24 hours. At about $0.02 per GB-month, standard storage is roughly $4,000 a month, or about $336,000 over seven years - clearly wrong. An archive tier at roughly a tenth of that cost, plus per-GB retrieval for two restores a year, is far cheaper, and a 12–48 hour restore still fits the 24-hour RTO if retrievals are planned rather than emergency. Design: lifecycle transition at day 30, retention lock at seven years, and a small hot index table so you can locate objects before paying retrieval fees.
Traps that cost marks
- Optimizing storage cost and ignoring retrieval and egress charges.
- Assuming replication is backup - deletions replicate instantly.
- Mixing transactional and analytical workloads on one system and blaming the engine for the performance.