Storage and Retrieval
Storage and Retrieval tests how you size, tier, index, protect and eventually migrate a repository so a specific document can be found within an acceptable time for its whole retention period.
Storage tiers and media
- Online: disk or SSD with millisecond access, used for active images and indexes.
- Near-line: automated optical jukebox or tape library, seconds to minutes, cheaper per gigabyte.
- Offline: removable media on a shelf; retrieval is manual, so it suits low-probability reference only.
- Media: HDD, LTO tape (long shelf life but requiring technology refresh every few generations), WORM optical for tamper-evident retention, object storage in the cloud. Flash is fast, not archival.
Capacity planning
Image size derives from resolution, bit depth and compression; multiply by annual page count, add index overhead (often 10 to 30 percent of the image store) and temporary workspace, then apply retention years and the number of copies. Keep primary storage, backup and archive as three separate budgets.
Indexing and retrieval
A unique document ID plus structured metadata drives fielded search; a full-text index enables keyword search but inflates storage and build time. Fielded queries are precise and fast; full-text queries trade precision for recall. COLD/ERM report archives index by report structure and page range rather than by image. Retrieval quality is judged by recall (did we find every relevant item), precision (how much noise) and response time.
Protection and lifecycle
RAID 5 or 6 protects against a drive failure; that is availability, not backup. Backup answers accidental deletion and corruption; archive answers retention. Plan media migration and format migration as separate workstreams, verify with checksums and sampling, and keep WORM copies where regulation demands immutability.
Worked example
Two million images a year at 60 KB average. Reasoning: 2,000,000 times 60 KB equals 120 GB per year; seven-year retention gives 840 GB; add about 15 percent for indexes, 10 percent growth headroom, and a second-site copy, and you provision roughly 2.2 TB. Primary on RAID 6 disk, nightly backup to tape, annual tape refresh against the vendor roadmap.
Traps that cost marks
Treating RAID as a backup. Confusing backup with archive. Accepting vendor shelf-life claims without a migration plan. Forgetting index and workspace overhead in sizing. Assuming near-line storage is as fast as online storage.