Storage Architecture
Storage questions probe the Distributed Storage Fabric: where a write goes, how copies are protected, and which features save capacity without hurting latency.
The I/O path
- A VM write is handled by the local CVM's Stargate service and lands in the OpLog, a low-latency persistent write buffer on SSD/NVMe.
- With RF2 the write is mirrored to a second CVM before the guest is acknowledged; it is later drained to the Extent Store.
- Curator runs background MapReduce jobs: disk scrubbing, ILM tiering, and rebuild/re-protection after a failure.
Metadata and services
- Cassandra holds distributed metadata (extent map, vDisk ownership); Zookeeper manages configuration and leader election; Genesis monitors and starts services.
- Data locality keeps reads on the local CVM where possible; a lost CVM forces remote reads until rebuild completes.
Capacity features
- Inline compression such as LZ4 sits on the I/O path; deduplication is post-process on cold data, so it never adds write latency but never helps the hot working set either.
- Erasure coding (EC-X) applies only to cold extents after ILM moves them, and needs enough nodes to stripe across.
Worked example
A 200 GB database VM is cloned for testing. The clone is near-instant because redirect-on-write points new metadata at existing extents and only changed blocks are written. The test VM then runs a full table scan: dedup does not help because the data is hot, compression does, and the scan may be served remotely if the clone landed on another host.
Traps
- Assuming dedup is inline, or that it will shrink a live transactional database.
- Expecting EC-X on the hot tier, or on a cluster too small to stripe.
- Confusing snapshots (metadata plus changed blocks) with full copies.
- Ignoring CVM resource starvation, which throttles the entire I/O path on that host.