Store the Data
Choosing the store
Cloud Storage is the landing zone and cheapest durable tier: know the classes (Standard, Nearline 30-day, Coldline 90-day, Archive 365-day minimums), Autoclass, Object Lifecycle Management, bucket retention and lock, and dual- versus multi-region. BigQuery is the warehouse: choose ingestion-time versus column partitioning, cluster on the columns you filter on (order matters, up to four), and set require_partition_filter to prevent full scans. Snapshots, time travel (2–7 days), materialized views, and BI Engine are part of the storage design conversation. Bigtable serves high-throughput single-row lookups; Spanner serves strongly consistent relational workloads with interleaved tables.
Schema and key design
Bigtable has no secondary indexes, so the row key is the schema. Leading with a monotonically increasing value (timestamp, auto-increment ID) creates a hotspot; salt, hash-prefix, or promote fields to spread writes. Avoid tall-and-narrow rows when a single-row read would do. In BigQuery, denormalize into nested and repeated fields instead of joining large tables, and remember clustering improves pruning but does not guarantee global ordering.
Traps
Partitioning on a high-cardinality column such as user_id creates too many small partitions; partitioning by date but filtering on another column triggers a full scan. Changing a clustered column effectively rewrites the table. Deleted or updated rows do not reclaim storage until the time-travel window passes. GCS objects are immutable - a rename is a copy plus delete, so design prefixes for cheap lifecycle transitions.
Rule of thumb: pick a partition key with hundreds to thousands of values, cluster on the most-filtered columns, and verify with INFORMATION_SCHEMA bytes-scanned numbers rather than intuition.