Fixing a Monotonic Rowkey Hotspot in an HBase IoT Table
An IoT platform stores 4 billion sensor readings per day in HBase 2.x on a 10-node cluster. The rowkey is built as timestamp + deviceId, for example 20240601123045-sensor-000123. Writes arrive at about 60,000 ops/sec. One RegionServer consistently shows 90% CPU and 85% disk utilization, while the other nine stay near 20%. Read queries fetch the last 24 hours for one device and are timing out. There are 30 pre-created regions, and the cluster has 10 RegionServers. Each row is small (about 200 bytes), and the table has a single column family. The team added two RegionServers but the hotspot remained. The platform owner asks why scaling out did not help and how to fix the rowkey and table design without losing the ability to scan a device's recent history.
1. Diagnose the hotspot. Explain why adding two RegionServers did not help.
Read the worked answer
The rowkey begins with a timestamp, and timestamps increase monotonically. HBase sorts rows lexicographically by rowkey and splits the table into regions by rowkey range. All new writes therefore land in the region covering the newest timestamp range - effectively the last region. That region lives on one RegionServer, so one node absorbs nearly all 60,000 writes per second while the other nine idle.
Adding RegionServers does not fix this because the hot region is not split by adding hardware. HBase may split a large region, but the split still leaves one region covering the newest range, and the writes continue to concentrate there until the next split. The 30 pre-created regions are also too few for 4 billion rows per day and their boundaries were not chosen to distribute timestamps. Scaling out helps only after the write distribution is changed; otherwise the new nodes host cold regions and the hotspot persists.
2. Redesign the rowkey to distribute writes while preserving device history scans. Explain the trade-off.
Read the worked answer
Introduce a distribution prefix and change the sort order within it. One design is hash(deviceId) % N + deviceId + reverseTimestamp, where N is a small bucket count such as 16. Writes for different devices spread across bucket prefixes, and within a device the reverse timestamp keeps the newest reading first, which is efficient for 'last 24 hours' scans.
A common alternative is salt + deviceId + timestamp, with the salt being a short hash prefix. This also distributes writes, but it may split one device's history across multiple salt buckets if the salt is derived only from deviceId? If the salt is derived from deviceId, all rows for that device share one salt, so device history stays contiguous while different devices distribute. That preserves the scan pattern.
Trade-off: a pure reverse-timestamp rowkey without a device prefix would distribute writes but destroy per-device scan locality; a query for one device would scan the whole table. The hash-prefix design keeps device locality, but a query must know the device's bucket, and hot individual devices can still concentrate load. Choose the bucket count to balance write spread against scan overhead.
3. What table and cluster changes should accompany the rowkey redesign? Include pre-splitting, column family, and region sizing.
Read the worked answer
Rowkey redesign alone is not enough; the table must be created with split boundaries that reflect the new key distribution.
Pre-splitting: create the table with pre-split boundaries based on the hash prefix range, for example 64 or 128 splits, so the initial regions distribute across all RegionServers. Avoid 30 coarse regions for 4 billion rows/day; target region sizes of roughly 10–20 GB and monitor splits.
Column family: keep one column family. HBase flushes and compacts at the family level; multiple families with different access patterns cause unnecessary I/O. If some readings are large or rarely read, separate them into another table rather than adding families casually.
Compression and compaction: use SNAPPY or GZIP for the family to reduce disk usage, and tune major compaction windows to avoid competing with peak writes. With 200-byte rows and 60,000 ops/sec, write amplification matters.
Cluster: verify that region placement is balanced after pre-splitting, and use the HBase balancer. Adding RegionServers helps only after the rowkey distributes writes; before that, new nodes have no hot region to serve. Also cache the device-to-bucket mapping so reads can locate the correct region directly.