HBase Hot Region from a Timestamp-First Rowkey

A table's rows are ordered lexicographically by rowkey and each region owns a contiguous key range. With the timestamp as the leading bytes, all writes for 'now' fall into the single region whose range covers the current time, so that region absorbs essentially the entire 150 MB/s write stream until it reaches the split size, splits, and the new upper region immediately becomes the hot one. Because the hot region keeps moving but stays on whichever RegionServer holds the newest range, one server ends up with 45% of writes. This is the classic monotonically increasing key hotspot, and the fact that four regions are 80 GB shows splitting has not kept the table balanced.

A better key is bucket + hostId + (Long.MAX_VALUE - timestamp), where bucket = (hostId.hashCode() & 0xFF) gives 256 write buckets. Writes for different hosts spread across 256 key ranges and therefore across all 12 RegionServers, while all rows for one host still share the same bucket prefix, so the read 'last 1 hour for host X' remains a single contiguous scan with a start row of bucket+host+now and a stop row of bucket+host+one hour ago. Newest-first ordering is preserved by the inverted timestamp.

'Top 20 hosts by metric' is the hard case: it is a full-table aggregation and no rowkey can serve it. Build a derived table maintained by a Flink or Spark job that stores per-metric, per-minute aggregates keyed by metricId + inverted minute, or use a secondary-index layer (Phoenix or an Elasticsearch index) for dashboard queries. Keep the raw table for drill-down only.

Regions can exceed hbase.hregion.max.filesize temporarily because splitting is asynchronous and only evaluated when store files close. A region receiving a continuous write burst keeps flushing memstores, and the parent continues serving writes while daughter regions are created and assigned; with the timestamp-first key the newest region is rewritten faster than the split/assignment cycle can move load away. So the 10 GB setting is not being ignored, it is being outrun.

Plan:

1. Pre-split the table along the 256 bucket boundaries, or use RegionSplitter with HexStringSplit to create on the order of 512-1,000 initial regions. At 60 TB and a 20-30 GB target region size, the steady-state target is roughly 2,000-3,000 regions total, or 170-250 per RegionServer, which is comfortable for a 32 GB heap.
2. Split each existing 80 GB region online with split at a row boundary near the middle, one region at a time per RegionServer, and wait for assignment and compaction of the daughters before moving on. Do not run major compaction on the whole cluster at the same time.
3. After the key change, verify balance with region-count and write-request metrics per RegionServer; run balancer during a low-traffic window.
4. Confirm the split policy is IncreasingToUpperBoundRegionSplitPolicy or SteppingSplitPolicy as intended; a disabled or custom policy would explain persistent oversized regions.

Also raise the client-side write batching: BufferedMutator with a 2-4 MB write buffer and hbase.client.write.buffer sized accordingly, so 150k 1 KB puts/s does not become 150k RPCs/s against a small number of handlers.

RegionTooBusyException is server-side backpressure: a region's memstore has grown past hbase.hregion.memstore.block.multiplier (default 4) x hbase.hregion.memstore.flush.size (256 MB), i.e. about 1 GB, because flushes cannot keep up with the 150 MB/s write stream. Handlers block on that region, the call queue fills, and P99 rises for all clients of that server. The 12 s spikes align with major compaction because a major compaction rewrites every store file in the region, consuming all disk IO and competing with WAL sync and flushes; on 80 GB regions that is hours of heavy IO.

Compaction plan:

• Raise hbase.hregion.majorcompaction from 24 hours to 7 days (604800000 ms) and run major_compact manually in a rolling, off-peak schedule so only one region per server compacts at a time.
• Keep hbase.hstore.compactionThreshold at 3 and lower hbase.hstore.blockingStoreFiles if needed, but the driver here is major compaction, not minor.
• For a uniform, high write-rate workload, evaluate stripe compaction to avoid the periodic full rewrite.
• Cap memstore: hbase.regionserver.global.memstore.size around 0.4 and block cache around 0.4 of heap; with 32 GB heap the remainder covers overhead.

RegionServer plan:

• Increase hbase.hstore.flusher.count from the default 2 to 6-8 so memstore flushes drain faster; this is the direct fix for RegionTooBusy under sustained write load.
• Increase hbase.regionserver.handler.count from 30 to about 64 only after flush capacity is addressed; more handlers on a backed-up region just adds threads waiting.
• Set client-side backpressure and retry: bounded hbase.client.max.perserver.tasks, exponential backoff, and a fallback path that buffers to Kafka rather than hammering the region.
• Watch flushQueueLength, memstoreSize, blockedFlushCount and compactionQueueSize per server; these tell you whether the fix is flush capacity, disk throughput, or region count.