Cloudera CDP Data Analyst: Mutable Alerts: Kudu, HBase, or Parquet

Mutable Alerts: Kudu, HBase, or Parquet

A fraud-monitoring platform must serve two access patterns over the same 250-million-row alert table (growing about 4 million rows per day, retained 24 months): point lookups by account_id at 2,000–3,000 requests per second from an application API, and analyst range scans by alert_ts with filters on status, run from Impala and a BI tool using standard SQL. The status of each alert changes 3–5 times during investigation, and analysts must see updates within seconds. The current prototype is an HBase table with a Java service for scans; analysts cannot query it without the team writing custom endpoints. Cluster: 10 workers, 24 cores and 128 GB RAM each, 12 × 4 TB disks; Kudu and Impala are installed and colocated. Alert rows are about 600 bytes.

1. Recommend Kudu, HBase, or Parquet-on-Impala, and defend the choice against the other two on all three requirements (mutable status, point lookups, SQL range scans).

Read the worked answer

Recommendation: Kudu.

Against Parquet-on-Impala: Parquet is immutable. A status change means rewriting a file or partition, which cannot be visible in seconds, and concurrent rewrites create reader conflicts. Point lookups by account_id would require partitioning or bucketing by account_id, which destroys pruning for alert_ts range scans. Parquet is the right archive tier, not the operational store.
Against HBase: HBase handles point lookups and cell updates well, but there is no native SQL path for Impala or a BI tool - that is exactly the custom-endpoint problem the team already has. Making range scans by alert_ts efficient requires a carefully designed rowkey, and rowkey design cannot simultaneously optimize account_id lookups and alert_ts scans. Status filtering is a full scan unless you duplicate the table by status.
For Kudu: a primary key of (hash(account_id), alert_ts, alert_id) supports the API lookup (hash prefix) and the analyst range scan (ordered alert_ts within a bucket). Rows are updated in place with UPDATE ... WHERE, visible to Impala readers within seconds. Columnar storage and predicate pushdown make the BI scans fast, and SQL is standard.

Caveats to state: Kudu is not for multi-terabyte full-table analytics; keep 90 days hot in Kudu and offload older months to Parquet, and do not run unpruned status-only scans across all tablets.

2. Design the partitioning. Show the arithmetic for tablet count and explain why one range partition per day is a poor choice here.

Read the worked answer

Size estimate: 4M rows/day × 600 B ≈ 2.4 GB/day raw, ×3 replication ≈ 7.2 GB/day. Over 730 days that is roughly 1.75 TB raw and 5.25 TB replicated - well within 10 workers × 12 × 4 TB.

Design:

PRIMARY KEY (hash(account_id), alert_ts, alert_id)
PARTITION BY HASH (account_id) PARTITIONS 16,
             RANGE (alert_ts) (
               PARTITION VALUES < '2025-02-01',
               PARTITION VALUES < '2025-03-01', ... )

Tablet count: 16 hash buckets × 24 monthly ranges = 384 tablets, about 4.5 GB raw each (13.6 GB with replicas). With 10 tablet servers that is ~38 tablets per server - comfortable. A good working range is roughly 1–20 GB per tablet and well under ~100 tablets per server.

Why not daily ranges: 730 days × 16 = 11,680 tablets. Each tablet carries Raft and memory overhead, slows the Kudu master and Impala metadata loading, and multiplies compaction work - for tablets of only ~2.4 GB raw, which is under-partitioned benefit for the cost. Daily ranges are only defensible with a much shorter retention (for example 90 days, giving ~1,440 tablets).

Also note the key order: hashing account_id first spreads API lookups and ingest uniformly, and keeping alert_ts next preserves sorted order for range scans. Monthly range boundaries must be added ahead of time; a missing range rejects inserts.

3. Six weeks in, ingest spikes to 12 million rows per day and analysts report slower scans. What do you check and change, and what do you do with data older than the retention window?

Read the worked answer

Diagnose before resizing.

Scan latency first. Check the predicate shape: if analysts filter on status alone, Kudu cannot prune - it scans every tablet. Filters must include account_id or an alert_ts range to use the primary-key prefix. If the common question is 'all open alerts', create a small companion table keyed by (hash(status), alert_ts, alert_id) maintained by the same writer. That is cheaper than changing the primary key, which requires a full table rewrite.

Ingest spike. 12M rows/day × 600 B ≈ 7.2 GB/day raw is still modest; slow writes are more often a hotspot or a compaction backlog than raw volume. Inspect per-tablet-server CPU, disk, and flush/compaction metrics. A monotonically increasing alert_ts concentrates new inserts on the newest monthly range's 16 buckets - if those servers saturate, increase hash buckets (rewrite required) or pre-split the next range earlier. Do not add partitions blindly; 384 is already adequate for this volume.

Retention and growth. Do not delete row by row. Archive the expiring month to Parquet, verify counts, then ALTER TABLE ... DROP RANGE PARTITION - instantaneous and it frees the underlying data. Pre-create the next monthly ranges in the same change window, and alert if a write ever fails with a missing-range error, since that indicates the maintenance job did not run.