Cross-Site Disaster Recovery for Hive, Ranger, and Kudu
An insurer runs CDP Private Cloud Base in a primary data center with 1 PB in HDFS and 8,000 Hive tables. A secondary site 300 km away has 40% of the compute and storage; the 1 Gbps link is already 60% utilized by daytime replication of Kafka and database logs. The business requires RPO of 4 hours and RTO of 8 hours for Hive and Impala reporting; Kudu supports a fraud-detection application with a 15-minute RPO. The DR plan must cover Hive metastore, Ranger policies, and Atlas lineage, not just HDFS files. Replication Manager is available. During a recent test, HDFS files replicated, but tables were missing in the DR metastore, Ranger denied valid queries, and the failover took 14 hours because DNS and JDBC URLs were undocumented. What should the disaster-recovery architect change?
1. Define the replication scope and method for each data and metadata component, given the RPO/RTO targets.
Read the worked answer
Separate data from metadata and treat each RPO independently. For HDFS, use Replication Manager HDFS policies with incremental schedules that meet 4 hours; for Hive, use Hive replication so the metastore and data stay consistent, and explicitly test external tables and views. For Ranger, do not rely on HDFS replication: export policies from the primary Ranger and import them at the DR site on a schedule, then test. For Atlas, schedule metadata export/import or rebuild lineage from replicated Hive; lineage can lag but must be documented. Kudu has no native Replication Manager coverage for a 15-minute RPO, so use application-level dual-write or Kudu backup/restore with a much longer RPO, or renegotiate. Kafka can use MirrorMaker but the link is constrained; prioritize Hive/Impala and metadata over non-critical Kafka topics. Publish an RPO/RTO table per component so the business sees where 15 minutes is not met.
2. Why did the test fail on Hive tables and Ranger, and what sequencing and validation would prevent it?
Read the worked answer
The test failed because HDFS file replication and Hive metastore replication are different operations. If only HDFS policies ran, the DR metastore never learned about new tables or partitions; external tables also need correct path mapping. Ranger policies live in a separate database and are not carried by HDFS replication, so valid DR queries were denied. Fix the sequence: replicate or restore the metastore first, then data, then verify table/partition lists; import Ranger policies before opening the cluster; validate with the same personas and negative tests used in production. Add a failover runbook with DNS changes, JDBC URLs, Knox endpoints, and service start order. Define validation gates: row counts for tier-1 tables, partition counts, ACID checks, Ranger allow/deny tests, and a timed drill. The 14-hour failover was an operational gap, not a replication gap; time each step and assign owners.
3. The 1 Gbps link is 60% utilized. How would you meet the RPO without starving daytime traffic, and what would you tell the business about Kudu?
Read the worked answer
Use bandwidth controls and scheduling: set Replication Manager bandwidth limits and run bulk HDFS replication outside the daytime peak, keeping incremental metadata and critical tables on a shorter cycle. Compress and deduplicate where possible, exclude non-critical data, and prioritize tier-1 tables and metastore/Ranger exports. Measure whether the 4-hour RPO is achievable at the allowed bandwidth; if not, either add link capacity or reduce scope. For Kudu, be explicit: file-based replication cannot deliver a 15-minute RPO; application dual-write or a Kudu-specific backup/restore is required, and dual-write has latency and consistency costs. If the business will not fund that, it must accept a longer Kudu RPO in writing. Do not let a single 1 Gbps link carry every component at every RPO without prioritization.