Single-AZ Data Lake must become resilient without a rebuild window
Eighteen months ago a CDP environment was provisioned with a non-HA Data Lake in a single availability zone, to move fast. It now underpins nine Data Hub clusters, three Cloudera Data Warehouse virtual warehouses, two Cloudera Data Engineering virtual clusters and about 400 users; data lives in S3 buckets. Last month an AZ-level incident took the environment down for 90 minutes, and every workload failed, including read-only dashboards. Leadership has approved budget for "HA" and wants it delivered within six weeks. The platform team has proposed adding a second availability zone to the existing Data Hub clusters, arguing that this addresses the resilience gap. A separate team owns the storage buckets and has a change freeze on bucket policies for another quarter.
1. Evaluate the proposal to add a second AZ to the Data Hub clusters. What does it fix and what does it not fix, and what is your recommendation to leadership?
Read the worked answer
What it fixes: nothing material for this incident. The Data Hub multi-AZ change improves resilience of compute against a single AZ failure for those clusters, but the 90-minute outage was driven by the shared Data Lake - the SDX layer that supplies Kerberos and LDAP, authorization policy, metadata and the gateway that all nine Data Hubs, the warehouses and the engineering clusters depend on. Lose that and every workload fails regardless of how many AZs the compute sits in.
What it does not fix: the single point of failure, and the recovery time. It also adds cost and complexity to clusters that are not the constraint.
Recommendation to leadership: state plainly that HA for a Data Lake is a property decided when the environment is provisioned; there is no supported in-place conversion of an existing single-AZ Data Lake into an HA one. Verify this against the current release documentation before committing budget, but do not plan a migration on the assumption that a toggle exists. That leaves two honest paths:
- Parallel HA environment and migrate - the only path that actually removes the single point of failure.
- Accept the risk and mitigate - documented RTO/RPO, tested recovery runbook, cross-region backup of policy and metadata, and a communication plan. Cheaper, but the risk stays.
Recommend path 1, scoped as a migration programme, and reject the Data Hub multi-AZ proposal as a substitute - it spends money without addressing the failure that occurred.
2. Produce the sequencing for building the HA environment and migrating workloads. What has to be inventoried, what order do the workloads move, and where are the traps?
Read the worked answer
Phase 0 - inventory and dependencies. Catalogue everything that is not data: Ranger policies and zones, Atlas classifications and lineage sources, Hive metastore database and table definitions, Knox topologies and any external clients pointing at gateway URLs, IDBroker role mappings and workload credentials, SSO/IdP configuration, CDW/CDE/CML definitions, Data Hub templates and cluster definitions, scheduling and external orchestrators. Anything not in this list will be discovered on cut-over night.
Phase 1 - build in parallel. Provision the new environment with an HA Data Lake in the same region. Do not touch the existing environment; it must stay fully functional as the fallback.
Phase 2 - storage decision. The buckets are owned by another team under a change freeze, which is the biggest scheduling trap. Preferred: new buckets, copy data, then migrate the policies and IDBroker access in one coordinated change. Reusing existing buckets means the new environment's roles need bucket-policy changes that the freeze forbids - either negotiate an exception early or design around it. Cross-account and endpoint restrictions are where this stalls.
Phase 3 - metadata and policy migration. Recreate policies and zone structure in the new Data Lake, import metastore definitions, validate classifications. Do this before workloads, because nothing works without it.
Phase 4 - pilot and cut over per workload. Move a low-risk Data Hub plus one warehouse first; validate Kerberos, gateway access, masking and a representative query set with real users. Then move workloads in order of increasing criticality, each with a freeze window, a rollback to the old environment, and a named owner. Retire the old environment only after a full billing and access cycle has passed.
Traps: external clients with hard-coded gateway URLs; scheduled jobs still pointed at old endpoints; policies that exist only in the old environment's UI; and discovering during cut-over that a workload's identity mapping was undocumented.
3. Six weeks is aggressive. What do you tell leadership, and what interim mitigations do you put in place while the migration proceeds?
Read the worked answer
Give leadership a dated, risk-stated plan rather than either a refusal or a promise you cannot keep.
What to say: six weeks is achievable for a pilot environment and one or two low-risk workloads; it is not credible for moving nine Data Hubs, three warehouses, two engineering clusters and 400 users with a storage change freeze in force. Present the plan as phases with go/no-go points, name the external dependencies (bucket policy freeze, security review, user communications) as the schedule risks they are, and state the residual risk for each phase. Ask for a decision on the storage approach, because that is the item that can move the end date by weeks.
Interim mitigations, all deliverable quickly:
- Publish and test an AZ-failure runbook with a measured RTO, including how to determine whether the Data Lake is recovering or must be restored.
- Back up the metastore database, Ranger policy export and Atlas metadata to a second region on a schedule, and verify a restore. Untested backups are assumptions.
- Negotiate the bucket-policy exception early, or confirm that the new environment can use new buckets.
- Fix the alerting gap exposed by the incident: the platform team learned about the outage from users.
- Communicate the maintenance and incident model to the 400 users: what to do during an AZ event, where status is posted, who is on call.
Finish with the honest framing: the migration removes the single point of failure; the interim measures only reduce the duration and confusion of the next one.