Data Hub launch fails only in the newly added subnet

Fault domain. The failure is below Cloudera Manager: host bootstrap has three external dependencies - the Kerberos/LDAP identity service (FreeIPA), the object-store endpoint, and package repositories. A bootstrap that installs CM but never registers hosts usually means a host could not complete kinit/LDAP bind or could not reach storage. The evidence already triangulates it:

• Inbound SSH works → the node's own security group is not blocking inbound management traffic.
• DNS resolves FreeIPA → not a resolver or DHCP option-set problem.
• TCP 88 times out from dh-c only, while dh-a succeeds → a path/filtering difference between the two subnets, not a FreeIPA service failure (a dead service would fail from dh-a too).

Ordered checklist:

1. FreeIPA security group ingress. Most likely it is scoped to the dh-a CIDR (or to a list of CIDRs that predates dh-c), so ports 88/464/749/389/636 are not permitted from the new range.
2. Network ACLs on both subnets. They are stateless: if the ephemeral return-port range is not allowed on the FreeIPA side, the TCP handshake dies exactly as observed.
3. Route table for dh-c. Confirm there is a route to the FreeIPA subnet or peering/VPC attachment; if FreeIPA is in a peered VPC, the return route on that side is often forgotten.
4. Only then the S3 gateway endpoint route and bucket policy for the new subnet's route table - a second, later bootstrap failure waiting to happen.

Why not CM logs first: they will show an agent that cannot obtain credentials, which is a symptom already implied by the launch log. Reading them costs the remaining window without narrowing the layer. Logs come after connectivity is proven.

Terminate and relaunch, after fixing the network. A repair is attractive because it preserves the hours already spent, but this cluster never reached a coherent state: 9 of 12 hosts did not register, so principals, CM database rows and node-level packages are partially applied. Repairing unregistered hosts frequently leaves duplicate or stale Kerberos principals and half-configured roles, and the failure mode of a repair is another 25-minute wait followed by an ambiguous error - the worst outcome with three hours left.

Preconditions before relaunch:

1. FreeIPA security group and NACLs updated for the dh-c range, applied to both directions.
2. Independent proof of the path: launch a throwaway EC2 instance in dh-c and run nc -vz to 88, 749 and 389, a real kinit with a test principal, and an aws s3 ls against the Data Lake bucket using the instance profile. Do not accept "the security group looks right".
3. S3 gateway-endpoint route present in the dh-c route table and any bucket policy that restricts source VPC endpoint or CIDR updated.
4. Instance quota confirmed for 12 nodes in the region/AZ; a quota failure at minute 25 would burn the window twice.
5. Confirm no orphaned EC2 instances from the failed attempt remain, so names and host entries do not collide.

Then relaunch from the identical template and keep the old failure's logs for the post-change review. The relaunch is deterministic and testable; the repair is a bet.

The root cause is a change that was validated at one layer ("the CDP environment accepted the subnet") and never at the layer that matters ("a host in this subnet can reach every shared service"). Control-plane acceptance says nothing about data-plane connectivity.

Add to the procedure:

• A connectivity pre-flight per new subnet, executed by a test instance before any cluster launch: FreeIPA ports, object-store access with the IDBroker/instance role, package repository reachability, DNS forward and reverse for the Data Lake endpoints.
• Security-group referencing instead of CIDR lists. Put Data Hub nodes in a dedicated SG and allow the identity-service ports from that SG rather than from hand-maintained CIDR ranges. This class of change then needs no edit at all.
• Treat subnet addition as a full change, with the same review, rollback and validation as a cluster launch, not as a network ticket.
• Codify the environment (Terraform or CDP CLI scripts) so subnet, route table, endpoint route and security-group rules are created together and reviewed as one diff.
• A post-launch smoke test as an acceptance gate: kerberos ticket, one Hive query, one Spark job writing to storage. A cluster that can do those three things will not surprise you at 02:00.

Finally, record the failure mode in the runbook: "bootstrap stalls, hosts not registering, one subnet only" should point the next engineer at identity-service reachability in minutes, not hours.