Hub-and-Spoke over ExpressRoute: When a Private Endpoint Call Times Out
Contoso connects its Chicago datacenter to Azure over a 1 Gbps ExpressRoute circuit that terminates in a hub VNet in East US 2. Spokes host prod, nonprod, and dev workloads, and a fourth spoke runs in West Europe. On-premises address space is 10.10.0.0/16; Azure uses 10.20.0.0/16. Policy states that all internet-bound egress must pass through an Azure Firewall in the hub, and that prod must not reach nonprod by any path. On-premises servers must resolve the private endpoint names of PaaS services. Recently the team added a 0.0.0.0/0 route to the firewall and pointed spoke DNS at an on-premises domain controller. Now VMs can browse the internet, but calls to a storage private endpoint in the hub time out, and nslookup from the same VM returns a public IP address. The circuit carries no encryption.
1. Diagnose the private endpoint failure. Give the minimal set of changes to restore connectivity without weakening the egress policy.
Read the worked answer
Two symptoms, one root cause plus one consequence.
Root cause: name resolution. The VM returns a public IP, so the privatelink.blob.core.windows.net zone is either not linked to the spoke VNet or the on-premises domain controller has no conditional forwarder for it. Confirm by querying the Azure-provided resolver (168.63.129.16) directly from the VM; if that returns the private address, the fault is purely DNS.
Consequence: traffic path. Because the name resolves publicly, the connection is sent to the storage account's public endpoint, which is presumably firewalled off or has public network access disabled. This is not a routing bug in the UDR: the private endpoint's host route is a more specific prefix, so longest-prefix match would still have used it had the right address been resolved.
Minimal fix
- Link the private DNS zone to every spoke VNet that must resolve it, or host Azure DNS Private Resolver in the hub and forward to it.
- Add conditional forwarders on-premises for the privatelink zones to that resolver.
- Verify the private endpoint subnet NSG permits 10.20.0.0/16 and that the effective routes on the spoke show the endpoint prefix.
Do not solve this by adding an internet egress rule through the firewall; that bypasses the private path entirely.
2. Design prod-to-nonprod isolation at the lowest reasonable cost. Explain why peering alone does not deliver it.
Read the worked answer
VNet peering is not transitive. Prod and nonprod both peer with the hub, and their default next hop for each other's address space is the hub gateway or firewall, not a direct path. So isolation is enforced at the hub, with NSGs as defense in depth.
Design
- Azure Firewall policy in the hub with explicit allow rules for on-premises and internet egress, and no rule permitting 10.20.prod to 10.20.nonprod. Traffic with no matching rule is denied, which is the safe default.
- Spoke route tables sending 0.0.0.0/0 and the sibling spoke prefixes to the firewall's private IP, so no spoke-to-spoke traffic can shortcut through the peering.
- NSGs on subnets denying inbound from the other environment's prefix, so a future firewall misconfiguration does not become a breach.
- Separate service principals and RBAC scopes per environment; network isolation alone is not a security boundary for the control plane.
Do not build a second hub per environment for this: two firewalls, two gateways, and duplicated ExpressRoute peering roughly doubles the fixed monthly cost for a requirement that a default-deny rule set already satisfies. Revisit only if the two environments need different regulatory scopes or different on-premises circuits.
3. The CISO asks how the circuit is protected in transit, and the CIO asks what happens if it fails. Address both with specific mechanisms.
Read the worked answer
Encryption in transit. ExpressRoute does not encrypt data by default; it provides a private, non-internet path, which is a different property. Options, in order of preference:
- MACsec on ExpressRoute Direct ports, giving link-layer encryption. Only available with ExpressRoute Direct, not provider-managed circuits.
- IPsec site-to-site VPN over the ExpressRoute private peering, which is supported and encrypts over the private path.
- Application-layer TLS for every protocol that supports it, which is the one control that survives the circuit topology changing.
Circuit failure. Run a Site-to-Site VPN as an active-passive backup in the same GatewaySubnet region, with BGP on both. Advertise the same prefixes over both paths and use AS-path prepending on the VPN so ExpressRoute is always preferred when healthy; when the circuit drops, the VPN path is used automatically. Size the VPN gateway expecting a fraction of the circuit's throughput, and rehearse the failover. Note the VPN backup is only a resilience control; it does not satisfy the encryption requirement unless you also treat it as the encrypted path.