Private Endpoint Reachable from Nowhere: DNS and Forced Tunneling
Northwind runs a hub-and-spoke network. The hub VNet is 10.0.0.0/16 and contains Azure Firewall at 10.0.1.4 plus subnet 10.0.2.0/24 reserved for private endpoints. Spoke VNet 10.1.0.0/16 has 40 VMs in 10.1.1.0/24; that subnet has a UDR sending 0.0.0.0/0 to the firewall, and the firewall policy allows the spoke to reach the Storage service tag on 443. A new storage account has public network access set to Disabled and a private endpoint in 10.0.2.0/24. The privatelink.blob.core.windows.net zone is linked only to the hub VNet. On-premises servers reach Azure over ExpressRoute and use on-premises DNS, which conditional-forwards azure.net lookups to two domain controllers in the hub VNet; those DCs forward to 168.63.129.16. Symptoms: spoke VMs resolve the account to a public IP and get 403 responses; on-premises backup jobs fail with name resolution errors.
1. Explain both symptoms and give the fix for each.
Read the worked answer
Spoke VMs: the private DNS zone is linked to the hub VNet only. A VNet link is what makes a private zone authoritative for a VNet's resolvers, and links are not transitive across peering. The spoke therefore falls through to public DNS, gets the public endpoint, and with public network access disabled the request is refused (403).
Fix: add a VNet link to the spoke VNet (registration disabled) for privatelink.blob.core.windows.net. Then verify the spoke UDR: 0.0.0.0/0 sends the flow to the Azure Firewall, so the firewall policy must allow 10.1.1.0/24 to 10.0.2.x on 443, and the NSG on the private-endpoint subnet must allow it inbound.
On-premises: the on-premises servers query on-premises DNS, which conditional-forwards to the hub DCs. The DCs are in a linked VNet, so their own lookups resolve correctly - but only if the conditional forwarder on-premises actually names the privatelink zone. Forwarding azure.net is not enough; the private zone is privatelink.blob.core.windows.net.
Fix: add an explicit conditional forwarder on-premises for privatelink.blob.core.windows.net to the hub DCs, and confirm the DCs' forwarders point at 168.63.129.16. Alternatively deploy Azure DNS Private Resolver in the hub as the single forwarding target for on-premises.
2. An engineer proposes dropping the private endpoint and using a service endpoint on the app subnet instead, since the storage account is already in the same region. Evaluate this proposal.
Read the worked answer
It solves less than it appears to and it breaks the inspection model.
- Service endpoints require the public endpoint. They optimize the route to the storage account's public IP; they do not work when public network access is set to Disabled. You would have to re-enable public access and restrict it with a virtual network rule.
- Service endpoints bypass the firewall. Enabling a service endpoint injects a more specific system route for the service prefix, which wins over the 0.0.0.0/0 UDR. Traffic from the app subnet would no longer traverse Azure Firewall, so the policy you built becomes decorative for that flow.
- On-premises can never use it. Service endpoints are subnet-scoped inside Azure; the backup servers over ExpressRoute still need the private endpoint or a public allow-list.
- Blast radius. A virtual network rule admits the whole subnet, not the workload.
Keep the private endpoint. If the objection is cost, the defensible alternative is a single private endpoint reused by all consumers in the hub, which is already the design.
3. How do you prove the fix end to end before declaring the incident closed?
Read the worked answer
Test from each resolution path, because they fail independently:
- From a spoke VM:
nslookup <account>.blob.core.windows.netmust return the private endpoint IP in 10.0.2.0/24, andTest-NetConnection <account>.blob.core.windows.net -Port 443must succeed. Then run an actual blob operation with a managed identity or SAS - resolution and authorization are separate failures and you want to see a 200, not just an open socket. - From an on-premises server: resolve the same FQDN against the on-premises DNS server it is actually configured to use, then against the hub DC. Both must return the private IP. A pass on the DC alone does not prove the client path.
- From the firewall: confirm the allow rule is hit in the firewall log for the spoke-to-private-endpoint flow - a silent drop looks identical to a DNS problem from the client side.
- Negative test: confirm the public IP of the account is no longer resolvable/reachable from a network that should not have access, proving public access really is disabled.
Record all four results; the recurrence risk here is that someone re-creates the private DNS zone or breaks a VNet link during a later deployment.