Micro-Segmentation Rollout Without Breaking a Live Tier
An operator is introducing micro-segmentation on a CX 10000 pair protecting a live 3-tier app (web/app/db). Today all tiers share one flat subnet and rely on host firewalls. The plan: enforce default-deny between tiers at the DPU, permit web->app:443 and app->db:5432, and log denies. During a pilot on one rack, the app tier loses db connectivity intermittently; host firewalls were also recently tightened. The operator wants to flip enforcement fabric-wide within the same change window to 'get it done.'
1. What is wrong with the rollout plan, and how should enforcement be staged?
Read the worked answer
Two errors: enforcement before visibility and fabric-wide blast radius in one window.
Correct staging:
- Monitor-only first: deploy the policy in log/observe mode, collect actual flows per tier, and confirm the intended permits cover real traffic (including health checks, monitoring, backups, and DNS on unexpected ports).
- Reconcile with host firewalls: the intermittent db loss during pilot suggests either the DPU deny or the tightened host firewall is dropping a flow the other expects. Correlate DPU deny logs with host firewall logs on the app and db VMs before enforcing.
- Enforce per rack, canary first: flip one rack, observe for a defined soak period, then proceed rack by rack - not fabric-wide in one window.
- Keep a fast rollback: pre-staged revert to monitor mode.
Never enforce default-deny on a live tier without a flow-visibility phase; and never make the first enforced change fabric-wide.
2. How do you determine whether the intermittent db loss is the DPU policy or the host firewall?
Read the worked answer
Correlate from both sides on the same timestamps:
- DPU side: check deny counters/logs for app->db:5432 and any ephemeral return ports; look for resets at the DPU. If the DPU is permitting but the connection still fails, suspicion shifts to the host.
- Host side: check the db VM's host firewall logs for drops from app-tier source IPs/ports at the same times.
- Flow symmetry: confirm return traffic takes the same DPU path; asymmetric return causes stateful drops that mimic host-firewall symptoms.
- Port coverage: verify the permit includes return/ephemeral ranges and any db health-check ports, not just 5432.
- Controlled test: temporarily widen one variable at a time (e.g., permit all app->db in monitor mode, or relax the host rule on one db VM) and observe. Change one thing per test.
Conclude only when DPU and host evidence agree on the drop point.