Sizing a DPU-Bound Firewall Policy for a Bursty East-West Workload
A CX 10000 pair enforces stateful policy on traffic between a 3-tier application: web (12 VMs), app (20 VMs), and db (6 VMs). The app tier issues short-lived, high-concurrency connections to the db tier (each app VM peaks near 40k concurrent TCP flows for ~90 seconds during batch windows, twice per day). The security team wants default-deny between tiers with permits only web->app:443 and app->db:5432. During the first batch window, db-tier flows intermittently reset and the DPU reports flow-table pressure while CPU stays low. The team proposes widening the permit rules to 'any' between tiers to stabilize traffic.
1. Diagnose the resets and evaluate the proposal to widen the rules. What is the correct remediation path?
Read the worked answer
The symptom pattern (intermittent resets during bursts, low CPU, DPU flow-table pressure) points to stateful session-table exhaustion, not a rule-matching problem. Stateful firewalls on the DPU must hold a state entry per concurrent flow, plus aging overhead; 20 app VMs x ~40k flows = ~800k simultaneous sessions against the db tier, well above typical per-policy or per-DPU session ceilings.
Widening to any does not reduce state consumption - it may increase it, since more flows become stateful candidates and the same concurrency still exists. It also destroys the segmentation intent.
Correct path:
- Confirm the ceiling: read the DPU session/flow counters and the per-policy limits during a batch window.
- Reduce concurrency: enable connection pooling/reuse at the app tier, or cap per-VM pool sizes so aggregate sessions stay under the ceiling.
- If concurrency is genuinely required, spread load across both CX 10000s (ECMP/hashing) and verify the session budget per DPU.
- Only then re-test; keep default-deny and the narrow permits.
Rule of thumb: rule count affects lookup cost; concurrent sessions affect table pressure. Tune the workload or the capacity, never the security posture.
2. How would you prove the firewall is not the cause before escalating to the DPU vendor?
Read the worked answer
Isolate variables:
- Baseline without state: temporarily put the db tier behind a stateless ACL permit (no state tracking) in a lab replica and replay the batch load. If resets vanish, state tracking is implicated.
- Counter correlation: align timestamps of resets with DPU flow-table high-water marks and session-aging events; a reset that coincides with a table-full counter is strong evidence.
- Rule-hit counters: verify permits are matching as intended (web->app:443, app->db:5432) and that no implicit deny is catching retries.
- Asymmetric path check: confirm return traffic traverses the same DPU; asymmetric routing causes stateful drops that look like random resets.
- Aging tuning: test a longer TCP established timeout to see if short-lived flows are being aged prematurely.
Document counters before/after each change so the vendor case is evidence-based, not anecdotal.