The Flush Agent That Missed One Dispatcher

The failure is most likely in Dispatcher invalidation, not in replication. Publisher-1 receives the activation and its flush agent invalidates dispatcher-1 and dispatcher-2, but dispatcher-3 is never told that /content/acme/us/en/campaign changed. Dispatcher-3 therefore keeps serving the cached page until the 600-second TTL expires. Because there are three Dispatchers behind the load balancer, only the requests routed to dispatcher-3 are stale, which matches the roughly one-third symptom.

Evidence to collect first:

• The flush agent transport URLs on publisher-1, and whether publisher-2 has any flush agent at all.
• Dispatcher-3 access/error logs around the activation time, looking for missing or rejected flush requests.
• Response headers such as X-Dispatcher and cache-status headers from repeated anonymous requests, to confirm that stale responses come from one Dispatcher.
• The Dispatcher allowedClients setting on all three instances, to rule out a rejected flush from publisher-1.

If all three Dispatchers were stale, I would suspect TTL or a bad statfile level. If none were stale, I would suspect no invalidation problem. One-third stale points to a single Dispatcher not receiving valid flushes.

With statfileslevel "2", the Dispatcher creates stat files at depth two of the content tree, for example /content/acme. Invalidating /content/acme/us/en/campaign updates the stat file at /content/acme, so every cached page under /content/acme whose cached stat time is older is treated as stale. The campaign update therefore causes broader collateral invalidation than necessary.

To narrow the blast radius, raise the level so the stat file sits closer to the changed content. At level 4 the stat file would be at /content/acme/us/en; at level 5 it would be at /content/acme/us/en/campaign. Level 4 limits invalidation to the English site, while level 5 limits it to the campaign page. The tradeoff is operational overhead: higher levels create more stat files and more file-system activity, and they require a consistent content depth. I would choose level 4 or 5 only after confirming the site tree depth and Dispatcher file-system capacity. Invalidation breadth is not the cause of the one-third stale symptom here, but it is a separate efficiency issue worth correcting.

Fix the invalidation path first: add dispatcher-3 to publisher-1's flush agent transport URLs, and add a correctly configured flush agent to publisher-2 so either publisher can invalidate the full farm. Confirm that each Dispatcher's allowedClients includes both publisher IPs; otherwise the newly added flushes will be rejected.

Verification sequence:

1. Activate a harmless test page and watch the flush agent queue and logs on both publishers.
2. Request the page repeatedly while checking X-Dispatcher so that all three Dispatcher instances are exercised.
3. Confirm the old hero disappears from dispatcher-3 without waiting for the 600-second TTL.
4. Repeat with publisher-2 as the activation target to prove redundancy.

To prevent recurrence, make the Dispatcher inventory part of the deployment checklist and configuration management: every new Dispatcher must be added to every publisher flush agent, and the allowedClients list must be updated in the same change. Monitor flush agent queue length and Dispatcher flush errors, not just replication queue health.