Staging Cache Bleed into Production After a Hotfix
A Drupal site runs on Acquia Cloud with a Dev, Stage, and Prod environment. Prod sits behind Acquia Cloud CDN and uses Varnish in front of a 3-web-node cluster; anonymous pages are cached aggressively (TTL 6 hours). A content editor reports that a corrected price on a product page still shows the old value in Prod 90 minutes after the change, but the correct value appears when logged in. The change was made by editing a node and saving it; no code was deployed. The team recently ran a drush cr on Stage while debugging, and Stage shares the same CDN origin configuration template as Prod. The page is anonymous-only and served from a single URL with no query string. Monitoring shows web node CPU is low and database load is normal.
1. Diagnose the most likely layer still serving stale content, and explain why the logged-in view differs.
Read the worked answer
The difference between anonymous and authenticated views is the key signal. Authenticated users bypass Varnish and CDN (Acquia Cloud does not cache responses for authenticated sessions by default), so they hit Drupal and see the new value. Anonymous users are served from a cache layer.
Working outward from Drupal: Drupal's render/page cache would normally be invalidated on node save, so a correctly configured site should not serve stale Drupal cache after an edit. That leaves Varnish and CDN. Because the change was a content edit with no deployment, the most likely culprits are (a) the CDN edge holding the object for its TTL, and (b) Varnish holding the object for its 6-hour TTL.
The mention of a drush cr on Stage is a distractor unless Stage and Prod share a cache namespace or origin - verify that the CDN origin hostname and cache-key configuration are actually distinct per environment. If they are distinct, the Stage purge is irrelevant to Prod. Confirm by requesting the URL directly against a web node (bypassing CDN) and then against the CDN; whichever still shows the old value localizes the problem.
2. Describe the corrective action and the preventive control you would put in place.
Read the worked answer
Corrective: purge the specific URL at the CDN and invalidate the Varnish object for that path (Acquia provides cache clear via the UI/API and drush-driven purges). Purge the narrowest scope possible - the single path - rather than a full site purge, to avoid a thundering herd on the origin.
Preventive controls:
- Confirm environment isolation: CDN origin, Varnish cache keys, and any shared cache prefixes must differ between Stage and Prod. A purge in one environment must never touch another.
- Add tag-based invalidation: configure Drupal to emit cache tags and have the purge module invalidate CDN/Varnish on node save, so content edits don't wait for TTL expiry.
- Reduce anonymous TTL for volatile content types (e.g., product pages) or split TTLs by content class.
- Add a post-deploy/post-edit smoke check that curls the anonymous URL through the CDN and asserts the expected value.
Note that a full cache clear on Prod is a blunt instrument; prefer targeted purges and verify origin health before purging so the origin can absorb the refill traffic.