Plant Site with a WAN That Drops Daily
A manufacturer runs Secrets Manager Enterprise with a leader in data centre A and a standby in data centre B, both on-premises VMware. Workloads include 3,000 containers on a managed Kubernetes service in eu-west-1 and 40 application servers at a plant in southern Brazil. The plant's WAN link to Europe drops for 10 to 25 minutes on most days and has 180 ms latency. Firewall policy forbids plant-to-cloud traffic except through a forward proxy, and the plant network is a separate security zone with no inbound rules. Plant applications retrieve database credentials at shift start and refresh them every four hours; a forced restart of the line takes 12 minutes and stops production. The platform team proposes opening the plant firewall directly to the leader so applications can call the API without a local component, arguing that replicating to a remote site will be slow and hard to support. A second proposal is to cache credentials in a local file with a 24-hour TTL.
1. Evaluate both proposals and recommend a topology for the plant.
Read the worked answer
Both proposals treat the symptom (latency and outages) by weakening a control.
Proposal 1 - direct plant-to-leader access: it breaches the stated firewall policy and puts a daily 10-25 minute hard dependency on a link that is known to fail. Shift start is exactly when the link is most likely to be needed, and a 12-minute restart cost makes any failed retrieval a production event. It also expands the leader's attack surface to a third network zone and gives you no local fault domain.
Proposal 2 - 24-hour file cache: availability improves, but you have recreated the problem the project exists to solve. The file is an unmanaged, copyable secret with no requester identity in the audit trail, it will be stale after any rotation, and a 24-hour TTL means a revoked credential keeps working for a day. It also cannot answer the auditor's question 'who read this credential'.
Recommendation: deploy a follower (replica) in the plant's application zone, replicating from the leader over the existing outbound-only proxy path. Applications resolve to the local follower endpoint; reads stay inside the security zone, survive WAN outages, and drop latency from 180 ms to local. Size it for 40 hosts' request rate with headroom, encrypt its storage because the plant is now part of the trust boundary, and treat it as a read-only replica: never load policy or originate rotation there. Keep short-lived in-process caching in the application only as a second-order optimisation, with a TTL measured in minutes.
2. A three-hour WAN partition occurs. What still works at the plant, and what must be coordinated once the link returns?
Read the worked answer
Still works: reads against data already replicated - variable values and host entitlements are local, so shift-start and four-hourly retrievals succeed against the follower's copy. Authentication for existing host identities should also be local; confirm this in a lab test rather than assuming it, and check the token TTL against the outage duration, because tokens that expire mid-partition and require a leader round trip will fail at the worst moment.
Does not work: policy loads, new host onboarding, rotation initiated at the leader, and any write-path operation. The plant keeps serving the last replicated value.
Coordination on reconnect:
- Rotation is the real hazard. If the leader rotates a database password during the partition, the plant databases may already accept only the new value while plant apps still hold the old one, or vice versa depending on how the target enforces change. Freeze rotation of in-scope plant credentials during known outage windows, or schedule plant restarts after replication catches up.
- Verify replication lag before declaring recovery; a follower that reconnects but is behind will serve stale values silently.
- Audit records generated at the plant must buffer locally and ship on reconnect. Size the buffer for the longest expected outage and test that a three-hour backlog survives a follower restart, otherwise you have an audit gap exactly when you need the evidence.
- Alert on partition duration, not just on link state, so the operator knows when the local buffer or token TTL is about to be exhausted.
3. Design the monitoring that would tell you the plant is degraded before an operator notices.
Read the worked answer
Monitor the thing the business depends on, not just process liveness.
Signals to collect:
- Synthetic retrieval: a canary host at the plant authenticates and fetches a non-production variable every five minutes, end to end, and alerts on two consecutive failures. This is the only signal that proves the whole path (identity, auth, entitlement, network) works.
- Replication lag: time since last successful replication from the leader, and a count of pending changes. Alert at a threshold well inside the daily outage window, and page if lag exceeds the point where a rotation would be missed.
- Partition duration: alert when the follower has been unable to reach the leader for longer than 15 minutes, with severity escalating at the point where the local audit buffer or token TTL is at risk.
- Token issuance failure rate at the plant, split by authenticator, so you catch auth problems that do not affect already-cached sessions.
- Audit buffer depth and forwarding success, so a silent evidence gap is caught in hours rather than at audit time.
What to ignore: raw CPU and memory on the follower are useful capacity signals but say nothing about whether the plant can read a secret. Keep them as a secondary dashboard.
Pair the monitoring with a documented runbook: partition detected - confirm reads still succeed - freeze plant credential rotation - on reconnect, verify lag is zero and audit backlog drained, then confirm the next scheduled rotation completes.