Heartbeat and RPC Failures After a Firewall Change
TransitCo runs Secret Server 11.x on-premises with two Distributed Engines. After a firewall change, heartbeat failures rose from 2% to 35% in one hour. Most failures are Windows local administrator accounts on 300 servers in a branch site; Linux SSH secrets still work. The branch engine is healthy and can reach Secret Server on 443. The firewall team blocked SMB and RPC from the engine to server subnets except for one /24 because of a ransomware incident. The Secret Server RPC account is a domain account with local admin on all servers, and its password was changed last week by the domain team without updating Secret Server. Operations wants to disable heartbeat to stop alerts. The CISO wants the cause, a safe fix, and a plan to prevent recurrence.
1. Diagnose the likely causes and the order in which you would check them.
Read the worked answer
The scope points to two likely causes acting together. Linux SSH secrets still work, so the engine-to-Secret Server path and the engine itself are probably healthy. Windows local administrator heartbeat depends on SMB/RPC reachability to the target servers; the new firewall rule blocked all but one /24, which would explain failures concentrated in the branch. The RPC account password change is a second suspect: if the domain team changed it and Secret Server still holds the old value, heartbeat and RPC will fail even where ports are open. Check engine logs and heartbeat failure text first, then test one affected server from the engine, then verify the RPC account credential in Secret Server. Do not disable heartbeat; that removes detection and hides the problem.
2. What safe remediation and longer-term controls would you recommend?
Read the worked answer
Restore only the required connectivity from the branch engine to the specific server subnets on the required SMB/RPC ports, not a broad any-any rule. Update the RPC account credential in Secret Server, test heartbeat on a small pilot group, and then let the normal schedule resume. If the account is locked out, coordinate with AD before retrying repeatedly. Stagger heartbeat schedules to avoid a load spike. Longer term, use a site-scoped RPC account with rights only in that branch instead of a domain account with local admin everywhere, and document the firewall dependencies as part of change management. Add alerting on heartbeat failure rate and on RPC credential failures. The goal is to restore detection and rotation safely, not to silence alerts.
3. How would you reduce the blast radius of the RPC account itself?
Read the worked answer
Replace the single domain-wide RPC account with per-site or per-scope RPC accounts that have only the rights needed to change local administrator passwords on their target servers. Store each RPC account as its own secret with restricted View and Edit permissions, and assign it to the Distributed Engine or Site configuration for that scope. Do not give the RPC account interactive logon or broader domain rights. Rotate the RPC account through a controlled process and update Secret Server before the old credential expires. Monitor its use and lockout events. This limits the damage if one RPC credential is compromised and makes firewall and credential changes easier to reason about. Document the account-to-site mapping so a future AD password change does not silently break heartbeat.