Hot Wallet Compromise at a Cross-Chain Bridge
A bridge operator secures roughly $40M in pooled assets. The signing service runs on three nodes in one cloud region, using a 2-of-3 threshold key. At 02:14 an on-call engineer sees an anomalous outbound transfer of $6.2M to an unknown address. Logs show the transfer was authorized by two signatures. One signer node's SSH key was reused from a 2023 incident and never rotated; its session shows a login from an unfamiliar ASN 40 minutes before the transfer. The bridge's withdrawal limit is $10M per transaction with no daily cap. The incident commander must decide within 30 minutes whether to pause the bridge (halting $1.8M/hour in legitimate volume and triggering SLA penalties) or allow operations while investigating. The team has no pre-written runbook for key compromise.
1. What should the incident commander do first, and how should the decision be justified against business impact?
Read the worked answer
Pause the bridge. The evidence already meets a reasonable threshold for confirmed key compromise: an unexplained signer login from a new ASN, a reused credential with known prior exposure, and an outbound transfer to an unknown destination. A 2-of-3 threshold provides no protection once one key is compromised and one signer is socially or technically induced to co-sign; the remaining honest node cannot stop a valid quorum.
Weigh the costs explicitly. Legitimate volume loss is about $1.8M/hour plus SLA penalties, but the exposed pool is roughly $40M and the per-transaction cap of $10M with no daily cap means an attacker could drain the pool in four transactions. Expected loss from continuing clearly exceeds the cost of pausing. Pausing is reversible; an on-chain transfer is not.
Sequence: pause withdrawals, rotate or revoke the compromised signer's credentials and SSH key, invalidate sessions, then preserve logs and snapshot the signing service before remediation. Notify counsel and compliance per the incident policy. The absence of a runbook is itself a finding to remediate afterward.
2. What control failures does this incident reveal, and which should be prioritized?
Read the worked answer
Several layered failures:
- Key management: a 2-of-3 threshold all hosted in one cloud region and one provider is a single point of failure. Geographic and provider diversity, plus hardware-backed key storage, would raise the bar.
- Credential hygiene: an SSH key from a 2023 incident was never rotated. A documented rotation schedule with enforced expiry would have closed this.
- Transaction limits: a $10M per-transaction cap with no daily aggregate cap is inadequate for a $40M pool. A daily velocity limit and anomaly-based hold would have limited loss.
- Monitoring and response: detection came from a human at 02:14, not automated alerting on new-ASN signer logins. Automated alerts on signer authentication anomalies are high priority.
- Runbook gap: no key-compromise runbook delayed a decision that should have been automatic.
Prioritize by risk reduction per effort: daily caps and automated signer-login alerts are cheap and fast; key custody redesign is expensive but necessary. Rotation enforcement sits in between.