SSH Brute Force, Stale Keys, and a Stolen Laptop

Order: preserve, contain, then harden.

1. Preserve evidence first - copy auth.log*, last -aiF, lastlog, and journalctl _comm=sudo off the host before you edit anything. Local-only logs with four weeks of retention mean anything you rotate away is gone permanently, and you may already be outside the window where compromise would be visible.
2. Contain the realistic path. The brute force is noise; the credible risk is a key. Remove the departed contractors' keys and the stolen key from authorized_keys on all twelve hosts via configuration management, and disable the shared deploy account's interactive login. Do this from an already-open session.
3. Harden sshd safely - PermitRootLogin prohibit-password, PasswordAuthentication no, AllowUsers, then sshd -t to validate and systemctl reload sshd (reload, not restart) so the live session survives. Sequence this after key distribution so you cannot lock everyone out.
4. Then consider fail2ban or IP filtering. Blocking 40 IPs of automated scanners is cosmetic: they rotate addresses daily, and if an attacker already had a working key, no password attempt would ever appear. Spend that hour on key hygiene and evidence instead.

Start with grep -E "Accepted" /var/log/auth.log* and enumerate every accepted login with source IP, user, and method, then reconcile against known admin egress IPs and VPN ranges. Any Accepted publickey for a departed contractor or for deploy from an unfamiliar address is a confirmed incident. Cross-check last -aiF for sessions that do not match ticket records, and look for Accepted entries followed immediately by sudo commands.

Beyond auth.log:

• journalctl _comm=sudo and /var/log/auth.log sudo lines for privilege escalation on the bastion and the app hosts.
• Persistence: crontab -l for every user plus /etc/cron.*, systemctl list-unit-files --state=enabled diffed against a known-good baseline, ~/.ssh/authorized_keys mtimes and any command= forced-command entries.
• Lateral movement: the bastion's ~/.ssh/known_hosts and shell history, plus ss -tnp for unexpected outbound connections.
• New or altered accounts: awk -F: '$3>=1000' /etc/passwd, and getent group sudo.

State the limitation explicitly: with four weeks of local logs you can prove recent activity but not rule out older access - that gap is itself a finding.

1. Replace static authorized_keys with short-lived credentials - an SSH certificate authority or a federated/SSO-gated access path. The root problem is that keys never expire, so offboarding requires touching twelve hosts and fails silently when it does. Short-lived certificates make departure irrelevant and give an auditable issuance log. If a CA is too large a first step, at minimum put authorized_keys under configuration management so drift is detected.
2. Centralize authentication logs off-host. Local-only logs mean an attacker with root can erase the evidence, and four weeks of retention is too short to investigate a slow intrusion. Shipping auth.log and sudo events to a separate, append-only destination is cheap and changes the incident-response ceiling permanently.
3. Remove NOPASSWD: ALL for %devs, and drop agent forwarding. Forwarding lets anyone with bastion access borrow a developer's key to reach all twelve hosts; ProxyJump gives the same reach without exposing the agent socket. Scope sudo to specific commands, keep one audited break-glass account with a password and alerting.

These beat more IP blocking or a port change because they remove the failure modes that actually appeared: permanent credentials, invisible logs, and unbounded privilege.