SSH Hardening After Credential Stuffing Without Locking Out Admins
An Ubuntu 22.04 jump host is internet-facing and used by 12 administrators. Eight use password plus PAM MFA; four use SSH keys. /etc/ssh/sshd_config has PermitRootLogin yes, PasswordAuthentication yes, MaxAuthTries 6, and no AllowGroups. Logs show roughly 50,000 failed root logins per day from 198.51.100.0/24 and 203.0.113.0/24. Three administrator accounts are currently locked by pam_faillock after mistyped passwords. sudoers contains %admins ALL=(ALL) NOPASSWD: ALL. One administrator copied the same private key onto three laptops, and a backup of that key is on a shared NFS home directory. You have 24 hours to reduce the attack surface, but you cannot lock out the on-call team or lose emergency access. Change control requires a tested rollback and a second session during any sshd change. The business will not approve disabling password authentication for everyone until key enrollment is complete.
1. Rank the risks and explain why setting PermitRootLogin no and PasswordAuthentication no tomorrow would be unsafe.
Read the worked answer
The highest risks are internet-wide password guessing against root and admins, NOPASSWD: ALL for the whole admins group, and one private key copied to three laptops and backed up on a shared NFS home. The shared key turns any one laptop or NFS compromise into access to every server that trusts it. The missing AllowGroups means every local account is exposed to SSH.
Disabling password authentication tomorrow would lock out the eight admins who rely on password plus PAM MFA before their keys are enrolled, and it could remove the on-call team's fallback. Setting PermitRootLogin no without a verified named sudo path and console break-glass can also strand you. pam_faillock has already locked three accounts, so a blind sshd reload during an incident can turn a partial lockout into a full one. The safe sequence is network restriction first, then named accounts and keys, then reduce root and password exposure, then tighten sudo.
2. Give a 24-hour rollout plan with concrete safeguards and rollback. Include how to handle the three locked accounts.
Read the worked answer
Keep two sessions open, one from out-of-band console. Back up /etc/ssh/sshd_config, /etc/pam.d/sshd, and sudoers; test a candidate with sshd -t -f. First restrict SSH at the network or security-group layer to the office, VPN, and on-call addresses, so brute-force traffic stops before sshd. Do not allow the attacking /24s. Create an ssh-admins group, add only the 12 named admins, and confirm at least two key-based admins can log in before changing AllowGroups.
Then set AllowGroups ssh-admins, MaxAuthTries 3, LoginGraceTime 30, and keep password authentication temporarily for the MFA users. Move root to prohibit-password or no only after named sudo works from console. For the locked accounts, verify identity, inspect faillock --user <name>, then faillock --user <name> --reset; do not disable faillock. Remove NOPASSWD from sudo after piloting with visudo -c, and require a password for sudo. Rotate the shared key off the NFS home, issue per-admin keys or SSH certificates, and revoke the old key. Rollback is restoring the saved configs and running systemctl reload ssh, with console access if sshd fails.
3. What detective controls and evidence would you add after the hardening change?
Read the worked answer
Centralize /var/log/auth.log, sudo logs, and auditd events, and alert on successful root logins, logins from new source networks, repeated faillock failures, and changes to authorized_keys or sudoers. Use audit watches on /etc/ssh, /etc/sudoers, and admin home .ssh directories; file integrity monitoring should catch the shared key reappearing. Review last, lastb, ss -tnp, and journalctl -u ssh during on-call handover, and keep a weekly report of key fingerprints per admin.
Do not rely only on fail2ban: it reacts to failed attempts, can be evaded, and can lock out legitimate admins behind a shared NAT. Keep the network ACL as the primary control. Record the change ticket, old and new sshd -T output, test login evidence, and the key rotation inventory so you can prove the shared key was removed and each remaining key belongs to one person.