The deploy Account Logged In at 03:14

Treat this as a probable root-level compromise. deploy has NOPASSWD: ALL, so a login to that account is equivalent to root, and the unfamiliar key plus cleared history make persistence likely. Do not simply disable the account and move on, and do not wait until after the 16:00 release.

Sequence:

1. Preserve evidence before changing anything: copy /var/log/auth.log and rotated versions, last, lastlog, journalctl _COMM=sudo, the current authorized_keys for all accounts, sudoers and /etc/sudoers.d, running processes (ps auxf), network connections (ss -tupna), and cron/systemd timers. Hash the copies.
2. Contain: remove the unknown key, revoke the deploy key at the CI secret store, kill active sessions for deploy (pkill -KILL -u deploy), and block the three source addresses at the edge. Disable password authentication immediately if the platform supports it.
3. Keep CI running safely: issue a new deploy key or short-lived certificate, restrict it to the pipeline runner's egress address, and re-run queued pipelines only after the account is re-secured. If that cannot be done before 16:00, the release slips; a compromised root-equivalent account is not an acceptable release risk.
4. Escalate per incident policy and start the scope investigation in parallel.

Scoping means answering three questions: what did the attacker touch, did they gain persistence, and did they reach other systems from this host. Check all accounts, not just deploy: authorized_keys, ~/.ssh/config, shell profiles, sudo -l for every user, and /etc/sudoers.d. Look for persistence in cron (/etc/cron.*, user crontabs), systemd units and timers, at, ld.so.preload, and init scripts. Look for changes with find / -xdev -mtime -3 -type f and package verification (dpkg -V or rpm -Va); unexpected SUID/SGID files are a strong signal. Review authentication and privilege logs across the fleet for the three source IPs and for the new key's fingerprint. Check outbound connections and DNS for unusual destinations.

Rebuild if you find any of: a modified system binary or package, an unknown SUID binary, a persistence mechanism you cannot fully explain, evidence that sudo was used from the deploy session, or outbound traffic to an unknown host. Because deploy is root-equivalent, the default assumption is rebuild unless you can show the session never escalated and left no artifacts. Cleaning a root-level compromise in place is usually not defensible for a customer-facing host.

The goal is to remove internet-exposed password authentication and root-equivalent static keys, and to make every privileged action attributable.

• SSH: PermitRootLogin no, PasswordAuthentication no, KbdInteractiveAuthentication no, AllowUsers/AllowGroups limited to named admins and the deploy identity, and a firewall or security-group allowlist rather than 0.0.0.0/0. Put administrative SSH behind the VPN or a bastion, and rate-limit with fail2ban or an edge control.
• CI identity: replace the shared deploy key with short-lived certificates from an SSH CA scoped per pipeline, or use a dedicated deployment agent with narrowly scoped sudo commands instead of NOPASSWD: ALL. Log every sudo invocation.
• MFA for interactive administrators, and a break-glass account with a sealed credential and alerting.
• Ship auth, sudo and audit logs to the SIEM and alert on successful logins from new ASNs or countries.

Trade-offs: a bastion or CA is another dependency and a potential single point of failure, so it needs its own HA and break-glass path. Short-lived certificates require CI to talk to the CA, so pipeline failures become possible. Allowlists complicate remote work. Accept these in exchange for removing a standing root credential from the internet. Document the break-glass procedure and test it quarterly.