02:14 on the UPI Middleware Server: Ordering Containment Without Destroying Evidence

1. Isolate, do not switch off. Remove the host from the network at the security-group/switch level (deny all traffic except a forensic jump host). This kills command-and-control and the exfiltration channel while keeping memory and running processes intact. Powering the server off destroys volatile evidence and may trigger a destructive payload; it also drops the transactions in flight on that node.

2. Preserve before you delete. Capture a memory image, the process tree, live connections, cron/systemd entries and the sudoers modification, and export the SSH/auth trail and SIEM records. Logs are off-host and therefore safe, but the host-side artefacts are not - and they are the only attribution evidence for who created svc_upd8. Deleting the rogue account now would destroy exactly what is needed later.

3. Block at the perimeter. Deny the three source IPs and the egress destination, and apply a virtual patch at the WAF for the deserialization path so that other nodes running the same library are protected while patching proceeds.

4. Eradicate and recover. Disable svc_upd8, revoke every session, token and API key issued to the host, rotate the shared admin credential and all secrets on the host, then rebuild from a known-good image rather than cleaning in place. Notify the CISO/compliance at once so the reporting clock (six hours from noticing, under the CERT-In directions) is met.

A password reset closes one door and leaves three open.

Persistence. The attacker held sudo and created a second account. Standard persistence mechanisms - an SSH key in authorized_keys, a cron job or systemd unit, a modified PAM or sshd binary - survive a password change. The rogue account may already have its own key, in which case the shared password is irrelevant to the attacker.

Credential sprawl. The password was known to three engineers and is likely embedded in scripts, configuration files, monitoring probes and the DR copy. Resetting it in one place leaves working copies elsewhere; the honest assumption is that every secret stored on that host is now compromised.

Valid sessions and data loss. Existing sessions and tokens remain valid until revoked, and 1.2 GB has already left the network. That traffic turns the incident into a possible data-breach assessment with its own notification obligations; a password reset does nothing about it.

Correct sequence: contain, image, disable the rogue account, revoke sessions/tokens, rotate every secret, then rebuild. The password reset belongs inside that sequence, not at the top of it.

Not immediately, and not automatically. Two concrete objections from the facts.

The DR copy is six hours behind. Failing over now would take the service back to a state missing up to six hours of UPI transactions, creating a reconciliation and customer-harm problem that may be larger than the availability gain. The switch's own journal, not the middleware host, should be the reconciliation source.

The DR likely shares the weakness. If it runs the same unpatched library and the same shared admin credential, failover simply relocates the compromise and doubles the incident scope.

What I would do instead: keep the service on the remaining clean nodes behind the load balancer and drain the affected node (40 tps is well within a multi-node pool); verify the DR image's patch level and credentials before any switch; and if failover is unavoidable, do it only to a verified-clean DR after rotation and patching, with a written reconciliation plan.

Who decides: the availability-versus-containment trade-off is an incident-manager decision taken jointly with the switch owner and compliance, documented with timestamps - not a call the SOC makes alone at 02:40.