Memory-Only Intrusion on a 400-Day Production Linux Host
A 32 GB RAM e-commerce web server running nginx and PHP-FPM shows a webshell that executes only from memory, with no matching file on disk. The host has 400 days of uptime, kernel 5.4, no EDR agent, and remote syslog shipping to a SIEM that normalises timestamps to receive time. Egress filtering is weak; outbound TLS to an unfamiliar address has been seen in proxy logs. The business processes payments and estimates downtime at a significant per-minute cost, but the PCI scope requires evidence-grade handling. The on-call engineer wants to reboot immediately to kill the webshell and then image the disk. A disk snapshot is available through the storage array. The host clock appears 40 seconds fast relative to the SIEM.
1. Memory or disk first? Justify the decision under the production constraint and state what you document.
Read the worked answer
Memory first. The disk is comparatively stable and can be snapshotted at the storage layer, while process memory holds the injected code, the C2 configuration, keys and the parent process chain that explains how the webshell was launched. A reboot destroys all of it, and the engineer's plan guarantees the most important evidence is gone before collection begins.
Sequence: record date -u, uptime, dmesg -T and kernel version; capture volatile process and network state; acquire physical memory with a tool appropriate to the kernel; hash the image on the acquisition host, not on the suspect; then take the array snapshot and image the disk from it. Do not reboot until both are complete and verified.
Containment runs in parallel and does not require a reboot: block the observed C2 address and destination port at the firewall, preserve existing connections in the capture, and disable the vulnerable ingress path at the load balancer. Document the acquisition tool, version, invocation, start and end times, memory size, and any errors. Note that the kernel was 5.4 and that whichever acquisition method was used altered the running system to some degree; that fact belongs in the report, not in a footnote. Also record who authorised the continued operation of a compromised payment host, because that decision will be scrutinised.
2. If memory acquisition requires loading a kernel module, is the resulting image still defensible?
Read the worked answer
Yes, provided the alteration is disclosed and bounded. Loading a module is a known, documented change to kernel state; it does not invalidate memory contents wholesale, but it can allocate buffers and overwrite small regions. The defensibility comes from the record, not from pretending nothing changed.
Document: the module's source and compiled binary hash, the exact kernel build it was compiled against, the insertion timestamp, resulting dmesg output, the module parameters including output path and format, and any pre- and post-acquisition system state captures. Prefer a module-free acquisition path if the kernel exposes a supported interface, and state why it was or was not used. If a module was necessary, acquire in one pass, verify the output file size against installed RAM, and hash immediately. Where possible, capture a second, smaller region such as the suspicious process's memory map through a different mechanism to corroborate.
In the report, describe the method as a documented intervention with a bounded footprint, and give the defence the information needed to test that claim. What would be indefensible is silently loading a module and presenting the image as an untouched snapshot. Also keep the acquisition host clean: hash on a trusted workstation, not on the compromised server.
3. How do you handle the 400-day uptime and the 40-second clock offset when correlating with the SIEM?
Read the worked answer
Treat both as uncertainty to be quantified rather than obstacles to be ignored. A host up for 400 days has accumulated clock drift, and the SIEM stores receive time rather than event time, so a naive join will misorder events by seconds to minutes.
Establish the offset empirically: compare date -u on the host with an authoritative source at the moment of acquisition, and check the NTP or chrony offset and last synchronisation. Record the offset with its sign and the time it was measured. Then convert host timestamps to a common timeline and state the residual uncertainty, for example 'host clock +40s, measured at 14:02 UTC, drift rate unknown'. Use monotonic boot-relative times from dmesg and process start times to order events within the host, and use the SIEM's receive time only to bound when an event reached the collector.
For the report, present the timeline with both raw and normalised times, and flag any correlation that depends on sub-minute ordering, such as whether the webshell spawned before or after the outbound TLS connection. Do not silently adjust timestamps without recording the transformation.