Malware Analysis and Reverse Engineering Fundamentals
Malware Analysis and Reverse Engineering Fundamentals
This domain tests whether you can safely characterise a sample far enough to support detection and response, and whether you know which questions static and dynamic methods can each answer. It is fundamentals, not expert reversing: expect file format, packing, behavioural artifacts and lab safety.
Static analysis, in order
Hash the sample (MD5 and SHA-256), determine the true file type by magic bytes rather than extension, extract printable strings with encoding awareness (ASCII and UTF-16), and parse PE headers: machine type, compile timestamp (forgeable, so corroborate), sections and their entropy, imports, exports, resources and the rich header. A tiny import table with only LoadLibrary and GetProcAddress, or a section with entropy near 7.9, suggests packing or runtime resolution of APIs. Identify known packers such as UPX before deeper work, and unpack a copy, never the evidence original.
Dynamic and behavioural analysis
Run in an isolated lab with no route to production, snapshots for rollback, and simulated internet services for DNS and HTTP so the sample reveals callbacks. Observe process creation and parentage, file and registry writes, service or scheduled-task creation, injected threads, and network traffic captured in the lab. Behavioural results are what feed EDR and network detections.
Anti-analysis and code-level work
Watch for virtual-machine detection, timing or sleep checks, debugger and sandbox artifact checks, obfuscated scripts, encrypted payloads with a decryption stub, and API hashing. Only after behavioural triage should you move to disassembly and debugging with tools such as Ghidra or x64dbg, and even then the goal is usually configuration extraction - C2 addresses, keys, mutexes - not full source reconstruction.
Worked example: a Windows executable has a .text section with entropy 7.8 and imports only LoadLibrary and GetProcAddress, so you treat it as packed. In the sandbox it drops a copy into a user AppData path, creates a Run key, and connects to an IP on port 8443 every 45 seconds. Memory strings reveal the User-Agent and URI pattern. You submit the hash, write a detection on the combination of the Run key path and the beacon URI, and record the C2 for blocking.
Traps: executing a sample on the analysis workstation; trusting the compile timestamp as evidence; equating packed with malicious; treating a low import count as benign; forgetting to capture network traffic so the C2 is never identified.