OSED common mistakes
OSED · Understand the traps and how to avoid them.
When hunting for a stack overflow, the candidate only looks at obvious `strcpy`/`gets` calls and stops there, missing overflows in `sprintf` with `%s`, `memcpy` with an attacker-controlled length, or a loop that copies into a fixed buffer.
Candidates learn overflow hunting from a handful of textbook functions and treat the vulnerability as a pattern-match problem. They don't trace the data flow from a network or file read to the copy, so they miss indirect copies where the length field is parsed from the same input.
What to do instead
Build a habit of tainting input backward: identify the recv/ReadFile/fread sink, then follow the buffer forward through every function that receives it. Explicitly inspect every call that takes both a destination buffer and a length or unbounded source, including memcpy, sprintf, vsprintf, lstrcpy, and hand-rolled copy loops. Note the buffer size next to each sink before deciding it is safe.
The candidate calculates the EIP offset by eyeballing the crash or trusting a single cyclic pattern run, then uses an offset that is wrong by a few bytes and spends the rest of the exam debugging a payload that never lands.
A crash shows overwritten registers that may be attacker-influenced in the middle of the copy rather than at the return address. Candidates confuse a controlled register with the return address, or they test in a debugger where the stack layout differs slightly from the unmanaged run.
What to do instead
Use a mona-style cyclic pattern of distinct length, find the exact value that lands in EIP, and confirm it with pattern_offset. Verify the offset by sending a payload of that exact length plus a marker such as 0x42424242 and watching EIP in the debugger. Re-verify in the same execution context (same debugger attach state, same input channel) you will use for the final exploit.
The exploit works in the debugger but fails when run standalone, and the candidate keeps tweaking the shellcode instead of examining the environment difference.
Debuggers change the stack: they push environment variables, alter the PEB/command line, and shift the initial ESP, so a hardcoded stack return address or a jmp-esp landing point moves. Candidates don't realize the debugger itself relocates the very thing their exploit depends on.
What to do instead
After getting control in the debugger, run the target outside the debugger and attach after the crash, or use a debugger setting that reduces environment differences. Prefer position-independent landing techniques (a jmp esp/call esp gadget, a register-held pointer to the buffer) over hardcoded stack addresses. When a hardcoded address is unavoidable, confirm it in the standalone process, not only under the debugger.
The candidate writes shellcode that contains a null byte or a newline because the payload travels through `strcpy` or a line-based protocol, and the exploit truncates silently.
Assembly generated by a compiler or written casually uses instructions with zero operands or immediate values that encode 0x00, and candidates test the shellcode by running it directly rather than through the same transport the target uses.
What to do instead
Encode the payload with an encoder matched to the bad-character set (for example an alphanumeric or XOR encoder), or rewrite the offending instructions (use xor eax,eax alternatives, push/pop tricks, or sub/add to build constants without zero bytes). Determine the bad-character set empirically by sending all 256 byte values through the vulnerable path and observing which are stripped or terminate the copy.
The candidate's custom shellcode crashes on return because it assumes a clean register state or a writable location, and it does not restore the stack or handle the fact that it runs in the middle of a function epilogue.
Shellcode is tested in isolation in a harness that zeroes registers and gives it a fresh stack. In the real target, ESP points into the overwritten buffer, registers hold arbitrary values, and the shellcode may overwrite its own instructions if it stores data on the stack.
What to do instead
Write shellcode that is self-contained: save and restore any register it clobbers, avoid assuming ESP is valid until it sets up its own frame, and keep writable scratch space away from the code bytes (or make the code position-independent and copy data below ESP). Test the shellcode by injecting it through the actual vulnerable path, not only in a standalone runner.
The candidate builds a DEP bypass but returns into a `VirtualProtect` call with arguments that are wrong (wrong size, wrong protection constant, or a return address that is not executable), and the process dies before the shellcode runs.
ROP chain construction is error-prone: the calling convention requires the arguments and a return address on the stack in the right order, and PAGE_EXECUTE_READWRITE is 0x40, not the more familiar read/write constants. Candidates also forget that the region must cover the whole shellcode and that the return address after VirtualProtect must point at the shellcode.
What to do instead
Lay the chain out explicitly on paper before building it: address of VirtualProtect, return address = shellcode start, then lpAddress, dwSize, flNewProtect = 0x40, lpflOldProtect (a writable address). Verify each gadget's side effects (does it pop extra bytes? does it clobber a register you need?) and single-step the chain in the debugger to confirm the call returns into the shellcode.
The candidate treats ASLR as defeated because one run succeeded, using a hardcoded module base or stack address that happened to be correct once.
ASLR randomizes per boot or per process depending on the OS and configuration, and a single lucky run looks like success. Candidates don't check whether the target is actually ASLR-enabled or whether the module they rely on is a non-ASLR module.
What to do instead
Confirm the mitigation state first (check module characteristics such as DYNAMICBASE, and observe base addresses across several runs). If a non-ASLR module exists, use its gadgets deliberately and document why. Otherwise build an information leak into the exploit (see the format-string primitive) to recover a base at runtime, or use a partial-overwrite/brute-force strategy only where the entropy makes it reliable.
The candidate finds a format string bug and immediately tries to write with `%n`, crashing the process because the argument index or the target address placement is wrong.
Write primitives depend on knowing exactly which stack slot holds the attacker's buffer, and that index changes with the length of the format string and the position of the address bytes. Candidates skip the read step and guess the index.
What to do instead
First build a read primitive: send %p.%p.%p... or %x sequences to map stack slots, and use %s with a crafted pointer to read memory. Only after the index is confirmed should you attempt %n, and even then start with a harmless target (a writable scratch address) to validate the write before aiming at a function pointer or GOT entry.
The candidate uses `%n` to write a value but ignores the byte count already consumed by the format string, so the written value is offset by the length of the prefix and the target pointer ends up wrong.
%n writes the total number of characters printed so far, including the address bytes and any padding. Candidates compute the desired value in isolation and forget the prefix, or they place the target address at the start of the format string where its bytes count toward the total.
What to do instead
Compute the write as (desired value minus bytes already printed), pad with %<n>c to reach the target, and account for the address bytes themselves. Prefer splitting a 4-byte write into two 2-byte writes with %hn for reliability, place target addresses at the end of the format string, and verify the resulting value in the debugger before relying on it.
The candidate's exploit depends on a fixed offset or address that changes when the target is run with a different working directory, environment, or input length, and the candidate never re-tests after changing the payload.
Stack addresses shift with environment size and argument length, and buffer contents shift when the payload length changes. Candidates iterate on the payload without re-measuring the offset, so a previously correct address silently becomes stale.
What to do instead
After any change to payload length or environment, re-measure the offset and re-confirm the landing address. Keep the exploit parameterized (offset and address as variables) so changes are one-line edits, and run a final end-to-end test in the exact environment the exploit will be judged in, including the same launch method.