OSEE common mistakes
OSEE · Understand the traps and how to avoid them.
Treating a crash as proof of exploitability: reporting the EIP overwrite or access violation as the finding and then attempting to build a full exploit from it, without first checking which bytes actually survive into the target buffer and in what order.
Debuggers stop at the faulting instruction, so the overwritten instruction pointer looks like a clean win. Candidates skip the cyclic-pattern offset and bad-character step because it feels like a formality, then spend the remaining time fighting a payload that is being truncated or reordered by the input handler.
What to do instead
Before writing any shellcode, send a four-byte cyclic marker, compute the exact offset to the return address, then send an ascending byte sequence 0x01 through 0xFF and diff the bytes that appear in memory against what was sent. Record every rejected byte (null, CR, LF, space, and any application-specific terminator) and exclude them from the encoder alphabet. Only claim control of execution after the byte map is verified.
Hardcoding an address that is only valid in the local debugger session and presenting the exploit as reliable, ignoring ASLR, stack layout shifts, and environment-dependent heap placement.
A single successful local run against a non-rebased binary creates false confidence. Candidates test once, see the shell, and move on, not realizing that the same exploit fails after a reboot, under a different working directory, or when the process is launched by the service wrapper rather than by hand.
What to do instead
Run the exploit at least ten times across process restarts and confirm the target address is stable, or explicitly document the address as non-ASLR-dependent and justify why (for example, a non-PIE binary with a fixed .bss pointer). Where ASLR applies, use a technique that does not need a fixed address, such as a partial overwrite, a stack pivot into a register-held pointer, or a ret2reg/ret2esp path. State the assumptions in the report.
Building a ROP chain from gadgets that were grepped out of the binary without verifying that each gadget's side effects and register state match what the previous gadget leaves behind, producing a chain that crashes at the third or fourth link.
Gadget search tools return matches by byte pattern, and candidates chain them by eye, assuming each pop; ret is clean. They miss that a gadget also clobbers a register used later, that a leave gadget pivots the stack unexpectedly, or that the gadget address contains a byte the input filter rejects.
What to do instead
For each gadget, disassemble the full instruction sequence rather than just the matched bytes, note every register and flag it writes, and confirm no gadget address contains a forbidden byte. Single-step the chain in the debugger link by link, checking the stack pointer and the target register after each return before adding the next gadget.
Assuming a heap overflow gives direct control of a function pointer, then spending hours trying to overwrite a vtable pointer at a fixed offset without mapping chunk metadata, allocation order, and the allocator's consolidation behavior.
Heap exploitation requires understanding what the allocator does between the overflow and the free/reallocation that triggers the corruption. Candidates jump to the classic unlink or vtable overwrite without first tracing which chunks are adjacent, which are freed, and how the target build's allocator (glibc version, hardened allocator, or custom pool) differs from the tutorial they studied.
What to do instead
Instrument allocation and free calls (breakpoints on malloc/free or the target's allocator wrappers) and draw the heap layout: chunk sizes, user-data addresses, and the order of allocation. Identify the exact primitive the overflow gives (forward overwrite of the next chunk header, overwrite of an in-use object's field) and choose a technique that matches that primitive and the observed allocator version. Verify the corrupted state after each step in the debugger before triggering the free.
Writing shellcode that calls execve directly without accounting for the calling convention, stack alignment, or the fact that the payload must be position-independent and self-contained after a jump into arbitrary memory.
Candidates assemble a short snippet in a normal compiled context, extract the bytes, and paste them into the buffer, not realizing the snippet relies on an absolute address, a PLT entry, or a stack alignment that no longer holds at the injection point. The result is a crash inside the payload that looks like a bad offset.
What to do instead
Assemble the payload as a standalone blob, extract the raw bytes, disassemble them back to confirm no relocations or absolute references remain, and test the blob in isolation by jumping to it from a debugger with a known register state. Confirm stack alignment at the entry point and set up any registers the payload depends on before the jump. If the target architecture requires it, prepend a short decoder stub and verify the decoder survives the byte filter.
Skipping the mitigation inventory: not checking NX, stack canaries, PIE, RELRO, ASLR, and the OS-level hardening before choosing an exploitation strategy, then discovering mid-exploit that the intended technique is blocked.
Candidates pattern-match the target to a familiar tutorial scenario and start writing the exploit immediately. The mitigation check feels like overhead, and its results often invalidate the plan that was already half-built, so it gets deferred until the exploit fails.
What to do instead
First action on any target: enumerate mitigations (checksec-style inspection of the binary, plus a runtime check of ASLR and any sandbox or seccomp policy). Write the results down, then select a technique that is compatible with all of them. If a technique is blocked, pivot before writing payload code, not after.
Fuzzing or crash-triaging without a reproducible harness: feeding inputs by hand, not recording the exact input that produced each crash, and not minimizing the crashing input before analysis.
Manual testing produces crashes that cannot be replayed exactly, especially when the input contains timestamps, session tokens, or length fields that must be consistent. Candidates then analyze a crash they cannot reliably reproduce, and every debugger session starts from a slightly different state.
What to do instead
Build a scripted harness that takes an input file and delivers it to the target deterministically, log every input that crashes, and minimize each crash to the smallest byte sequence that still faults. Confirm the minimized input reproduces the same fault address and instruction across three consecutive runs before starting root-cause analysis.
Confusing the offset to the saved return address with the offset to the buffer start or to the saved frame pointer, then placing the payload at the wrong position and concluding the offset calculation was wrong.
The cyclic pattern offset points at the return address, but the payload must be laid out relative to the buffer, and the distance between them depends on local variables, compiler padding, and the frame layout. Candidates compute one number and use it for both purposes.
What to do instead
After finding the return-address offset, inspect the stack frame in the debugger to locate the buffer start and compute the buffer-to-return-address distance separately. Lay out the payload as padding to the return address, then the return address, then any stack-based payload, and verify the buffer start address matches where the payload actually lands.
Ignoring the difference between the exploit working under the debugger and working standalone, and not testing in the same launch context the grader or service uses.
Debuggers change the environment: they may disable ASLR, alter stack and heap addresses, add environment variables that shift the stack, and change the working directory. An exploit tuned in the debugger frequently fails when launched by the service, and candidates blame the payload instead of the environment.
What to do instead
After the exploit works under the debugger, test it against the process launched exactly as the service launches it (same wrapper, same user, same working directory, same environment). If addresses shift, recompute them for the real launch context or use an address-independent technique, and keep a non-debugger test in the loop from the start.
Reporting an exploit as complete when it only achieves a crash or a controlled instruction pointer, without demonstrating the intended impact such as code execution, a shell, or a read of protected memory, and without documenting the reliability.
The exam rewards demonstrated impact, but candidates stop at the first sign of control because further work is hard and time is short. They also omit reliability data, so a one-in-five success rate is presented as a working exploit.
What to do instead
Define the success criterion before starting (for example, a shell that returns a known command output, or a controlled read of a specified address) and do not stop until that criterion is met. Measure reliability over at least twenty runs, report the success rate and failure mode, and state the preconditions (mitigations, target version, launch context) under which the exploit is valid.