Buffer Overflow 32 Bit
Apr 15, 2025This is a short write-up I created for the Secure Software module, where we explored how to build secure applications and how common attacks actually work. I decided to share it here because I learned a lot by visualizing what happens on the stack. Our work received the highest grade, but I can’t guarantee that everything is 100% accurate. If you spot anything off, I’d love to hear from you!
What happens on the stack
The original C code is first translated into assembly language by the compiler, and then into machine code. To better understand what happens on the stack, we visualized the relevant registers and analyzed the key operations in the assembly code. The following registers are of particular importance:
- EBP – Base Pointer (points to the beginning of the current stack frame)
- ESP – Stack Pointer (points to the current top element of the stack)
- EIP – Instruction Pointer (points to the next instruction to be executed)
By examining the assembly instructions, we gain deeper insight into the memory operations taking place.
To understand the structure of the stack during function calls, we examine the so-called function prologue, which is automatically generated by the compiler at the beginning of each function. The purpose of this prologue is to save the old base pointer (EBP), establish a new stack frame, and allocate space for local variables if necessary. Typically, a function prologue consists of the following instructions:
push ebp ; save the old base pointer
mov ebp, esp ; start a new stack frame
sub esp, X ; allocate space for local variables
This process can be observed in the example of the function bof() in the disassembly of retlib, as shown below:
124d: f3 0f 1e fb endbr32
1251: 55 push %ebp
1252: 89 e5 mov %esp, %ebp
1254: 53 push %ebx
1255: 83 ec 34 sub $0x34, %esp
When a function returns, the stack is dismantled accordingly. The compiler typically uses the two instructions leave and ret for this purpose. The leave instruction essentially serves as a shorthand for the following operations:
mov esp, ebp ; restore the stack pointer to the current base pointer
pop ebp ; load the previous base pointer from the stack
The pop ebp instruction loads the top value of the stack into the ebp register and simultaneously increases the stack pointer (esp) by 4 bytes (esp += 4). This increment is implicit in the pop instruction.
The ret instruction then loads the return address (i.e., the address of the calling function) from the stack into the eip register and jumps to that address. This operation is equivalent to pop eip, which means eip = [esp], followed by esp += 4.
Together, leave and ret form the standard mechanism for cleanly exiting a function and returning control to the caller. These steps are clearly illustrated in Figure 8, which shows the regular stack behavior of a program.
In contrast, Figure 9 shows the manipulated stack behavior during a so-called return-to-libc attack. In this case, the stack is changed on purpose so that the ret instruction doesn’t jump back to the original return address, but instead directly to the system() function in the libc.
The important part is that system() starts with a typical function prologue: push ebp followed by mov ebp, esp. This prologue expects a specific stack structure, just like the one the compiler normally sets up during a regular function call. With the manipulated stack, we mimic this structure: right after the return address (now eip = system), we place the desired return value (exit()) and the argument for system(), which in our case is “/bin/sh”. Since system() expects the same stack layout as any regular C function, the attack works reliably, because the manipulated stack fakes both a valid prologue and proper argument passing. Figure 9 also shows how system() runs its usual prologue, but this time on a stack that has been fully crafted and controlled by the attacker.
Proof
The attached screenshot shows the successful exploit. We have highlighted the relevant areas in the hex dump that were specifically overwritten in the exploit. This makes it clear which values are included in the payload, where they are located in memory, and how they affect the program flow.
