3761
Security Core Lecture
05 Software Security II – Stack
and Calling Conventions
Prof. Dr. Thorsten Holz | 07.11.2023
3761
Recap: QUIZTIME: Conditional Jump #1 (ex12.asm)
§ When is ebx set to 1?
Assembly
cmp eax, 10
jge a1
jmp a2
a1:
mov ebx, 1
a2:
nop
2
3761
Recap: QUIZTIME: Conditional Jump #1 (ex12.asm)
§ When is ebx set to 1?
Assembly
cmp eax, 10
jge a1
jmp a2
a1:
mov ebx, 1
a2:
nop
jge = jump greater or equal
Answer B is correct
3
3761
Recap: QUIZTIME: Conditional Jump #2 (ex12.asm)
§ What is the equivalent C code for this snippet?
(Assume rbx is variable b and rax is variable a; both are
signed 64-bit integers)
Assembly
mov rcx, 1
shl rcx, 63
test rax, rcx
jz b1
mov rbx, 1
b1:
nop
4
3761
Recap: QUIZTIME: Conditional Jump #2 (ex12.asm)
§ What is the equivalent C code for this snippet?
(Assume rbx is variable b and rax is variable a; both are
signed 64-bit integers)
Assembly
mov rcx, 1
shl rcx, 63
test rax, rcx
jz b1
mov rbx, 1
b1:
nop
test: bit-wise AND of two values
Answer B is correct 5
3761
Recap: QUIZTIME: Conditional Jump #3 (ex12.asm)
§ Can this code be simplified (still using the test
instruction)?
Assembly
mov rcx, 1
shl rcx, 63
test rax, rcx
jz b1
mov rbx, 1
b1:
nop
6
3761
Recap: QUIZTIME: Conditional Jump #3 (ex12.asm)
§ Can this code be simplified (still using the test
instruction)?
Assembly
mov rcx, 1
shl rcx, 63
test rax, rcx
jz b1
mov rbx, 1
b1:
nop
We could rewrite the code to not use JZ
but Jump if (not) negative
7
3761
This Lecture
§ Last lectures
- Control flow operations
- (un)conditional jumps
- Loops and similar code constructs
§ This lecture
- Stack and push/pop instructions
- Calling conventions for x64 / x86
8
3761
Functions and the Stack
3761
Stack
§ Stack: Memory in LIFO layout
- Local variables, function arguments, return values, ...
§ Two basic assembly primitives:
- push: Add data <op> to top of stack (and reduce rsp by number of bytes)
- pop: Remove data from top of stack into register <op> (and increase rsp)
§ rsp points to top of stack (i.e., to last element on stack)
10
3761
Stack II
§ push/pop examples
Assembly
SECTION .data
myvar dq 0xcafe;
SECTION .text
lea rax, [myvar] ; rax contains address of myvar
push 0xcabe ; push constant 0x000000000000CABE on stack
push rax ; push value in rax on stack
pop rax ; pop into rax (rax := rax)
pop qword [myvar] ; pop into variable (myvar := 0xCABE)
; default push/pop size is 8B in x64, but 2B/4B can be specified
push ax ; push value in ax on stack (RSP –= 2)
push word 0xcabe ; push constant 0xCABE on stack (RSP -= 2)
11
3761
Challenges for Calling Functions
§ One functions calls another and passes arguments to the callee - how is the
communication handled?
Code
int foo (int var1, char *var2) {
…
tmp = bar (23, 42);
…
}
12
3761
Function Calls
§ Invoke function: call fun
- call implicitly performs these two operations:
1. Store return address on stack (next instr.):
push rip (implicitly rsp -= 8 in x64)
2. Change control flow to callee: jmp fun
§ Return to caller: ret (<op>)
- Return value stored in rax
- ret instruction implicitly performs these operations:
1. Optional: remove <op> bytes from stack (cleanup; see later)
2. Restore return address from stack: pop saved rip (implicitly rsp += 8)
3. Change control flow back to caller: jmp rip
13
3761
Linux Calling Convention in x64
§ First 6 parameters passed via registers: rdi, rsi, rdx, rcx, r8, r9
§ Additional parameters are passed on the stack
§ Return value in rax
§ Who is responsible for saving registers? Fixed in calling convention
§ Non-volatile, callee-saved registers: rbp, rbx, r12-r15, rsp
- The called function (callee) has to save/restore registers before/after
modification
- Caller can rely on the fact that the callee does not alter the registers
§ Volatile, caller-saved registers: all other registers
- Callee may alter registers at free will
- If still required, calling function needs to store registers before call and
restore after call
14
3761
Example
https://godbolt.org/z/9WSBds
15
3761
Memory Layout: Stack in x64
§ Call stack (stack frame) is subroutine’s perspective on stack, includes the
following info
- arguments (parameter values) passed to routine (if any);
- return address back to the routine’s caller
- space for local variables (if any).
- Stack frame is 16B aligned in x64, 8B aligned in x86
§ Stack grows to lower addresses
- rsp: top of stack (stack pointer)
- rbp: pointer into current frame (base pointer)
16
3761
Memory Layout: Stack in x64 II
§ Contents of specific addresses
- rbp: caller’s RBP
- rbp+8: return address
- rbp+16: 7th parameter
- Negative displacement to rbp: access to local
variable
- below rsp: ephemeral scratch space
§ Stack’s base address
- Loader determines memory location of data section
of program (see PE/ELF header)
- Usually the stack is high up, but not adjacent to
0x07FF...FF
17
3761
QUIZTIME: Stack
§ Assembly Challenge
- Does push change rsp?
18
3761
QUIZTIME: Stack
§ Assembly Challenge
- Does push change rsp?
Answer B is correct
19
3761
QUIZTIME: Function Parameters
§ Assembly Challenge
- How many parameters can be
passed to a function in x64?
20
3761
QUIZTIME: Function Parameters
§ Assembly Challenge
- How many parameters can be
passed to a function in x64?
Slide 14:
First 6 parameters passed via registers:
rdi, rsi, rdx, rcx, r8, r9
Additional parameters are passed on
the stack
Answer C is correct
21
3761
Memory Layout: Stack in x64 (Example)
§ Stack layout after function prologue of summing()
C Code
uint64_t summing(uint64_t a, uint64_t b,
uint64_t c, uint64_t d,
uint64_t e, uint64_t f,
uint64_t g, uint64_t h) {
uint64_t x = a + b + c;
uint64_t y = d + e + f;
uint64_t z = g + h;
uint64_t s = x + y + z;
return s;
}
int main(int argc, char* argv[]) {
summing(1, 2, 3, 4, 5, 6, 7, 8);
nxt return 0; RDI=1, RSI=2, RDX=3,
} RCX=4, R8=5, R9=6
22
3761
Memory Layout: Setting Up the Stack
§ Function prologue
1. Save old rbp: push rbp
2. Establish frame: mov rbp, rsp
3. Reserve stack space for local vars: sub rsp, <size_local_vars>
§ Function epilogue
1. Restore old frame (alternative: leave)
• mov rsp, rbp
• pop rbp
2. Return to caller (ret)
23
3761
Memory Layout: Stack in x64 (Example)
C Code
void funA(int a, int b) {
uint64_t vA, vB;
vA = vA + vB;
}
void funB(int x, int y) {
uint64_t v1,v2,v3,v4;
funA(v1,v4+v3);
nB: v2=v1+v3+v4;
}
int main(...) {
funB(1, 2);
nM: return 0;
}
24
3761
QUIZTIME: Function Calls
§ Assembly Challenge
- Suppose fun_A calls fun_B
- How does the stack frame (i.e., the
registers rbp and rsp) of fun_A look like
after fun_B has returned to fun_A?
- Assume x64 calling convention
25
3761
Memory Layout: Frame Pointer Omission (FPO)
§ rbp is not mandatory
- Addressing can be done relative to rsp
- rsp-relative addressing is fairly common in x64
§ Implementation
- Flag –fomit-frame-pointer in gcc (enabled in O1-3) or /Oy in VC does
this
- All stack addresses are relative to rsp
- rbp can be used as a general-purpose register
- Function prologue omits mov rbp, rsp
- Function epilogue omits pop rbp
26
3761
Calling Conventions in x86
§ x86 (32-bit) has many different calling conventions
§ Vary in how to pass parameters
- Parameters in registers vs. parameters in stack
- Order of parameters in stack
- Set of registers of parameters
- Set of registers that need to be saved by callee
27
3761
Calling Conventions in x86 (Example)
§ cdecl example
C Code Assembly (int main(), caller)
int addall(int a, int b, int c) { public main
int x;
x += a + b + b + c + c + c; ...
return x; push ebp ; store ebp
} mov ebp, esp ; set ebp
push ecx ; store register ecx
int main() { push 3 ; push 3x4 = 0x0C bytes
addall(1, 2, 3); push 2
return 0; push 1
} call addall ; call addall()
add esp, 0x0C. ; clean up stack
pop ecx ; restore ecx
mov eax, 0. ; return value
...
28
3761
Memory Layout: Stack in x86 (Example)
Assembly (int main(), caller)
§ Stack layout (cdecl)
; in main
01: push 0x3
02: call recurse
C Code 03: add esp,0x4
uint32_t recurse(uint32_t a) {
; in recurse
if (a <= 1) {
04: push ebp
return 1;
05: mov ebp,esp
} else { 06: push ebx
return a+recurse(a-1); 07: mov ebx,[ebp+0x8]
} 08: cmp ebx,0x1
fin: 09: mov eax,0x1
} 10: jbe end_recurse
11: lea eax,[ebx-0x1]
12: push eax
int main(...) {
13: call recurse
recurse(3);
14: add esp,0x4
fin: 15: add eax,ebx
} 16: end_recurse:
17: pop ebx
18: leave
19: ret
29
3761
Memory Layout: Stack in x86 (Example) II
30
3761
Stack Setup in x86 (32b_ex02.asm)
§ push vs. sub and mov
- push X adds X to stack
§ So does: sub esp, 4 and mov [esp], X
Assembly Code
; cdecl: sum_three_numbers(1, 2, 3)
push 3
push 2
push 1
call sum_three_numbers
add esp, 0x0c ; 3*4 = 12 bytes
; alternative using sub esp and moves instead of push
sub esp, 0x0c
mov [esp+0], dword 1
mov [esp+4], dword 2
mov [esp+8], dword 3
call sum_three_numbers
add esp, 0x0c ; 3*4 = 12 bytes 31
3761
System Calls using the Linux x64 ABI
§ Calling convention like normal function calls
- 4th parameter is different (r10 instead of rcx), thus:
rdi, rsi, rdx, r10, r8, r9
- System call number placed in rax
- Return value in rax
§ syscall instruction initiates system call
- Usually wrapped by system libraries (e.g., libc)
- Could be used, in principle, by program itself (or by shellcode)
32
3761
System Calls using the Linux x64 ABI II
§ Example system call
Assembly Code
; sys_exit(42)
mov rax,60 ; system call number (sys_exit)
mov rdi,42 ; system call return value
Syscall
; sys_write(stdout, buf, buflen)
mov rax,1 ; system call number (1 = sys_write)
mov rdi,1 ; arg1: fd (1 = stdout)
mov rsi,buf ; arg2: buffer
mov rdx,[buflen]. ; arg3: buflen
Syscall
§ Good overview of system calls:
- http://blog.rchapman.org/posts/Linux_System_Call_Table_for_x86_64/
- https://github.com/torvalds/linux/blob/master/arch/x86/entry/syscalls/sysc
all_64.tbl
33
