I'm posting now to help me remember tomorrow. I used to be heavily invested in this sort of thing. I think I may have posted my detouring library on here at some point (takes function address, decompiles function header, relocates/recompiles it somewhere else, inserts jump to your function where header was; so all function calls get redirected then return to the original function). Anyway, I want to look at what you did when I'm awake. Too exhausted to look now.
[[EDIT]]
Okay, back awake.
So, what it comes down to, is the call and ret instructions in ASM. The call instruction is functionally equivalent to
// RIP is the 64 bit pointer to the currently executing instruction.
// Y is the size of a push instruction plus the size of the jmp function.
// I say Y because in x86/64 land, these instructions come out to
// different machine code width different numbers of bytes depending on
// the arguments. For instance, a jmp instruction can be anywhere between 2
// and 14 bytes long
push rip+Y // Push the return address (directly after the jmp) onto the stack
jmp SomeFunction // Move the instruction pointer to the function we want to call
Meanwhile, the ret instruction is functionally equivalent to
// Pull the top value off the stack and put it in the instruction
// pointer, effectively jumping there and removing it from the
// stack simultaneously
pop rip
So, obviously, as soon as you're in your function, the top stack entry is the return address, right? Wrong.
Just like in C/C++, in ASM there is the concept of stack scope, which is managed by RBP and RSP. RBP points to the current bottom of the stack, and RSP to the current head. This means calling push is functionally equivalent to
sub RSP, sizeof(X) // Move the head of the stack to make room (stack grows down for some reason lost to time)
mov [RSP], X // Store X at the current address of RSP, the stack head
And calling pop is the reverse.
What about RBP then? I said there was scope, and RBP is where it happens. The standard entry sequence to a function (the stuff that gets called before your variables are initialized) is
push RBP // Store the old base of the stack on the stack
mov RBP, RSP // Set the new base as the current top of the stack
sub RSP, X // Allocate X off the stack, where X is the sizeof all local variables
This means (because, again, the stack grows down for some reason) inside a function, you can use [RBP - X] to access local variables and [RBP + X + 8] to access the parent scope's variables (+8 to skip the return address). Writing into [RBP + X] would be bad, since that means you're modifying outside your scope. Of course, there's no safety here, so nothing is stopping you.
What it comes down to, is a stack frame for the following would be something like...
#SelectExpand
1void foo()
2{
3 int x;
4 // stack frame for right here below
5}
6
7void bar()
8{
9 int a, b;
10 // ...
11 foo(a + b)
12foo_done:
13}
14
15int main()
16{
17 bar();
18bar_done:
19}
// Note: This is assuming x64 with a __fastcall calling convention,
// meaning the first two parameters are passed in registers, rather
// than on the stack
// Address Value
0x100 0x100 // Absolute base of the stack
0x108 bar_done // Return address after bar is called. The next instruction
0x110 0x100 // The base of the stack for main
0x118 a // bar::a
0x120 b // bar::b
0x128 foo_done // The return address after foo is called.
0x130 0x110 // The stack base pointer for bar
0x138 x // foo::x
That's all generally true anyway. In reality, optimizing compilers will sometimes give you non-standard stack frames, particularly on leaf functions (functions that don't call anything). Furthermore, a nice debugging compiler will probably insert byte patterns in the callstack in and around function calls which they can check for on return to make sure you didn't smash the stack.
If you want to know more, such as how to properly hot patch/detour a function, how to inject code into a foreign process, how to evade process intrusion detection facilities ... ... all for academic purposes only, let me know.
]]>
