I've been a little bored waiting for my laptop power adapter to arrive so I can use my laptop again, and so I've been doing a little experimenting.

I managed to execute a function I never called. It's the little victories that make me happy.

Anyway, here's some (/dangerous/) source code. Run at your own risk. It prints an error message inside the function that was never called. Obviously, I managed to overwrite the return address of the stack frame with the address of my function, so when it returns, arbitrary code is executed.

This is really nothing special, seeing as how it's my own code and I can make it do anything I want. I won't go into attack vectors, as that's not the point.

Initial attempts to 'fix' what I broke...
{"name":"611989","src":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/1\/9\/197c512c6993d729cf3925d2303cd03c.png","w":1356,"h":1057,"tn":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/1\/9\/197c512c6993d729cf3925d2303cd03c"}

So you can see, it gets the address of an object on the stack, and then decrements the address, overwriting memory until the return address is overwritten with the address of the function I want to call.

NOTE : This will destroy your stack, and may? destroy your computer. Maybe you should run it in a VM, through something safe like GDB.

I can only test on Windows XP at the moment. This is with MinGW GCC 8.1.

So far I've managed to get it to run arbitrary code, but I can't seem to put humpty dumpty back together right yet.

{"name":"611990","src":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/6\/7\/6793e6aa09dd2ff2b5e982912205ada2.png","w":1356,"h":1057,"tn":"\/\/djungxnpq2nug.cloudfront.net\/image\/cache\/6\/7\/6793e6aa09dd2ff2b5e982912205ada2"}

For now, it halts with an abort(); so it doesn't turn into Skynet.

Oh sweet youth

I'm posting now to help me remember tomorrow. I used to be heavily invested in this sort of thing. I think I may have posted my detouring library on here at some point (takes function address, decompiles function header, relocates/recompiles it somewhere else, inserts jump to your function where header was; so all function calls get redirected then return to the original function). Anyway, I want to look at what you did when I'm awake. Too exhausted to look now.

[[EDIT]]
Okay, back awake.

So, what it comes down to, is the call and ret instructions in ASM. The call instruction is functionally equivalent to

// RIP is the 64 bit pointer to the currently executing instruction. 
// Y is the size of a push instruction plus the size of the jmp function. 
// I say Y because in x86/64 land, these instructions come out to 
// different machine code width different numbers of bytes depending on
// the arguments. For instance, a jmp instruction can be anywhere between 2
// and 14 bytes long
push rip+Y // Push the return address (directly after the jmp) onto the stack
jmp SomeFunction // Move the instruction pointer to the function we want to call

Meanwhile, the ret instruction is functionally equivalent to

// Pull the top value off the stack and put it in the instruction
// pointer, effectively jumping there and removing it from the
// stack simultaneously
pop rip

So, obviously, as soon as you're in your function, the top stack entry is the return address, right? Wrong.
Just like in C/C++, in ASM there is the concept of stack scope, which is managed by RBP and RSP. RBP points to the current bottom of the stack, and RSP to the current head. This means calling push is functionally equivalent to

sub RSP, sizeof(X) // Move the head of the stack to make room (stack grows down for some reason lost to time)
mov [RSP], X // Store X at the current address of RSP, the stack head

And calling pop is the reverse.
What about RBP then? I said there was scope, and RBP is where it happens. The standard entry sequence to a function (the stuff that gets called before your variables are initialized) is

push RBP // Store the old base of the stack on the stack
mov RBP, RSP // Set the new base as the current top of the stack
sub RSP, X // Allocate X off the stack, where X is the sizeof all local variables

This means (because, again, the stack grows down for some reason) inside a function, you can use [RBP - X] to access local variables and [RBP + X + 8] to access the parent scope's variables (+8 to skip the return address). Writing into [RBP + X] would be bad, since that means you're modifying outside your scope. Of course, there's no safety here, so nothing is stopping you.

What it comes down to, is a stack frame for the following would be something like...

// Note: This is assuming x64 with a __fastcall calling convention,
// meaning the first two parameters are passed in registers, rather
// than on the stack
// Address Value
   0x100   0x100    // Absolute base of the stack
   0x108   bar_done // Return address after bar is called. The next instruction
   0x110   0x100    // The base of the stack for main
   0x118   a        // bar::a
   0x120   b        // bar::b
   0x128   foo_done // The return address after foo is called.
   0x130   0x110    // The stack base pointer for bar
   0x138   x        // foo::x

That's all generally true anyway. In reality, optimizing compilers will sometimes give you non-standard stack frames, particularly on leaf functions (functions that don't call anything). Furthermore, a nice debugging compiler will probably insert byte patterns in the callstack in and around function calls which they can check for on return to make sure you didn't smash the stack.

If you want to know more, such as how to properly hot patch/detour a function, how to inject code into a foreign process, how to evade process intrusion detection facilities ... ... all for academic purposes only, let me know.