61. API Hooking - Custom Code

API Hooking - Custom Code


So far, open source libraries have been used to implement API hooking. However, a major issue with this approach is that the source code for these libraries is publicly available, making it straightforward for security researchers and security product vendors to build IoCs. For this reason, API hooking will be implemented manually in this module, although not as sophisticated as the previously demonstrated libraries, but enough to achieve the desired result without IoCs.

Custom hooking code can be a better option if the intent is to hook a single function. This avoids the additional effort of linking other libraries, and avoiding the additional weight these libraries add to the binary's size.

Creating The Trampoline Shellcode

One of the ways to hook a function is to overwrite its first few instructions with new ones. These new instructions are the trampoline which is responsible for altering the execution flow of the function to the replacement function. This trampoline is typically a small jump shellcode that executes a jmp instruction to the address of the function to be executed. To execute the jmp instruction, the address that needs to be jumped to must be saved inside of a register. In the presented example, the register will be eax on a 32-bit processor and r10 on a 64-bit processor. A mov instruction will be used to save the address inside of these registers.

This is all that is needed for the trampoline, a mov and a jmp instruction. Diving deeper into how these instructions are used is not the focus of this module. If one would like to explore them further, felixcloutier.com/x86/mov and felixcloutier.com/x86/jmp can provide more details.

64-bit Jump Shellcode

The 64-bit jump shellcode should be as follows:

mov r10, pAddress
jmp r10

Where pAddres is the address of the function to jump to (e.g. 0x0000FFFEC32A300). To use these instructions in the code they must first be converted to opcode.

0x49, 0xBA, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // mov r10, pAddress
0x41, 0xFF, 0xE2                                            // jmp r10

32-bit Jump Shellcode

And the 32-bit version:

mov eax, pAddress
jmp eax

Again, convert the instructions to opcode.

0xB8, 0x00, 0x00, 0x00, 0x00,     // mov eax, pAddress
0xFF, 0xE0                        // jmp eax

Note that pAddress is represented as NULL, which explains the 0x00 sequence. These 0x00 opcodes are placeholders that will be overwritten during runtime.

Retrieving pAddress

Since the hooks are installed during runtime, the pAddress value must be retrieved and added to the shellcode during runtime. The retrieval of the address can be done using GetProcAddress and once that's completed, memcpy is used to copy the address to the correct location in the shellcode.

64-bit Patching

uint8_t		uTrampoline[] = {
			0x49, 0xBA, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // mov r10, pFunctionToRun
			0x41, 0xFF, 0xE2                                            // jmp r10

uint64_t uPatch = (uint64_t)pAddress;
memcpy(&uTrampoline[2], &uPatch, sizeof(uPatch)); // copying the address to the offset '2' in uTrampoline

32-bit Patching

uint8_t		uTrampoline[] = {
	   0xB8, 0x00, 0x00, 0x00, 0x00,     // mov eax, pFunctionToRun
	   0xFF, 0xE0                        // jmp eax

uint32_t uPatch = (uint32_t)pAddress;
memcpy(&uTrampoline[1], &uPatch, sizeof(uPatch)); // copying the address to the offset '1' in uTrampoline

As previously mentioned, pAddress is the address of the function to jump to. The uint32_t and uint64_t data types are used to ensure that the address is the correct number of bytes, that is 4 bytes for 32-bit machines and 8 bytes for 64-bit machines. uint32_t is of size 4 bytes, and uint64_t is of size 8 bytes. memcpy will then place the address into the trampoline by overwriting the 0x00 placeholder bytes.

Writing The Trampoline

Before overwriting the target function's first few instructions with the prepared shellcode, it is important to mark the memory where the trampoline will be written as writable. In most cases, the memory region will not be writable, requiring the VirtualProtect WinAPI to change the memory permissions to PAGE_EXECUTE_READWRITE. It is worth noting that it must be writable and executable because when the program calls the function, it needs to execute instructions that will not be permitted on write-only memory.

With that in mind, the trampoline should first modify the permissions of the target function and then copy the shellcode over.

// Changing the memory permissons at 'pFunctionToHook' to be PAGE_EXECUTE_READWRITE
if (!VirtualProtect(pFunctionToHook, sizeof(uTrampoline), PAGE_EXECUTE_READWRITE, &dwOldProtection)) {
	return FALSE;

// Copying the trampoline shellcode to 'pFunctionToHook'
memcpy(pFunctionToHook, uTrampoline, sizeof(uTrampoline));

Where pFunctionToHook is the address of the function to hook, and uTrampoline is the jump shellcode.


When the hooked function is called, the trampoline shellcode should be able to work for both 64-bit and 32-bit architectures. However, the unhooking of the hooked function has not been discussed. To do this, the original bytes which were overwritten by the trampoline should be restored by using a buffer containing these bytes that were created prior to the installation of the trampoline shellcode. This buffer should then be used as the source buffer in the memcpy function when unhooking the function.

memcpy(pFunctionToHook, pOriginalBytes, sizeof(pOriginalBytes));

Where pFunctionToHook is the address of the hooked function and pOriginalBytes is the buffer that's holding the original bytes of the function which should have been saved before hooking, and can be done via a memcpy call. The size of the pOriginalBytes buffer should be the same as the trampoline shellcode size that way only the shellcode is overwritten. Lastly, it's recommended to revert the memory permissions which can be done via the code snippet below.

if (!VirtualProtect(pFunctionToHook, sizeof(uTrampoline), dwOldProtection, &dwOldProtection)) {
	return FALSE;

Where dwOldProtection is the old memory permission returned by the first VirtualProtect call.

HookSt Structure

To make the implementation easier, the HookSt structure was created. This structure will contain the needed information to hook and unhook a certain function. The value TRAMPOLINE_SIZE is set to 13 if the program is set to be compiled as a 64-bit application, and its set to 7 if the program is to be compiled in 32-bit mode. The values 13 and 7 are the sizes of the trampoline shellcode, denoted in the uTrampoline variable previously shown, in 64-bit and 32-bit systems, respectively.

typedef struct _HookSt{

	PVOID	pFunctionToHook;                  // address of the function to hook
	PVOID	pFunctionToRun;                   // address of the function to run instead
	BYTE	pOriginalBytes[TRAMPOLINE_SIZE];  // buffer to keep some original bytes (needed for cleanup)
	DWORD	dwOldProtection;                  // holds the old memory protection of the "function to hook" address (needed for cleanup)

}HookSt, *PHookSt;

Setting the TRAMPOLINE_SIZE value is done via the following preprocessor code

// if compiling as 64-bit
#ifdef _M_X64#define TRAMPOLINE_SIZE		13#endif // _M_X64// if compiling as 32-bit
#ifdef _M_IX86#define TRAMPOLINE_SIZE		7#endif // _M_IX86

Installing Hooks

The following function uses HookSt to install hooks.

BOOL InstallHook (IN PHookSt Hook) {

#ifdef _M_X64// 64-bit trampoline
	uint8_t	uTrampoline [] = {
			0x49, 0xBA, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // mov r10, pFunctionToRun
			0x41, 0xFF, 0xE2                                            // jmp r10

	// Patching the shellcode with the address to jump to (pFunctionToRun)
	uint64_t uPatch = (uint64_t)(Hook->pFunctionToRun);
	// Copying the address of the function to jump to, to the offset '2' in uTrampoline
	memcpy(&uTrampoline[2], &uPatch, sizeof(uPatch));
#endif // _M_X64#ifdef _M_IX86// 32-bit trampoline
	uint8_t	uTrampoline[] = {
	   0xB8, 0x00, 0x00, 0x00, 0x00,     // mov eax, pFunctionToRun
	   0xFF, 0xE0                        // jmp eax

	// Patching the shellcode with the address to jump to (pFunctionToRun)
	uint32_t uPatch = (uint32_t)(Hook->pFunctionToRun);
	// Copying the address of the function to jump to, to the offset '1' in uTrampoline
	memcpy(&uTrampoline[1], &uPatch, sizeof(uPatch));
#endif // _M_IX86

	// Placing the trampoline function - installing the hook
	memcpy(Hook->pFunctionToHook, uTrampoline, sizeof(uTrampoline));

	return TRUE;

Removing Hooks

The function below uses HookSt to remove hooks.

BOOL RemoveHook (IN PHookSt Hook) {

	DWORD	dwOldProtection		= NULL;

	// Copying the original bytes over
	memcpy(Hook->pFunctionToHook, Hook->pOriginalBytes, TRAMPOLINE_SIZE);
	// Cleaning up our buffer
	memset(Hook->pOriginalBytes, '\0', TRAMPOLINE_SIZE);
	// Setting the old memory protection back to what it was before hooking
	if (!VirtualProtect(Hook->pFunctionToHook, TRAMPOLINE_SIZE, Hook->dwOldProtection, &dwOldProtection)) {
		printf("[!] VirtualProtect Failed With Error : %d \n", GetLastError());
		return FALSE;

	// Setting all to null
	Hook->pFunctionToHook   = NULL;
	Hook->pFunctionToRun    = NULL;
	Hook->dwOldProtection   = NULL;

	return TRUE;

Populating The HookSt Structure

The InitializeHookStruct function is used to populate the HookSt structure with the necessary information to perform hooking.

BOOL InitializeHookStruct(IN PVOID pFunctionToHook, IN PVOID pFunctionToRun, OUT PHookSt Hook) {

	// Filling up the struct
	Hook->pFunctionToHook   = pFunctionToHook;
	Hook->pFunctionToRun    = pFunctionToRun;

	// Save original bytes of the same size that we will overwrite (that is TRAMPOLINE_SIZE)
	// This is done to be able to do cleanups when done
	memcpy(Hook->pOriginalBytes, pFunctionToHook, TRAMPOLINE_SIZE);

	// Changing the protection to RWX so that we can modify the bytes
	// We are saving the old protection to the struct (to re-place it at cleanup)
	if (!VirtualProtect(pFunctionToHook, TRAMPOLINE_SIZE, PAGE_EXECUTE_READWRITE, &Hook->dwOldProtection)) {
		printf("[!] VirtualProtect Failed With Error : %d \n", GetLastError());
		return FALSE;

	return TRUE;

The Main function

The main function below calls the previously demonstrated functions and hooks the MessageBoxA WinAPI.

int main() {

	// Initializing the structure (needed before installing/removing the hook)
	HookSt st = { 0 };

	if (!InitializeHookStruct(&MessageBoxA, &MyMessageBoxA, &st)) {
		return -1;

	// will run
	MessageBoxA(NULL, "What Do You Think About Malware Development ?", "Original MsgBox", MB_OK | MB_ICONQUESTION);

	//  hooking
	if (!InstallHook(&st)) {
		return -1;

	//  wont run - hooked
	MessageBoxA(NULL, "Malware Development Is Bad", "Original MsgBox", MB_OK | MB_ICONWARNING);

	//  unhooking
	if (!RemoveHook(&st)) {
		return -1;

	//  will run - hook disabled
	MessageBoxA(NULL, "Normal MsgBox Again", "Original MsgBox", MB_OK | MB_ICONINFORMATION);

	return 0;


Due to the trampoline-based hook, it is impossible to have a global original function pointer be called to resume execution. Therefore, the MessageBoxW WinAPI will be called in the MyMessageBoxA detour function.

Running the first MessageBoxA (Unhooked).

The original MessageBoxA instructions before hooking.

Running the second MessageBoxA (Hooked).

The trampoline shellcode is in memory.

Running the third MessageBoxA (Unhooked).