c++ - How to alloc a executable memory buffer?

Question

I would like to alloc a buffer that I can execute on Win32 but I have an exception in visual studio cuz the malloc function returns a non executable memory zone. I read that there a NX flag to disable... My goal is convert a bytecode to asm x86 on fly with keep in mind performance.

Does somemone can help me?

JS

Answer 1

You don't use malloc for that. Why would you anyway, in a C++ program? You also don't use new for executable memory, however. There's the Windows-specific VirtualAlloc function to reserve memory which you then mark as executable with the VirtualProtect function applying, for instance, the PAGE_EXECUTE_READ flag.

When you have done that, you can cast the pointer to the allocated memory to an appropriate function pointer type and just call the function. Don't forget to call VirtualFree when you are done.

Here is some very basic example code with no error handling or other sanity checks, just to show you how this can be accomplished in modern C++ (the program prints 5):

#include <windows.h>
#include <vector>
#include <iostream>
#include <cstring>

int main()
{
    std::vector<unsigned char> const code =
    {
        0xb8,                   // move the following value to EAX:
        0x05, 0x00, 0x00, 0x00, // 5
        0xc3                    // return what's currently in EAX
    };    

    SYSTEM_INFO system_info;
    GetSystemInfo(&system_info);
    auto const page_size = system_info.dwPageSize;

    // prepare the memory in which the machine code will be put (it's not executable yet):
    auto const buffer = VirtualAlloc(nullptr, page_size, MEM_COMMIT, PAGE_READWRITE);

    // copy the machine code into that memory:
    std::memcpy(buffer, code.data(), code.size());

    // mark the memory as executable:
    DWORD dummy;
    VirtualProtect(buffer, code.size(), PAGE_EXECUTE_READ, &dummy);

    // interpret the beginning of the (now) executable memory as the entry
    // point of a function taking no arguments and returning a 4-byte int:
    auto const function_ptr = reinterpret_cast<std::int32_t(*)()>(buffer);

    // call the function and store the result in a local std::int32_t object:
    auto const result = function_ptr();

    // free the executable memory:
    VirtualFree(buffer, 0, MEM_RELEASE);

    // use your std::int32_t:
    std::cout << result << "
";
}

It's very unusual compared to normal C++ memory management, but not really rocket science. The hard part is to get the actual machine code right. Note that my example here is just very basic x64 code.