get pointer on byte array from unmanaged c++ dll in c#

Question

in c++ I have such function

extern "C" _declspec(dllexport) uint8* bufferOperations(uint8* incoming, int size)

I am trying to call it from c# like this

[DllImport("MagicLib.DLL", CallingConvention = CallingConvention.Cdecl)]
//[return: MarshalAs(UnmanagedType.ByValArray)]//, ArraySubType=UnmanagedType.SysUInt)]
public static extern byte[] bufferOperations(byte[] incoming, int size);

But I get the Cannot marshal 'return value': Invalid managed/unmanaged type combination

((( The question is - how to marshal this correctly? Thanks for reading my question

Answer 1

byte[] is a .Net array type with known length. You can't marshal byte* to it, because .Net does not know the length of output array. You should try manual marshalling. Replace byte[] with byte*. Then, do like this:

[DllImport("MagicLib.DLL", CallingConvention = CallingConvention.Cdecl)]
public static extern byte* bufferOperations(byte* incoming, int size);

public void TestMethod()
{
    var incoming = new byte[100];
    fixed (byte* inBuf = incoming)
    {
        byte* outBuf = bufferOperations(inBuf, incoming.Length);
        // Assume, that the same buffer is returned, only with data changed.
        // Or by any other means, get the real lenght of output buffer (e.g. from library docs, etc).
        for (int i = 0; i < incoming.Length; i++)
            incoming[i] = outBuf[i];
    }
}