c++ - LoadLibrary failed with error code 126 if app started from installation option

Question

I have two application and install in different folders, let call it app A and B, A is main application and B is an COM module, A will start B through COM API after A started, there are some DLLs need to be loaded by B while B started, if I start A by double click the shortcut of A, every thing is ok, but if I install A, and start A by check the start A option in the last dialog of the installation, then B is started, but one of the DLLs load failed with error code 126 (ERROR_MOD_NOT_FOUND), if I exit and restart again by double click the shortcut, it works again.

Already do some googles and seems the only difference between start from shortcut and installation is current directory, ie, if start from installation option, same as start from the installer package folder with cmd, like open cmd, switch to the folder of installer package, then start app A with full path, I have try this, also works well.

My installer package is build by installshield.

Is anyone have some clues about this issue?

1. Already try to switch current directory to the install path of A and B, both can not solve this issue.
2. Already try to set dll directory to the install path of B, which also is the path of the failed DLL, not work too.
3. Already try to load the DLL with full path, also failed.

    //SetCurrentDirectory(L"C:Program Files (x86)install path of A"); <<<not work
    //SetCurrentDirectory(L"C:Program Files (x86)install paht of B");   <<<not work
    //SetDllDirectory(L"C:Program Files (x86)DLL path");   <<<not work
    //m_hLibrary = LoadLibrary((LPCWSTR)DLL full path);   //not work
    m_hLibrary = LoadLibrary((LPCWSTR)dllName.c_str());   //failed with error code 126

Answer 1

It sounds like you have diagnosed the symptoms. Solving them may be tricky, as at least some versions of InstallShield call APIs that reduce the ability to load unexpected DLLs before an application folder is established. It appears that the effects of these calls carry over to your process when you launch it directly from the installer.

So first, oversimplified option 1: remove the option to start your app from the last page of the wizard. Poof, problem goes away. But this probably makes someone else unhappy.

Instead let's try to dive into what's going on. Depending on the exact version of InstallShield, it may be calling some combination of APIs, but the most likely culprit is a call to SetDllDirectory(L""); According to some quick research, this should only have an effect on implicitly loaded DLLs in child processes, but that doesn't appear to be the scenario you describe.

You've tried undoing this call by explicitly adding a directory; here are my recommended (but untested) options 2 and 3:

• As documented on SetDllDirectory, call SetDllDirectory(NULL) to restore the default search order, or
• Call SetDllDirectory or AddDllDirectory to add your folder, and then call LoadLibraryEx(..., LOAD_LIBRARY_SEARCH_USER_DIRS)