winapi - Are TCP SOCKET handles inheritable?

Question

On Windows, most sorts of handles can be inherited by child processes. The expectation is that TCP sockets can also be inherited. However, when certain Layered Service Providers are installed, this does not work as expected (A/V products such as PCTools from Symantec used to cause problems with our application for customers).

The way Microsoft has architectured WinSock, should we expect to be able to inherit SOCKETs correctly?

Answer 1

Short answer

No, SOCKETs should not be marked inheritable. When certain Layered Service Providers (LSPs) are installed, the inherited handles simply can't be used in the child.

As an added irritation, see the related issue "Can TCP SOCKETS be marked non-inheritable?". Briefly, you can't rely on being able to inherit the sockets, but nor can you stop the sockets from being inherited!

Explanation

Sadly, this goes against some of Microsoft's own examples and documentation (such as KB150523). Briefly, Layered Service Providers are a way Microsoft offers for third-party software to insert itself between your application and Microsoft's TCP/UDP stack in their WinSock DLLs. Due to the way some LSPs function, they make it hard to transfer sockets between processes, because the LSP associates some local information with each socket which it requires to be present.

An LSP can only hook into WinSock functions; for example, calling DuplicateHandle on a SOCKET will not work when certain LSPs are installed, because it is a handle-level function and the LSP is never given a chance to copy the information it needs. (This is briefly but clearly stated in the DuplicateHandle documentation).

Similarly, attempting to set a SOCKET handle to be inheritable will copy the handle without informing the LSP, with the same result: the duplicate handle may not be recognized by Winsock in the child process. Typical errors are WSAENOTSOCK (10038, "Socket operation on nonsocket"), or even ERROR_INVALID_HANDLE (6, "The handle is invalid").

Example

Suppose you want to write a Windows program that launches a child with redirected stdin and stdout, sends it some data, signals EOF on the child's stdin so it knows to process the data, and then waits for the child to return.

Let's suppose further that some imaginative form of launching is performed, which means that you the child perhaps isn't a child at all (for example, gksu/runas to launch a wrapper which must exit immediately, leaving you with just the socket connection to the client). You don't therefore have the child's PID to wait on.

The behaviour will be similar to this:

int main(int argc, char* argv[]) {
  int handles[2];
  socketpair(AF_UNIX, SOCK_STREAM, 0, handles);
  if (fork()) {
    // child
    close(handles[0]);
    dup2(handles[1], 0);
    dup2(handles[1], 1);
    execl("clever-app", "clever-app", (char*)0);
  }

  // parent
  close(handles[1]);
  char* data[100];
  write(handles[0], data, sizeof(data)); // should at least check for EINTR...

  // tell the app we called there's nothing more to read from stdin: 
  shutdown(handles[0], SHUT_WR);

  // wait until child has exited (discarding all output)
  while (read(handles[0], data, sizeof(data)) >= 0) ;

  // now continue with the rest of the program...
}

Workaround

On a machine without Layered Service Providers, creating a connected pair of TCP sockets, and inheriting one in the child as stdin/stdout, does behave correctly. It's tempting to use this as the workaround for socketpair behaviour on Windows (remember to send a nonce!).

Sadly, but the SOCKET simply can't be reliably inherited. To write something with almost equivalent functionality on Windows, you need to use Named Pipes. Before calling CreateProcess, create a pair of connected HANDLES instead using CreateNamedPipe/ConnectNamedPipe and friends (GetOverlappedResult for an overlapped parent handle). (The handle for the child, to be used as stdin, must not be overlapped!) The handle for the child can set inheritable, and the child will communicate over it normally.

When you have finished piping data to the client, call FlushFileBuffers and CloseHandle on the parent handle.

Further issues

1. What about waiting for the child to exit before continuing, using only the handle? There isn't a way to do that directly with just the pipe to it; Windows pipes can't be half-closed. Ways to do this:

   1. (The Unix way, contorted to fit Windows) Make another dummy pair of connected handles, and inherit one of them in the child process, but don't tell the child about it (don't attach it as a std handle). Then, you can wait on the second handle in the parent to detect when the child exits.
   2. (The Windows-y way) A right pain, but rather powerful: Use the named pipe to get an actual process handle to the child in the parent. This is the 'correct' way on Windows to wait for a child to exit. (This is actually something you can't do on Unix; only the parent of a process can directly wait on a process to exit; OpenProcess converts a pid to a handle, so if you're careful to check the pid a second time after the OpenProcess call, you can get rid of the race condition that would make this not possible on Unix.) Using a process handle like this is nonetheless a right pain because you'll probably find you need a second named pipe connection to send it over, depending on how you write your runas wrapper.

2. A gotcha: How does the child receive notification that the parent has finished writing to its stdin? If the parent tries to call DisconnectClient, the child doesn't get a normal EOF. Depending on what you're trying to execute, this may be a problem. When the parent shuts down a SOCKET, you do get feof, but if a handle is connected to a child's stdin, the child will get a read error without getting EOF signalled to it. This may cause the child not to work in exactly the same way as if it were hooked up to stdin normally. Calling CloseHandle in the parent instead gives the right behaviour in the child.