locking - accept() with sockets shared between multiple processes (based on Apache preforking)

Question

I'm working on some Python code modeled on Apache's MPM prefork server. I am more an applications programmer than a network programmer and it's been 10 years since I read Stevens, so I'm trying to get up to speed in understanding the code.

I found a short description of how Apache's prefork code works, by Sander Temme.

The parent process, which typically runs as root, binds to a socket (usually port 80 or 443). It spawns children, which inherit the open file descriptor for the socket, and change uid and gid to the unprivileged user and group. The children construct a pollset of the listener file descriptors (if there is more than one listener) and watch for activity on it/them. If activity is found, the child calls accept() on the active socket and handles the connection. When it is done with that, it returns to watching the pollset (or listener file descriptor).

Since multiple children are active and they all inherited the same socket file descriptor(s), they will be watching the same pollset. An accept mutex allows only a single child to actually watch the pollset, and once that has found an active socket it will unlock the mutex so the next child can start watching the pollset. If there is only a single listener, that accept mutex is not used and all children will hang in accept().

This is pretty much the way the code I'm looking at works, but I don't understand a few things.

1) What is the difference between a "child" and a "listener"? I thought each child is a listener, which is true for the code I'm looking at, but in Temme's description there can be "a single listener" and "children." When would a child have multiple listeners?

2) (Related to 1) Is this a per-process mutex or a system mutex? For that matter, why have a mutex? Doesn't accept(2) do its own mutex across all listeners? My research says I do need a mutex and that the mutex must be across the entire system. (flock, semaphore, etc.)

Temme goes on to say:

Children record in a shared memory area (the scoreboard) when they last served a request. Idle children may be killed by the parent process to satisfy MaxSpareServers. If too few children are idle, the parent will spawn children to satisfy MinSpareServers.

3) Is there a good reference code for this implementation (preferably in Python)? I found Perl's Net::Server::Prefork, which uses pipes instead of shared memory for the scoreboard. I found an article by Randal Schwartz which only does the preforking but doesn't do the scoreboard.

The pre-fork example from the Perl Cookbook does not have any sort of locking around select, and Chris Siebenmann's Python example says it's based on Apache but uses paired sockets for the scoreboard, not shared memory, and use the sockets for controls, include the control for a given child to 'a'ccept. This does not match the Apache description at all.

Answer 1

In respect of (1), the listener is merely a reference to the existence of socket on which to accept connections. Since Apache can accept connections on multiple sockets at the same time, eg., 80/443, then there are multiple listener sockets. Each child process would need to listen on all of these sockets when it comes its time. Since accept() can only be done on one socket at a time, it is preceded by the poll/select so it is known on which listener socket the accept should be performed.

In respect of (2), it is a global or cross process mutex. That is, one process locking it will block out other processes trying to acquire the same lock. Although accept() will technically serialise the processes, the presence of multiple listener sockets means you cannot rely on that as you don't know before hand which socket to perform accept on. Even where a single listener socket, the reason for the accept mutex is that if there are large numbers of processes handling requests, then it could be quite expensive if operating system wakes up all processes to see which then has accept() return for it. Since Apache in prefork mode may have 100+ processes, that could cause a problem.

So, if you have only a single listener socket and know you only have a few processes wanting to do the accept() call then you can possibly do away with the cross process accept mutex.