java - Bad Gateway 502 error with Apache mod_proxy and Tomcat

Question

We're running a web app on Tomcat 6 and Apache mod_proxy 2.2.3. Seeing a lot of 502 errors like this:

Bad Gateway! The proxy server received an invalid response from an upstream server.

The proxy server could not handle the request GET /the/page.do.

Reason: Error reading from remote server

If you think this is a server error, please contact the webmaster.

Error 502

Tomcat has plenty of threads, so it's not thread-constrained. We're pushing 2400 users via JMeter against the app. All the boxes are sitting inside our firewall on a fast unloaded network, so there shouldn't be any network problems.

Anyone have any suggestions for things to look at or try? We're heading to tcpdump next.

UPDATE 10/21/08: Still haven't figured this out. Seeing only a very small number of these under load. The answers below haven't provided any magical answers...yet. :)

Answer 1

Just to add some specific settings, I had a similar setup (with Apache 2.0.63 reverse proxying onto Tomcat 5.0.27).

For certain URLs the Tomcat server could take perhaps 20 minutes to return a page.

I ended up modifying the following settings in the Apache configuration file to prevent it from timing out with its proxy operation (with a large over-spill factor in case Tomcat took longer to return a page):

Timeout 5400
ProxyTimeout 5400

---

Some backgound

ProxyTimeout alone wasn't enough. Looking at the documentation for Timeout I'm guessing (I'm not sure) that this is because while Apache is waiting for a response from Tomcat, there is no traffic flowing between Apache and the Browser (or whatever http client) - and so Apache closes down the connection to the browser.

I found that if I left the Timeout setting at its default (300 seconds), then if the proxied request to Tomcat took longer than 300 seconds to get a response the browser would display a "502 Proxy Error" page. I believe this message is generated by Apache, in the knowledge that it's acting as a reverse proxy, before it closes down the connection to the browser (this is my current understanding - it may be flawed).

The proxy error page says:

Proxy Error

The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET.

Reason: Error reading from remote server

...which suggests that it's the ProxyTimeout setting that's too short, while investigation shows that Apache's Timeout setting (timeout between Apache and the client) that also influences this.