multithreading - WCF Concurrent requests piling up on the server when using WSHttpBinding

Question

I have a WCF client/server app which is communicating over HTTP using the WSHttpBinding.

Server Setup: self-hosting, using the standard WCF ServiceHost. My actual service class is attributed as:

[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple, 
 InstanceContextMode = InstanceContextMode.PerSession, 
 UseSynchronizationContext = false)]

Client Setup: using a visual-studio generated client proxy using synchronous service calls (proxy.call_server_method blocks until such time as the server has responded in full.)

Scenario: I have one particular method call which takes 20 seconds to execute on the server. The client calls this method in a separate thread, so it is not being held up, and the ConcurrencyMode.Multiple means WCF should execute it in a separate thread on the server as well.

This theory is supported by the fact that when I configure my app to use NetTcpBinding, everything works fine.

Problem:
If I configure the app to use WSHttpBinding, then this long method call causes the http requests to 'back up'. I have verified this behaviour both from inspecting my logs, and by debugging the HTTP requests using fiddler.

Example:

• Client initiates 20-second long request on a background thread
• Client initiates request B and C on foreground thread
• Requests B and C get sent to the server, which doesn't process them until it is done with the 20-second long request

But sometimes:

• Requests B and C do not get sent (they don't even appear in fiddler) until the 20-second request comes back (this is rare).
  • Note: setting <add address="*" maxconnection="100"/> in the client's app.config made this (appear to) stop happening.
• Request B gets sent and receives a response immediately, while request C is held back until the 20-second one completes (this is rare)

Here's a timeline from fiddler demonstrating the problem: (click for bigger version)

As you can see, the requests are all getting backed up at the server. Once the 20-second request completes, the responses all come flooding through, but note that there are some requests which aren't getting held up...

So, Questions:

• What the heck is going on here? Why does it work fine using NetTcpBinding and not work using WSHttpBinding?
• Why the inconsistent behaviour?
• What can I do to fix it?

Notes:

• It's not locking on the server. I've set breakpoints and used !syncblk and it consistently reports no locks are being held.
• It's not my threading (NetTcpBinding shouldn't work otherwise)
• I have <serviceThrottling maxConcurrentCalls="1000" maxConcurrentInstances="1000" maxConcurrentSessions="1000" /> set in the server's app.config
• The 20-second call is just waiting on a timer, it's not thrashing the CPU or disk or network
• I'd prefer a solution that didn't involve re-architecting the application to use asynchronous calls... it's a big bunch of legacy code and I really don't want to be messing with stuff I don't understand.

Answer 1

There is some throttle outside of WCF (a .Net or Windows thing) that defaults to only letting a maximum of two simultaneous outbound HTTP connections. Unfortunately I can't remember for the life of me the name of the thing (and what you'd put in app.config or your app to override it). Given that you're not seeing the requests leave the client, and that it's only HTTP, I think you're hitting "that thing". I'll keep looking for its name.

Update: Found it - try this on the client (but change the '2' to a bigger number):

<configuration>
  <system.net>
    <connectionManagement>
      <add address = "*" maxconnection = "2" />
    </connectionManagement>
  </system.net>
</configuration>