(Multiple edits, following a number of comments.)
setWantClientAuth is used to request client certificate authentication, but keep the connection if no authentication is provided. setNeedClientAuth is used to request and require client certificate authentication: the connection will terminate if no suitable client certificate is presented.
You can find more on this topic in the Client Certificate section of the TLS specification. For a bit of history:
In version 1.2, it says "If no suitable certificate is available, the client MUST send a certificate message containing no certificates.". Before that, it was just a "SHOULD", but the Sun JSSE clients send an empty list this case anyway.
Version 1.2 also added:
> Also, if some aspect of the certificate chain was unacceptable (e.g.,
> it was not signed by a known, trusted CA), the server MAY at its
> discretion either continue the handshake (considering the client
> unauthenticated) or send a fatal alert.
This gives some flexibility regarding what to do when a unacceptable certificate is sent. The JSSE chooses to send a fatal alert. (`setWantAuth` could in principle carry on with invalid certificates, but not treat the peer as authenticated, as if no client certificate was sent, but this isn't the case.)
Previous versions of the TLS spec said "*If client authentication is required by the server for the handshake to continue, it may respond with a fatal handshake failure alert.*". This is the difference between need or want as implemented in the JSSE: using "need", the server responds with a fatal handshake failure, whereas using "want", the server carries on with the connection, but doesn't treat it as authenticated.
I initially though that your client wasn't sending its certificate when you were using "need". Indeed, most clients won't send a client certificate at all if they can't find a client certificate that is issued by one of the issuers listed in the CA names send by the server during its request (or if the client can't build the chain themselves, which is a common problem).
By default, the JSSE uses the CAs in the truststore to build that list.
For this reason, your client would probably not send a client certificate at all if a suitable issuer isn't in the server's truststore.
You can check whether a client certificate is sent using Wireshark. If you're not running on a port normally used with SSL/TLS, you'll need to right click on a packet and choose "Decode As... -> Transport -> SSL".
There, you should see a Certificate Request message coming from the server. (For some reason, when I'm using the default JRE truststore with Wireshark, that message appears as an "Encrypted Handshake Message", just after the "Server Key Exchange" message. However, it's not encrypted: you can clearly see a number of CA names if you look at the ASCII rendering of the packet in the bottom panel. Perhaps this is because this message is too long, I'm not sure.) With a shorter list, for example, a trust store with a single CA, Wireshark you decode this properly as a Certificate Request message, and you should see the list of accepted CAs in the "Distinguished Names" section.
You should also see a Certificate message coming from the client (not the one coming from the server, of course). If the server requests (with want or need) a certificate, you should always see this message from the client anyway.
Assuming you have access to a test CA (with a client certificate issued by that CA), you can try the following experiments.
If you set up your trust store with that test CA cert, use setWantClientAuth(true), the client will send its client certificate, and the connection will proceed. The server can then get the client certificate from the SSLSession, as expected.
If you use the default trust store (that doesn't contain your test CA cert), use setWantClientAuth(true), the CA DN will not be in the Certificate Request. The client will send a Certificate message, but the certificate list will be empty (Certificates Length: 0 in Wireshark). Here, the client is actually not sending a client certificate, even if its keystore is configured to do so, simply because it can't find a suitable match. The connection will proceed (you may get an exception if you try to read the peer certificate from the SSLSession on the server, but that's not fatal). This is the use-case for setWantClientAuth(true); setNeedClientAuth(true) would have ended the connection immediately.
For the sake of this experiment, you can fake the list of DNs sent by the server in Java.
KeyManagerFactory kmf = //... Initialise a KMF with your server's keystore
TrustManagerFactory tmf = TrustManagerFactory
.getInstance(TrustManagerFactory.getDefaultAlgorithm());
tmf.init((KeyStore) null); // Use the default trust store
TrustManager[] trustManagers = tmf.getTrustManagers();
final X509TrustManager origTrustManager = (X509TrustManager) trustManagers[0];
final X509Certificate caCert = // Load your test CA certificate here.
X509TrustManager fakeTrustManager = new X509TrustManager() {
public void checkClientTrusted(X509Certificate[] chain,
String authType) throws CertificateException {
// Key the behaviour of the default trust manager.
origTrustManager.checkClientTrusted(chain, authType);
}
public void checkServerTrusted(X509Certificate[] chain,
String authType) throws CertificateException {
// Key the behaviour of the default trust manager.
origTrustManager.checkServerTrusted(chain, authType);
}
public X509Certificate[] getAcceptedIssuers() {
// This is only used for sending the list of acceptable CA DNs.
return new X509Certificate[] { caCert };
}
};
trustManagers = new X509TrustManager[] { fakeTrustManager };
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(kmf.getKeyManagers(), trustManagers, null);
In this case, the Certificate Request message sent by the server should contain the your test CA's DN. However, that CA isn't actually trusted by the trust manager, which still uses the default values.
The client will send its certificate, but the server will reject it, saying "javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed", and this will end the connection. This is at least the implementation using the SunJSSE provider, using the PKIX or SunX509 trust managers. This is also consistent with the JSSE specification of the trust manager: "The primary responsibility of the TrustManager is to determine whether the presented authentication credentials should be trusted. If the credentials are not trusted, the connection will be terminated."
The key point here is that if you're in a position to get the client certificate from the SSLSession, that certificate should have been authenticated by the trust manager (by this I mean the SSLSession you get once the handshake has completed, with SSLSocket.getSession() not the one you get during the handshake using getHandshakeSession(), introduced in Java 7).
You seem to indicate in comments that you're using another JSSE provider, and that your client was sending a client certificate anyway, whether or not the CA cert was in the server's trust store, because you also had another different CA cert in your trust store with the same Subject DN. Assuming those two CA certificates have different keys (otherwise they would effectively be the same CA), this would be a rather serious bug: applications that use client-certificate authentication are entitled to expect the client certificate to have been verified by the trust manager (as specified in the JSSE reference guide). If you're using Apache Tomcat, once it gets the client certificate, the remote connection is considered as authenticated with this cert. If at that point the servlet is able to use a client certificate that can't have been verified, no authentication has actually been made, which would be a serious flaw.
