java - getting javax.net.ssl.SSLException: Received fatal alert: protocol_version while scraping data using Jsoup

Question

I am trying to get data from a site using Jsoup. Link to the site is Click here!

Here is my code to fetch the data. `

    // WARNING: do it only if security isn't important, otherwise you have 
    // to follow this advices: http://stackoverflow.com/a/7745706/1363265
    // Create a trust manager that does not validate certificate chains
    TrustManager[] trustAllCerts = new TrustManager[]{new X509TrustManager(){
        public X509Certificate[] getAcceptedIssuers(){return null;}
        public void checkClientTrusted(X509Certificate[] certs, String authType){}
        public void checkServerTrusted(X509Certificate[] certs, String authType){}
    }};

    // Install the all-trusting trust manager
    try {
        SSLContext sc = SSLContext.getInstance("TLS");
        sc.init(null, trustAllCerts, new SecureRandom());
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
    } catch (Exception e) {
        ;
    }`

String url = "https://www.sos.nh.gov/corporate/soskb/SearchResults.asp?FormName=CorpNameSearch&Words=Starting&SearchStr="+query+"&SearchType=Search"; Connection.Response response = Jsoup.connect(url).timeout(30000) .method(Connection.Method.GET) .userAgent("Mozilla/5.0 (Windows NT 10.0; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0")
.execute(); Document document = response.parse();

Please suggest me what is my mistake here.

Answer 1

You want to use Java 8 here since it supports TLSv1.2 by default with additional required cipher suites.

Why not Java 7?

I tested on my box with Java 7 (1.7.0_45) and got the same error.

I activated the debugging messages and forced TLSv1.2.

System.setProperty("javax.net.debug", "all");
System.setProperty("https.protocols", "TLSv1.2");

Then I hit this new error:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Finally, I went to Comodoca's SSL analyzer and see something interesting. According to SSL analyzer, the site you're targeting has only enabled the following cipher suites:

Cipher Suites Enabled
Name  (ID)                                       Key Size (in bits)
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256  (0xC02F)  128   ECDH 256-bit (P-256) 
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384  (0xC030)  256   ECDH 256-bit (P-256) 
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256  (0x9E)      128   DH 2048-bit  
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384  (0x9F)      256   DH 2048-bit

(see Full details)

On my side, I don't have any of the above suites. Check if you have them:

SSLContext sc = SSLContext.getInstance("TLS");
sc.init(null, null, new java.security.SecureRandom());

String[] scs = sc.getSocketFactory().getSupportedCipherSuites();
Arrays.sort(scs);

for(String s : scs) {
   System.out.println(s);
}

See the SSLSocketFactoryEx for enabling the needed cipher suites.

Why Java 8?

On the other hand, I succeed in runnnig the code by moving from Java 7 to Java 8 (1.8.0_20) which support TLS v1.2 by default and provides the needed cipher suites.

Here is a trimmed list of supported cipher suites (71 suites in total) for Java 8 (1.8.0_20) on Windows 7.

TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
...
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

Snippet

try {
    // Create a trust manager that does not validate certificate chains
    TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() {
        public java.security.cert.X509Certificate[] getAcceptedIssuers() {
            return null;
        }

        public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType) {
        }

        public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType) {
        }
    } };

    // Install the all-trusting trust manager
    SSLContext sc = SSLContext.getInstance("SSL");
    sc.init(null, trustAllCerts, new java.security.SecureRandom());
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());

    // Fetch url
    String url = "https://www.sos.nh.gov/corporate/soskb/SearchResults.asp?FormName=CorpNameSearch&Words=All&SearchStr=facebook&SearchType=Search";

    Connection.Response response = Jsoup //
            .connect(url) //
            .timeout(60000) //
            .method(Connection.Method.GET) //
            .userAgent("Mozilla/5.0 (Windows NT 10.0; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0") //
            .execute();

    Document document = response.parse();
    System.out.println(document);
} catch (Exception e) {
    e.printStackTrace();
}

Final thought:

When it comes to security, ALWAYS use the latest updated version.