java - Handling connection errors and JSoup

Question

I'm trying to create an application to scrape content off of multiple pages on a site. I am using JSoup to connect. This is my code:

for (String locale : langList){
        sitemapPath = sitemapDomain+"/"+locale+"/"+sitemapName;
        try {
            Document doc = Jsoup.connect(sitemapPath)
                    .userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21")
                    .timeout(10000)
                    .get();

            Elements element = doc.select("loc");   
            for (Element urls : element) {
                System.out.println(urls.text());
                }
        } catch (IOException e) {
            System.out.println(e);
        }
    }

Everything works perfectly most of the time. However there are a few things I want to be able to do.

First off sometimes a 404 status will return or a 500 status maybe a 301. With my code below it will just print the error and move onto the next url. What I would like to be able to do is try to be able to return the url status for all links. If the page connects print a 200, if not print the relevant status code.

Secondly I sometimes catch this error "java.net.SocketTimeoutException: Read timed out" I could increase my timeout however I would prefer to try to connect 3 times, upon failing the 3rd time I want to add the URL to a "failed" array so I can retry the failed connections in the future.

Can someone with more knowledge than me help me out?

Answer 1

The above returns an IOException for me rather than the execute() returning the correct status code.

Using JSoup-1.6.1 I had to change the above code to use ignoreHttpErrors(true).

Now when the code returns the response rather than throwing an exception and you can check the error codes/messages.

Connection.Response response = null;
            try {
                response = Jsoup.connect(bad_url)
                        .userAgent("Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5")
                        .timeout(100000)
                        .ignoreHttpErrors(true) 
                        .execute();
            } catch (IOException e) {
                System.out.println("io - "+e);
            }

            System.out.println("Status code = " + response.statusCode());   
            System.out.println("Status msg  = " + response.statusMessage());

Output:

Status code = 404
Status msg  = Not Found