java - How to stream response body with apache HttpClient

Question

There's an api I need to perform octet-streaming from which does not have a length. It is just a stream of real time data. The issue that I'm having is that when I make my request, it seems to try to wait out for the end of the content before reading information into the inputstream, however it's not seeing the end of the content and timingout with NoHttpResponse exception. Below is a simplified version of my code:

private static HttpPost getPostRequest() {
    // Build uri
    URI uri = new URIBuilder()
            .setScheme("https")
            .setHost(entity.getStreamUrl())
            .setPath("/")
            .build();

    // Create http http
    HttpPost httpPost = new HttpPost(uri);

    String nvpsStr = "";
    Object myArray[] = nvps.toArray();
    for(int i = 0; i < myArray.length; i ++) {
        nvpsStr += myArray[i].toString();
        if(i < myArray.length - 1) {
            nvpsStr += "&";
        }
    }

    // Build http payload
    String request = nvpsStr + scv + streamRequest + "

";
    // Attach http data
    httpPost.setEntity(new StringEntity(URLEncoder.encode(request,"UTF-8")));

    return httpPost;
}

// Where client is simply
// private static final CloseableHttpClient client = HttpClients.createDefault();
private static runPostRequest (HttpPost request) {
    CloseableHttpResponse response = client.execute(request);
    try {
        HttpEntity ent = response.getEntity();
        InputStream is = ent.getContent();
        DataInputStream dis = new DataInputStream(is);
        // Only stream the first 200 bytes
        for(int i = 0; i < 200; i++) {
            System.out.println(( (char)dis.readByte()));
        }

    } finally {
        response.close();
    }
}

Answer 1

EDIT 2

So, if you're not comfortable with threads/runnables/Handlers and not comfortable with android AsyncTask, I would just go straight to HttpUrlConnection (drop the entire excercise with Apache HttpClient because, basically Google says that HttpUrlConnection will support streamed response and it does work!)

It may not be as easy instrumenting all the details like dumping headers. But with a normal streamed response Object, I think that it should just work.... see edit 3 for HttpsUrlConnection code sample

EndEdit2

Not clear from the question what 'stream' protocol is being used (progressive download or HTTP streaming) OR how you are actually managing the streamed response on your client.

Recommended to dump the headers from the connection to see exactly what the client and server are agreeing on??

I'm assuming that you are OFF the UI thread (either in AsyncTask or in the callback portion of a Handler); if that's not accurate you may have to refactor a little bit.

Assuming HTTP stream in use with Apache HttpClient 4.3.5+

If there is no length in the headers of the response, then you are doing a 'chunked' response on HTTP 1.1 where you have to read a buffer until you get a 'last-chunk' or decide to CLOSE either the stream or the Connection:

The server just starts sending (streaming) and the client should handle the 'input-stream' that it gets from the HTTP response by employing a buffer as per the detailed Apache notes on producing entity content.

I don't remember offhand if socket timeout of 30 seconds will pre-empt an active stream? Remember in Apache, separate settings exist in the builder for socket timeout, and read timeout. Don't want socket to close on you and don't want to timeout waiting on a readable stream's available bytes while server is providing the response.

Anyway, the client-side handler just needs to be aware of how the stream ends by inspection of what's read into the buffer...

If the protocol in place is "continue" & "chunked" then the response handler on the client should be in a stream handler loop until it sees the LAST-CHUNK from the http spec.

 response.getEntity().getContent() 

should give you the reference you need to process the response's stream until 'last-chunk'...

I think u should read here on how to consume a buffered entity where more than a single read is going to be require to wind up at the 'last-chunk' in the response. It's another reason why HttpURLConnection may be easier...

Do a loop that handles buffered reads until END signaled by the bytes matching 'last-chunk'.

Then close either the stream or connection as per the detailed Apache notes on consuming entities and reusable Connections.

EDIT code for streamed response in apache HttpClient

In a 'handler's callback or in asyncTask

 request.execute();
...

 processStreamingEntity(response.getEntity());
 response.close();

//implement your own wrapper as mentioned in apache docs

    private void processStreamingEntity(HttpEntity entity) throws IOException {
        InputStreamHttpEntityHC4 bufHttpEntity = new InputStreamHttpEntityHC4(entity);
        while not bufHttpEntity.LAST_CHUNK {
            handleResponse(bufHttpEntity.readLine())
}

EDIT 3

HttpURLConnection version if you go that way. ( uses a MessageHandler but you could consume the bytes in place as this is from a streaming speach example and the words from text are being sent back to UI here)

private void openHttpsConnection(String urlStr, Handler mhandler) throws IOException {
    HttpsURLConnection httpConn = null;
    String line = null;
    try {
        URL url = new URL(urlStr);
        URLConnection urlConn = url.openConnection();               
        if (!(urlConn instanceof HttpsURLConnection)) {
            throw new IOException ("URL is not an Https URL");
        }               
        httpConn = (HttpsURLConnection)urlConn;
        httpConn.setAllowUserInteraction(false);
        httpConn.setInstanceFollowRedirects(true);
        httpConn.setRequestMethod("GET");
        httpConn.setReadTimeout(50 * 1000);
        BufferedReader is =
                new BufferedReader(new InputStreamReader(httpConn.getInputStream()));                   
        
        while ((line = is.readLine( )) != null) {

                Message msg = Message.obtain();
                msg.what=1;  
                msg.obj=line;                       
                mhandler.sendMessage(msg);
            
        }               
    } catch (MalformedURLException e) {
        e.printStackTrace();
    } catch( SocketTimeoutException e){
        e.printStackTrace();
    } catch (IOException e) {

        e.printStackTrace();
        Message msg = Message.obtain();
            msg.what=2;
            BufferedInputStream in = new BufferedInputStream(httpConn.getErrorStream());
    
            line =new String(readStream(in));
            msg.obj=line;
            mhandler.sendMessage(msg);
          
    }
    finally {httpConn.disconnect();}

}