web services - Restful API - handling large amounts of data

Question

I have written my own Restful API and am wondering about the best way to deal with large amounts of records returned from the API.

For example, if I use GET method to myapi.co.uk/messages/ this will bring back the XML for all message records, which in some cases could be 1000's. This makes using the API very sluggish.

Can anyone suggest the best way of dealing with this? Is it standard to return results in batches and to specify batch size in the request?

Answer 1

You can change your API to include additional parameters to limit the scope of data returned by your application.

For instance, you could add limit and offset parameters to fetch just a little part. This is how pagination can be done in accordance with REST. A request like this would result in fetching 10 resources from the messages collection, from 21st to 30th. This way you can ask for a specific portion of a huge data set:

myapi.co.uk/messages?limit=10&offset=20 

Another way to decrease the payload would be to only ask for certain parts of your resources' representation. Here's how facebook does it:

/joe.smith/friends?fields=id,name,picture

Remember that while using either of these methods, you have to provide a way for the client to discover each of the resources. You can't assume they'll just look at the parameters and start changing them in search of data. That would be a violation of the REST paradigm. Provide them with the necessary hyperlinks to avoid it.

I strongly recommend viewing this presentation on RESTful API design by apigee (the screencast is called "Teach a Dog to REST"). Good practices and neat ideas to approach everyday problems are discussed there.

EDIT: The video has been updated a number of times since I posted this answer, you can check out the 3rd edition from January 2013