objective c - Downloading multiple files in batches in iOS

Question

I have an app that right now needs to download hundreds of small PDF's based on the users selection. The problem I am running into is that it is taking a significant amount of time because every time it has to open a new connection. I know that I could use GCD to do an async download, but how would I go about doing this in batches of like 10 files or so. Is there a framework that already does this, or is this something I will have to build my self?

Answer 1

This answer is now obsolete. Now that NSURLConnection is deprecated and NSURLSession is now available, that offers better mechanisms for downloading a series of files, avoiding much of the complexity of the solution contemplated here. See my other answer which discusses NSURLSession.

I'll keep this answer below, for historical purposes.

---

I'm sure there are lots of wonderful solutions for this, but I wrote a little downloader manager to handle this scenario, where you want to download a bunch of files. Just add the individual downloads to the download manager, and as one finishes, it will kick off the next queued one. You can specify how many you want it to do concurrently (which I default to four), so therefore there's no batching needed. If nothing else, this might provoke some ideas of how you might do this in your own implementation.

Note, this offers two advantages:

1. If your files are large, this never holds the entire file in memory, but rather streams it to persistent storage as it's being downloaded. This significantly reduces the memory footprint of the download process.

2. As the files are being downloaded, there are delegate protocols to inform you or the progress of the download.

I've attempted to describe the classes involved and proper operation on the main page at the Download Manager github page.

---

I should say, though, that this was designed to solve a particular problem, where I wanted to track the progress of downloads of large files as they're being downloaded and where I didn't want to ever hold the entire in memory at one time (e.g., if you're downloading a 100mb file, do you really want to hold that in RAM while downloading?).

While my solution solves those problem, if you don't need that, there are far simpler solutions using operation queues. In fact you even hint at this possibility:

I know that I could use GCD to do an async download, but how would I go about doing this in batches of like 10 files or so. ...

I have to say that doing an async download strikes me as the right solution, rather than trying to mitigate the download performance problem by downloading in batches.

You talk about using GCD queues. Personally, I'd just create an operation queue so that I could specify how many concurrent operations I wanted, and download the individual files using NSData method dataWithContentsOfURL followed by writeToFile:atomically:, making each download it's own operation.

So, for example, assuming you had an array of URLs of files to download it might be:

NSOperationQueue *queue = [[NSOperationQueue alloc] init];
queue.maxConcurrentOperationCount = 4;

for (NSURL* url in urlArray)
{
    [queue addOperationWithBlock:^{
        NSData *data = [NSData dataWithContentsOfURL:url];
        NSString *filename = [documentsPath stringByAppendingString:[url lastPathComponent]];
        [data writeToFile:filename atomically:YES];
    }];
}

Nice and simple. And by setting queue.maxConcurrentOperationCount you enjoy concurrency, while not crushing your app (or the server) with too many concurrent requests.

And if you need to be notified when the operations are done, you could do something like:

NSOperationQueue *queue = [[NSOperationQueue alloc] init];
queue.maxConcurrentOperationCount = 4;

NSBlockOperation *completionOperation = [NSBlockOperation blockOperationWithBlock:^{
    [[NSOperationQueue mainQueue] addOperationWithBlock:^{
        [self methodToCallOnCompletion];
    }];
}];

for (NSURL* url in urlArray)
{
    NSBlockOperation *operation = [NSBlockOperation blockOperationWithBlock:^{
        NSData *data = [NSData dataWithContentsOfURL:url];
        NSString *filename = [documentsPath stringByAppendingString:[url lastPathComponent]];
        [data writeToFile:filename atomically:YES];
    }];
    [completionOperation addDependency:operation];
}

[queue addOperations:completionOperation.dependencies waitUntilFinished:NO];
[queue addOperation:completionOperation];

This will do the same thing, except it will call methodToCallOnCompletion on the main queue when all the downloads are done.