web scraping - How to use R to download a zipped file from a SSL page that requires cookies

Question

I am trying to download a file from an https page that requires an "I Agree" button be pushed and then stores a cookie. My apologies if this answer is obvious somewhere..

When I open up the web page directly in Chrome and click "I Agree" - the file starts to download automatically.

http://www.icpsr.umich.edu/cgi-bin/bob/zipcart2?path=SAMHDA&study=32722&bundle=delimited&ds=1&dups=yes

I tried to replicate this example, but I don't think that hangseng website actually stores the cookie/authentication, so I don't know if that example should be all I need.

Beyond that, I believe the SSL complicates the authentication, since I think the getURL() call will require a certificate specification like cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl"))

I'm too much of a beginner with RCurl to know if this website is pretty difficult or if I'm just missing something obvious.

Thank you!

Answer 1

This is a bit easier to do with httr because it sets up everything so that cookies and https work seamlessly.

The easiest way to generate the cookies is to have the site do it for you, by manually posting the information that the "I agree" form generates. You then do a second request to download the actual file.

library(httr)
terms <- "http://www.icpsr.umich.edu/cgi-bin/terms"
download <- "http://www.icpsr.umich.edu/cgi-bin/bob/zipcart2"

values <- list(agree = "yes", path = "SAMHDA", study = "32722", ds = "", 
  bundle = "all", dups = "yes")

# Accept the terms on the form, 
# generating the appropriate cookies
POST(terms, body = values)
GET(download, query = values)

# Actually download the file (this will take a while)
resp <- GET(download, query = values)

# write the content of the download to a binary file
writeBin(content(resp, "raw"), "c:/temp/thefile.zip")