http - Fetch a file from a local url with Python requests?

Question

I am using Python's requests library in one method of my application. The body of the method looks like this:

def handle_remote_file(url, **kwargs):
    response = requests.get(url, ...)
    buff = StringIO.StringIO()
    buff.write(response.content)
    ...
    return True

I'd like to write some unit tests for that method, however, what I want to do is to pass a fake local url such as:

class RemoteTest(TestCase):
    def setUp(self):
        self.url = 'file:///tmp/dummy.txt'

    def test_handle_remote_file(self):
        self.assertTrue(handle_remote_file(self.url))

When I call requests.get with a local url, I got the KeyError exception below:

requests.get('file:///tmp/dummy.txt')

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/packages/urllib3/poolmanager.pyc in connection_from_host(self, host, port, scheme)
76 
77         # Make a fresh ConnectionPool of the desired type
78         pool_cls = pool_classes_by_scheme[scheme]
79         pool = pool_cls(host, port, **self.connection_pool_kw)
80 

KeyError: 'file'

The question is how can I pass a local url to requests.get?

PS: I made up the above example. It possibly contains many errors.

question from:https://stackoverflow.com/questions/10123929/fetch-a-file-from-a-local-url-with-python-requests

Answer 1

As @WooParadog explained requests library doesn't know how to handle local files. Although, current version allows to define transport adapters.

Therefore you can simply define you own adapter which will be able to handle local files, e.g.:

from requests_testadapter import Resp

class LocalFileAdapter(requests.adapters.HTTPAdapter):
    def build_response_from_file(self, request):
        file_path = request.url[7:]
        with open(file_path, 'rb') as file:
            buff = bytearray(os.path.getsize(file_path))
            file.readinto(buff)
            resp = Resp(buff)
            r = self.build_response(request, resp)

            return r

    def send(self, request, stream=False, timeout=None,
             verify=True, cert=None, proxies=None):

        return self.build_response_from_file(request)

requests_session = requests.session()
requests_session.mount('file://', LocalFileAdapter())
requests_session.get('file://<some_local_path>')

I'm using requests-testadapter module in the above example.