http - Python's `urllib2`: Why do I get error 403 when I `urlopen` a Wikipedia page?

Question

I have a strange bug when trying to urlopen a certain page from Wikipedia. This is the page:

http://en.wikipedia.org/wiki/OpenCola_(drink)

This is the shell session:

>>> f = urllib2.urlopen('http://en.wikipedia.org/wiki/OpenCola_(drink)')
Traceback (most recent call last):
  File "C:Program FilesWing IDE 4.0srcdebugserver\_sandbox.py", line 1, in <module>
    # Used internally for debug sandbox under external interpreter
  File "c:Python26Liburllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "c:Python26Liburllib2.py", line 397, in open
    response = meth(req, response)
  File "c:Python26Liburllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "c:Python26Liburllib2.py", line 435, in error
    return self._call_chain(*args)
  File "c:Python26Liburllib2.py", line 369, in _call_chain
    result = func(*args)
  File "c:Python26Liburllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

This happened to me on two different systems in different continents. Does anyone have an idea why this happens?

Answer 1

Wikipedias stance is:

Data retrieval: Bots may not be used to retrieve bulk content for any use not directly related to an approved bot task. This includes dynamically loading pages from another website, which may result in the website being blacklisted and permanently denied access. If you would like to download bulk content or mirror a project, please do so by downloading or hosting your own copy of our database.

That is why Python is blocked. You're supposed to download data dumps.

Anyways, you can read pages like this in Python 2:

req = urllib2.Request(url, headers={'User-Agent' : "Magic Browser"}) 
con = urllib2.urlopen( req )
print con.read()

Or in Python 3:

import urllib
req = urllib.request.Request(url, headers={'User-Agent' : "Magic Browser"}) 
con = urllib.request.urlopen( req )
print(con.read())