html - Getting ‘wrong’ page source when calling url from python

Question

Welcome To Ask or Share your Answers For Others

html - Getting ‘wrong’ page source when calling url from python

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

html - Getting ‘wrong’ page source when calling url from python

Trying to retrieve the page source from a website, I get a completely different (and shorter) text than when viewing the same page source through a web browser.

https://stackoverflow.com/questions/24563601/python-getting-a-wrong-source-code-of-the-web-page-asp-net

This fellow has a related issue, but obtained the home page source instead of the requested one - I am getting something completely alien.

The code is:

from urllib import request

def get_page_source(n):
    url = 'https://www.whoscored.com/Matches/' + str(n) + '/live'
    response = request.urlopen(url)
    return str(response.read())

n = 1006233
text = get_page_source(n)

This is the page I am targeting in this example: https://www.whoscored.com/Matches/1006233/live

The url in question contains rich information in the page source, but I end up getting only the following when running the above code:

text =

b'<html style="height:100%"><head><META NAME="ROBOTS" CONTENT="NOINDEX,
NOFOLLOW"><meta name="format-detection" content="telephone=no"><meta 
name="viewport" content="initial-scale=1.0"><meta http-equiv="X-
UA-Compatible" content="IE=edge,chrome=1"></head><body style="margin:0px;
height:100%"><iframe src="/_Incapsula_Resource?CWUDNSAI=24&
xinfo=0-12919260-0 0NNY RT(1462118673272 111) q(0 -1 -1 -1) r(0 -1) 
B12(4,315,0) U2&incident_id=276000100045095595-100029307305590944&edet=12&
cinfo=04000000" frameborder=0 width="100%" height="100%" marginheight="0px" 
marginwidth="0px">Request unsuccessful. Incapsula incident ID: 
276000100045095595-100029307305590944</iframe></body></html>'

What went wrong here? Can a server detect a robot even when it has not sent repetitive requests – if yes, how – and is there a way around?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…

1 Answer

深蓝 · Answer 1 · 2021-10-23T19:22:53+0000

There are a couple of issues here. The root cause is that the website you are trying to scrape knows you're not a real person and is blocking you. Lots of websites do this simply by checking headers to see if a request is coming from a browser or not (robot). However, this site looks like they use Incapsula, which is designed to provide more sophisticated protection. You can try and setup your request differently to fool the security on the page by setting headers - but I doubt this will work.

import requests

def get_page_source(n):
    url = 'https://www.whoscored.com/Matches/' + str(n) + '/live'
    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
    response = requests.get(url, headers=headers)
    return response.text

n = 1006233
text = get_page_source(n)
print text

Looks like the site also uses captchas - which are designed to prevent web scraping. If a site is trying this hard to prevent scraping - it's likely because the data they provide is proprietary. I would suggest finding another site that provides this data - or try and use an official API.

Check out this (https://stackoverflow.com/a/17769971/701449) answer from a while back. It looks like the whoscored.com uses the OPTA API to provide info. You may be able to skip the middleman and go straight to the source of the data. Good luck!

Categories

html - Getting ‘wrong’ page source when calling url from python

html - Getting ‘wrong’ page source when calling url from python

Please log in or register to add a comment.

Please log in or register to answer this question.

1 Answer

Please log in or register to add a comment.

Just Browsing Browsing

Most popular tags