python - Web scraping results in 403 Forbidden Error

Question

I'm trying to web scrape the earnings for each company off SeekingAlpha using BeautifulSoup. However, it seems like the site is detecting that a web scraper is being used? I get a "HTTP Error 403: Forbidden"

The page I'm attempting to scrape is: https://seekingalpha.com/symbol/AMAT/earnings

Does anyone know what can be done to bypass this?

Answer 1

You should try setting User-Agent as one of request headers. Value can be of any known browser.

Example:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36