python - Incomplete HTML-response on some sites using Requests & BeautifulSoup or Selenium

Question

I'm tying to scrape information from some urls using Requests and BeautifulSoup in Python. But some sites only return an partial HTML response missing the content of the page

This is the code, that is not working:

import requests
from bs4 import BeautifulSoup
url = "http://www.exampleurl.com"
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')

Here is the incomplete response: Picture

I tried to use Selenium with Chrome Webdriver instead, but ended up with the same issue.

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
browser = webdriver.Chrome(options=options)
browser.get(url)
html = browser.page_source

Any ideas?

question from:https://stackoverflow.com/questions/65640818/incomplete-html-response-on-some-sites-using-requests-beautifulsoup-or-seleniu

Answer 1

What happens

1. You do not get the expected html cause it is in an iframe
2. Try to get the src of the iframe soup.find('iframe')['src'] and request with it again.

Example

import requests
from bs4 import BeautifulSoup
url = "http://www.ingenieur-jobs.de/jobangebote/3075/"
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')

iframe = requests.get(soup.find('iframe')['src'])

soup = BeautifulSoup(iframe.content, 'html.parser')
soup