python - BeautifulSoup does not see element , even though it is present on a page

Question

I am trying to scrape listings from Airbnb. Every listing has its own ID. However, the output of the code below is None:

import requests, bs4

response = requests.get('https://www.airbnb.pl/s/Girona--Hiszpania/homes?refinement_paths%5B%5D=%2Fhomes&query=Girona%2C%20Hiszpania&checkin=2018-07-04&checkout=2018-07-25&allow_override%5B%5D=&ne_lat=42.40450221314142&ne_lng=3.3245690859736214&sw_lat=41.97668610374056&sw_lng=1.7960961855829964&zoom=10&search_by_map=true&s_tag=nrGiXgWC')  
soup = bs4.BeautifulSoup(response.text, "html.parser")

element = soup.find(id="listing-18354577")
print(element)

Why does the soup does not see this element, even though it is already loaded on the page?

Is it in a container of some type I need to scrape differently?

Answer 1

The element with id listing-18354577 is created via javascript after the initial HTML page has loaded into your browser. Requests is just an HTTP client, not a full-fledged browser engine, so it doesn't execute the Javascript that ends up fetching that element. The response from Requests is just the initial HTML of the page (which does not include listing-18354577).