selenium - How to parse several attributes of website with same class name in python?

Question

I want to parse addresses from this website (https://www.conad.it/) with Pyhthon after having searched fro a CAP in the search bar and entered the result. For many CAP's there are many addresses of stores that result and I want to scrape all of them, not just the first one (which is what my code is now doing).

Here's my code so far:

driver = webdriver.Chrome('pathtoChrome/chromedriver.exe')
driver.get("https://www.conad.it/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='javascript:void(0)']"))).click() # accept the cookies
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).send_keys("11100")
driver.find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click()
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[contains(@class,'col-md-8')]"))).get_attribute("innerHTML"))

Which has this as final output:

<h3>Conad</h3><p>Frazione Condemine 84, 11010  Sarre</p><div class="extra-services extra-services-buttons extra-services-desktop extra-services-simple"><ul class="carousel-services"></ul></div>

I would want only the output within the <p> in the upper output but for all attributes within the class 'col-md-8, so for this example of CAP also for the second address.

Optimally I want to store it in a data set which I can append over several loops of different CAP's, so something like this (which doesn't work yet..):

driver = webdriver.Chrome('pathtoChrome/chromedriver.exe')
driver.get("https://www.conad.it/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@href='javascript:void(0)']"))).click() # accept the cookies
CAPS = ['11100']
for CAP in CAPS:
   WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='location-input']"))).send_keys(CAP)
   driver.find_element_by_xpath("//input[@class = 'btn btn-default btn-lg btn-block']").click()
   print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[contains(@class,'col-md-8')]"))).get_attribute("innerHTML"))

Any help is appreciated!

Answer 1

You can use WebDriverWait() and wait for visibility_of_all_elements_located() and following xpath option to get all p tag value in a list.

print([item.text for item in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class,'col-md-8')]//p")))])

Your output would be like a list.

['Frazione Condemine 84, 11010 Sarre', 'Grand Chemin C/c Centreville 3, 11020 Saint-christophe', "Localita' Arensod 27, 11010 Sarre"]