html - rvest web scraping is returning an empty data frame when attempting to collect product price information

Question

I am trying to use 'rvest' to scrape product pricing from: https://www.lowes.com/pl/Lawn-garden-hand-tools-Outdoor-tools-equipment-Outdoors/4294612737?goToProdList=true&int_cmp=LawnGardenHandTools:C:Outdoors:Merch:shop_all_copy. I am using the below code:

library(rvest)
library(tidyverse)

url <- "https://www.lowes.com/pl/Lawn-garden-hand-tools-Outdoor-tools-equipment-Outdoors/4294612737?goToProdList=true&int_cmp=LawnGardenHandTools:C:Outdoors:Merch:shop_all_copy"

html <- read_html(url)

price <- html %>%
  html_node('body') %>%
  xml_find_all("//span[contains(@class, 'h5 js-price v-spacing-mini art-pl-price')]") %>% 
  html_text() %>%
  data.frame()

However - this is returning an empty data frame.

Any advice would be much appreciated.

Answer 1

rvest can only scrape static HTML content.

Most modern commercial websites use dynamic web content generated on the fly by a JavaScript.

In order to scrape such websites, you will first need to make the site generate the HTML content you are looking for, and then you'll be able to scrape it with rvest.

To do so, you'll need to use a web browser emulator like RSelenium or Splash to allow you to move around in the site and query data programmatically.

RSelenium needs installation of a Docker Selenium server. This is also recommended for Splash.

At the end of this long and interesting journey, you'll have to be creative so that the Website doesn't think that it's being queried by a robot :