Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.9k views
in Technique[技术] by (71.8m points)

web scraping - How to use Google-apps-script to scrape data from a web page that is lazy loaded (via an API)?

I'm trying to create an automated process using Google-apps-script for scraping price data from pages like this one:

https://www.barchart.com/stocks/quotes/$AVVN/price-history/historical

The challenging part is, that the data on the web page is 'lazy loaded', so the 'traditional' scaping methods, that I have used on other web pages, don't work here.

I have considered other ways of solving this problem - but:

  • Barchart does not provide data for e.g. $AVVN via http: //marketdata.websol.barchart.com/getHistory
  • I don't want to use the 'Download'-button - as this requires automated login.
  • ImportXML() does not work (it works for other tables on the web page, but not for the one I want).

I found a similar problem in the following post - that received a very detailed and informative reply from omegastripes: Open webpage, select all, copy into sheet

-but when I run my code:

function test(){
  var url = 'https://www.barchart.com/proxies/core-api/v1/historical/get?symbol=%24AVVN&fields=tradeTime.format(m%2Fd%2Fy)%2CopenPrice%2ChighPrice%2ClowPrice%2ClastPrice%2CpriceChange%2Cvolume%2CsymbolCode%2CsymbolType&startDate=2019-04-15&endDate=2019-07-15&type=eod&orderBy=tradeTime&orderDir=desc&limit=2000&meta=field.shortName%2Cfield.type%2Cfield.description&raw=1'; 
  var options = {
     "muteHttpExceptions": false
  };
  var response   = UrlFetchApp.fetch(url, options);   
  Logger.log(response);
}

-then I get the following error:

Request failed for https://www.barchart.com/proxies/core-api/v1/historical/get?symbol=%24AVVN&fields=tradeTime.format(m%2Fd%2Fy)%2CopenPrice%2ChighPrice%2ClowPrice%2ClastPrice%2CpriceChange%2Cvolume%2CsymbolCode%2CsymbolType&startDate=2019-04-15&endDate=2019-07-15&type=eod&orderBy=tradeTime&orderDir=desc&limit=2000&meta=field.shortName%2Cfield.type%2Cfield.description&raw=1 returned code 500. Truncated server response: <!doctype html> <html itemscope itemtype="http://schema.org/WebPage" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="wi... (use muteHttpExceptions option to examine full response) (line 57, file "DS#1")

Basically an "Oops, something's wrong. Our apologies ... there seems to be a problem with this page." ... if you paste the address into your browser.

So my question is: How can data be scraped from this page or has Barchart now succesfully blocked this scraping option?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

The only way i found to obtain the data was using your workaround, getting the request URL to fetch from the console, but additionally you have to add the “x-xsrf-token" and “cookie” headers to the options when using fetch() method [1].

You can get the “x-xsrf-token” and “cookie” request headers from the console as well. Only problem is that the cookies and xsrf-token are valid up to 2 hours, this is because they implemented cross site request forgery protection [2]:

enter image description here

Here is the code i tested and worked:

function testFunction() {
  var url = 'https://www.barchart.com/proxies/core-api/v1/historical/get?symbol=%24AVVN&fields=tradeTime.format(m%2Fd%2Fy)%2CopenPrice%2ChighPrice%2ClowPrice%2ClastPrice%2CpriceChange%2Cvolume%2CsymbolCode%2CsymbolType&startDate=2019-04-16&endDate=2019-07-16&type=eod&orderBy=tradeTime&orderDir=desc&limit=2000&meta=field.shortName%2Cfield.type%2Cfield.description&raw=1';

  var map = {
    "x-xsrf-token": "XXXXX",
    "cookie": "XXXXX"
  }

  var options = {
     "method": "get", 
     "muteHttpExceptions": false,
     "headers": map
  };
  var response = UrlFetchApp.fetch(url, options);   
  Logger.log(response);

  var json = JSON.parse(response);
  Logger.log(json.data[0]);
}

[1] https://developers.google.com/apps-script/reference/url-fetch/url-fetch-app

[2] Difference between CSRF and X-CSRF-Token


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...