I am trying to scrape a website and I am getting a 403 Forbidden error no matter what I try:
- wget
- CURL (command line and PHP)
- Perl WWW::Mechanize
- PhantomJS
I tried all of the above with and without proxies, changing user-agent, and adding a referrer header.
I even copied the request header from my Chrome browser and tried sending with my request using PHP Curl and I am still getting a 403 Forbidden error.
Any input or suggestions on what is triggering the website to block the request and how to bypass?
PHP CURL Example:
$url ='https://www.vitacost.com/productResults.aspx?allCategories=true&N=1318723&isrc=vitacostbrands%3aquadblock%3asupplements&scrolling=true&No=40&_=1510475982858';
$headers = array(
'accept:application/json, text/javascript, */*; q=0.01',
'accept-encoding:gzip, deflate, br',
'accept-language:en-US,en;q=0.9',
'referer:https://www.vitacost.com/productResults.aspx?allCategories=true&N=1318723&isrc=vitacostbrands:quadblock:supplements',
'user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36',
'x-requested-with:XMLHttpRequest',
);
$res = curl_get($url,$headers);
print $res;
exit;
function curl_get($url,$headers=array(),$useragent=''){
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_ENCODING, '');
if($useragent)curl_setopt($curl, CURLOPT_USERAGENT,$useragent);
if($headers)curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
$response = curl_exec($curl);
$header_size = curl_getinfo($curl, CURLINFO_HEADER_SIZE);
$header = substr($response, 0, $header_size);
$response = substr($response, $header_size);
curl_close($curl);
return $response;
}
And here is the response I always get:
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
You don't have permission to access
"http://www.vitacost.com/productResults.aspx?"
on this server.<P>
Reference #18.55f50717.1510477424.2a24bbad
</BODY>
</HTML>
See Question&Answers more detail:
os 
