web scraping - What to do when proxies are giving Max retries exceeded with URL from request module python

Question

strong text I am Facing an issue in python

HTTPSConnectionPool(host='www.indeed.co.in', port=443): Max retries exceeded with url: /jobs?as_phr=python&l=india&limit=15&sort=date&from=advancedsearch&sr=directhire

For that I am using proxy in python script but after some successful run it is showing Max retries exceeded with URL. and sometime getting ReCaptcha

import requests
from bs4 import BeautifulSoup as bs

proxy_host = " Host name"
proxy_port = "Port number " 
proxy_auth = "Key"
proxies = {"https": "https://{}@{}:{}/".format(proxy_auth, 
proxy_host,proxy_port), "http": "http://{}@{}:{}/".format(proxy_auth, 
proxy_host, proxy_port)}

urll = 'https://www.indeed.com/jobs?q=amazon+aurora+- 
   company%3Aamazon+aurora&l=ohio&radius=100&limit=15&sort=date&sr=directhire'

page = requests.get(urll,proxies=proxies,verify=False)
soup = bs(page.text,'html.parser')

        

question from:https://stackoverflow.com/questions/65626511/what-to-do-when-proxies-are-giving-max-retries-exceeded-with-url-from-request-mo

Answer 1

Waitting for answers