python - ConnectionResetError: [Errno 104] Connection reset by peer and ERR_NAME_NOT_RESOLVED on heroku with mobile testing through Selenium

Question

I'd like to test multiple mobile user agents with selenium and chrome. I'm using python 3.6 and deploying to heroku. Based on http://chromedriver.chromium.org/mobile-emulation .

You can download my project for both windows and heroku use at:

https://github.com/kc1/mobiletest

(keep in mind that if you deploy to heroku you have to set FLASK_CONFIG to production. Also please note that the code in the project is slightly different than in this question because I've been playing with the code over the past week.)

I have:

def some_long_calculation():
    driver = create_chromedriver('kkk')
    # driver = create_chromedriver()

    driver.get("https://www.yahoo.com/")
    .....

and :

def create_chromedriver(ua=False):
    options = webdriver.ChromeOptions()
    CHROMEDRIVER_PATH = os.getenv('$HOME') or basedir+'/chromedriver.exe'
    FLASK_CONFIG = os.getenv('FLASK_CONFIG')

    if ua:

        mobile_emulation = {"deviceName": "Nexus 5"}
        options.add_experimental_option("mobileEmulation", mobile_emulation)


    if FLASK_CONFIG and FLASK_CONFIG == "production":
        CHROMEDRIVER_PATH = '/app/.chromedriver/bin/chromedriver'
        GOOGLE_CHROME_SHIM = os.getenv('$GOOGLE_CHROME_SHIM') or 'no path found'
        options.binary_location = '/app/.apt/usr/bin/google-chrome-stable'

        options.add_argument('--disable-gpu')
        options.add_argument('--no-sandbox')

    return webdriver.Chrome(executable_path=CHROMEDRIVER_PATH, options=options)  

If I run it locally with the mobile browser enabled It works as expected:

If I run it on heroku with the mobile browser enabled :

Then I tried it on heroku with the mobile user disabled I get:

So at least I know the setup is working as far as chrome and chromedriver.

heroku Logs:

2018-07-15T17:37:53.967643+00:00 app[web.1]:     driver = create_chromedriver('kkk')
2018-07-15T17:37:53.967637+00:00 app[web.1]:     png = some_long_calculation()
2018-07-15T17:37:53.967645+00:00 app[web.1]:   File "/app/app/main/cl.py", line 120, in create_chromedriver
2018-07-15T17:37:53.967640+00:00 app[web.1]:   File "/app/app/main/cl.py", line 123, in some_long_calculation
2018-07-15T17:37:53.967648+00:00 app[web.1]:     return webdriver.Chrome(executable_path=CHROMEDRIVER_PATH, options=options)
2018-07-15T17:37:53.967651+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 75, in __init__
2018-07-15T17:37:53.967654+00:00 app[web.1]:     desired_capabilities=desired_capabilities)
2018-07-15T17:37:53.967656+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 156, in __init__
2018-07-15T17:37:53.967659+00:00 app[web.1]:     self.start_session(capabilities, browser_profile)
2018-07-15T17:37:53.967661+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 251, in start_session
2018-07-15T17:37:53.967669+00:00 app[web.1]:     response = self.command_executor.execute(driver_command, params)
2018-07-15T17:37:53.967664+00:00 app[web.1]:     response = self.execute(Command.NEW_SESSION, parameters)
2018-07-15T17:37:53.967667+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 318, in execute
2018-07-15T17:37:53.967672+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 472, in execute
2018-07-15T17:37:53.967674+00:00 app[web.1]:     return self._request(command_info[0], url, body=data)
2018-07-15T17:37:53.967677+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 496, in _request
2018-07-15T17:37:53.967679+00:00 app[web.1]:     resp = self._conn.getresponse()
2018-07-15T17:37:53.967682+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/http/client.py", line 1331, in getresponse
2018-07-15T17:37:53.967685+00:00 app[web.1]:     response.begin()
2018-07-15T17:37:53.967687+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/http/client.py", line 297, in begin
2018-07-15T17:37:53.967695+00:00 app[web.1]:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2018-07-15T17:37:53.967690+00:00 app[web.1]:     version, status, reason = self._read_status()
2018-07-15T17:37:53.967698+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/socket.py", line 586, in readinto
2018-07-15T17:37:53.967692+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/http/client.py", line 258, in _read_status
2018-07-15T17:37:53.967700+00:00 app[web.1]:     return self._sock.recv_into(b)
2018-07-15T17:37:53.967712+00:00 app[web.1]: ConnectionResetError: [Errno 104] Connection reset by peer

How can I fix this?

EDIT:

Thanks for your detailed answer. I've changed the code to incorporate the flags you mentioned. Chrome version is 67.0.3396.99. Chromedriver is 2.40 and selenium is 3.13. Unfortunately, there is no change in the result. I'm still getting the same error. As far as your stage 2 and 3 advice. I'm currently deploying to heroku so I do not have total control of the environment variables. Is there a way to make of these changes using python?

EDIT 2:

As I think about it more on https://sites.google.com/a/chromium.org/chromedriver/mobile-emulation the example uses

from selenium import webdriver
mobile_emulation = { "deviceName": "Nexus 5" }
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
driver = webdriver.Remote(command_executor='http://127.0.0.1:4444/wd/hub',
                  desired_capabilities = chrome_options.to_capabilities())

Are you suggesting the browser is at 'http://127.0.0.1:4444/wd/hub'

Answer 1

ConnectionResetError: [Errno 104] Connection reset by peer

In general when a client terminates abruptly without closing the connection a RST packet is sent by the TCP/IP stack of the underlying OS. Python converts this into an exception with the text Connection reset by peer. As per your error stack trace it means that once self._read_status() was invoked (internally) Python assumed to receive something but the connection was suddenly dropped and Python informs you of this error by raising the exception:

ConnectionResetError: [Errno 104] Connection reset by peer

The situation is somewhat similar to this expression:

"Connection reset by peer" is the TCP/IP equivalent of slamming the phone back on the hook. It's more polite than merely not replying, leaving one hanging. But it's not the FIN-ACK expected of the truly polite TCP/IP converseur.

There can be multiple probabilities behind this error as follows.

---

Solution Stage A

A quick and a precise solution will be to add a few recommended ChromeOptions along with the existing one as follows:

options.add_argument("start-maximized"); // open Browser in maximized mode
options.add_argument("disable-infobars"); // disabling infobars
options.add_argument("--disable-extensions"); // disabling extensions
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage"); // overcome limited resource problems

and then

return webdriver.Chrome(executable_path=CHROMEDRIVER_PATH, options=options) 

Note: You need to remove the argument options.add_argument('--disable-gpu') as it is applicable to windows OS only.

---

Solution Stage B

A couple of points:

• As per the documentation within Python Chrome Mobile Emulation the Key and Value pair seems to be "deviceName": "Google Nexus 5" (not "deviceName": "Nexus 5")

• You can tweak you code to invoke Remote() through either of the following ways-

  • Invoking Remote() through DesiredCapabilities():

    from selenium import webdriver
    # import DesiredCapabilities was missing in your program
    from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
    
    mobile_emulation = { "deviceName": "Google Nexus 5" }
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
    capabilities = DesiredCapabilities.CHROME
    capabilities = options.to_capabilities()
    driver = webdriver.Remote(command_executor='http://127.0.0.1:4444/wd/hub', desired_capabilities = capabilities)
    

  • You find a similar discussion on invoking Remote() through ChromeOptions() in How to add selenium chrome options to 'desiredCapabilities'?

  • Invoking Remote() through ChromeOptions():

    from selenium import webdriver
    mobile_emulation = { "deviceName": "Google Nexus 5" }
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
    driver = webdriver.Remote(command_executor='http://127.0.0.1:4444/wd/hub', options=chrome_options)
    

  • You find a similar discussion on invoking Remote() through ChromeOptions() in Remote WebDriver UnreachableBrowserException: Could not start a new session

• For your reference here is the link to the documentation on Webdriver.android
• For your reference here is the link to the documentation on Getting started with Selendroid

---

Solution Stage C

If you are still seeing the error, perform the following upgradation/cleanup tasks:

• Upgrade Selenium to current levels Version 3.13.0.
• Upgrade ChromeDriver to current ChromeDriver v2.40 level.
• Keep Chrome version between Chrome v66-68 levels. (as per ChromeDriver v2.40 release notes)
• Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
• (WindowsOS only) Use CCleaner tool to wipe off all the OS chores before and after the execution of your Test Suite.
• (LinuxOS only) Free Up and Release the Unused/Cached Memory in Ubuntu/Linux Mint before and after the execution of your Test Suite.
• If your base Web Client version is too old, then uninstall it through Revo Uninstaller and install a recent GA and released version of Web Client.
• Take a System Reboot.
• Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.
• Execute your @Test.

---

Solution Stage D

Looking out for a granular solution to specific error I stepped into Amazon S3 and "Connection Reset by Peer" where Garry Dolley summerizes the the cause of the problem to be a combination of the below mentioned factors:

• TCP Window Scaling
• Linux kernels 2.6.17 or newer

Linux kernels 2.6.17+ increased the maximum size of the TCP window/buffer, and this started to cause other gear to wig out, if it couldn’t handle sufficiently large TCP windows. The gear would reset the connection, and we see this as a “Connection reset by peer” message.

A pottential solution will be to put the following entries within /etc/sysctl.conf:

• net.ipv4.tcp_wmem = 4096 16384 512000
• net.ipv4.tcp_rmem = 4096 87380 512000

Note: This fix is easy peasy, but it’ll slow your maximum throughput inexchange of faster downloads.

---

PS

If applicable, ensure that /etc/hosts on your system contains the following entry :

127.0.0.1               localhost.localdomain localhost

---

Related Discussions

Here are some of the related discussions:

• Python socket.error: [Errno 104] Connection reset by peer
• Yet Another 'Connection reset by peer' Error
• Connection reset by peer [errno 104] in Python 2.7
• Remote WebDriver UnreachableBrowserException: Could not start a new session
• How to add selenium chrome options to 'desiredCapabilities'?

---

References

Here are the references of this discussion:

• Example Python Chrome Mobile Emulation Automated Unit Testing Using Selenium 2 WebDriver ChromeDriver
• Amazon S3 and "Connection Reset by Peer"
• How To: Network / TCP / UDP Tuning
• How To : Free Up and Release the Unused/Cached Memory in Ubuntu/Linux Mint
• Mobile Emulation
• selenium.webdriver.android.webdriver
• Getting started with Selendroid