python - Why does targeting shadow dom elements fail at the 5th element?

Question

Recently asked how to target elements in chrome settings here: How to edit chromes search and homepage with selenium/python?

I was told I should use 'shadow dom' elements, so I went ahead and figured out how to do that.

I was able to successfully target the search field on chrome's download page using shadow dom inception, but when applying almost identical logic/code to target the to open a specific page or set of pages on chrome://settings/ , python returned an error

no such element: Unable to locate element: {"method":"css selector","selector":"settings-on-startup-page"}

How do I solve the problem?

---

Eduard Florinescu commented: "when you usually have also dynamic content and more than 3 shadow elements one into another it makes impossible automation." here: How to handle elements inside Shadow DOM from Selenium

I'm hoping someone can explain the limitation or workaround here?

HERE IS EXAMPLE CODE THAT WORKS FOR TARGETING THE SEARCH FIELD ON CHROMES DOWNLOAD PAGE

import selenium
from selenium import webdriver
driver = webdriver.Chrome("C:/Users/John/Desktop/Documents/selenium/webdrivers/chromedriver.exe")

#define inception function
def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root


#start doing inception
driver.get("chrome://downloads")
root1 = driver.find_element_by_tag_name('downloads-manager')
shadow_root1 = expand_shadow_element(root1)

root2 = shadow_root1.find_element_by_tag_name('downloads-toolbar')
shadow_root2 = expand_shadow_element(root2)

root3 = shadow_root2.find_element_by_tag_name('cr-toolbar')
shadow_root3 = expand_shadow_element(root3)

root4 = shadow_root3.find_element_by_css_selector("cr-toolbar-search-field")
shadow_root4 = expand_shadow_element(root4)

root5 = shadow_root4.find_element_by_id("searchInput")

root5.send_keys('test')

HERE IS EXAMPLE CODE THAT DOE NOT WORK FOR TARGETING THE OPEN A SPECIFIC PAGE OR SET OF PAGES RADIO BUTTON.

from selenium import webdriver
driver = webdriver.Chrome("C:/Users/John/Desktop/Documents/selenium/webdrivers/chromedriver.exe")


#define inception function
def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root


#start doing inception
driver.get("chrome://settings/")
root1 = driver.find_element_by_tag_name('settings-ui')
shadow_root1 = expand_shadow_element(root1)

root2 = shadow_root1.find_element_by_tag_name('settings-main')
shadow_root2 = expand_shadow_element(root2)

root3 = shadow_root2.find_element_by_tag_name('settings-basic-page')
shadow_root3 = expand_shadow_element(root3)

root4 = shadow_root3.find_element_by_css_selector("settings-section")
shadow_root4 = expand_shadow_element(root4)

root5 = shadow_root4.find_element_by_css_selector("settings-on-startup-page")
shadow_root5 = expand_shadow_element(root5)

root6 = shadow_root5.find_element_by_css_selector("settings-radio-group")
shadow_root6 = expand_shadow_element(root6)

root7 = shadow_root6.find_element_by_name("4")

root7.click()

AS YOU CAN SEE, BOTH CODE SNIPPETS ARE ALMOST IDENTICAL IN SYNTAX AND STRUCTURE, YET THE SECOND DOESN'T WORK.

C:UsersJohnDesktopDocumentsseleniumprojectsstartpage_domain_testvenvScriptspython.exe C:/Users/John/Desktop/Documents/selenium/projects/startpage_domain_test/startpage_domain_test.py
Traceback (most recent call last):
  File "C:/Users/John/Desktop/Documents/selenium/projects/startpage_domain_test/startpage_domain_test.py", line 26, in <module>
    root5 = shadow_root4.find_element_by_css_selector("settings-on-startup-page")
  File "C:Python37-32libsite-packagesseleniumwebdriver
emotewebelement.py", line 430, in find_element_by_css_selector
    return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
  File "C:Python37-32libsite-packagesseleniumwebdriver
emotewebelement.py", line 659, in find_element
    {"using": by, "value": value})['value']
  File "C:Python37-32libsite-packagesseleniumwebdriver
emotewebelement.py", line 633, in _execute
    return self._parent.execute(command, params)
  File "C:Python37-32libsite-packagesseleniumwebdriver
emotewebdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "C:Python37-32libsite-packagesseleniumwebdriver
emoteerrorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"settings-on-startup-page"}
  (Session info: chrome=75.0.3770.142)


Process finished with exit code 1

Answer 1

It looks like you just have to recursively traverse the elements with shadowRoots.

Try it like this:

def find_in_shadow_dom(css):
  return driver.execute_async_script("""
    const traverse = e => {
      let el
      if(el = e.querySelector('""" + css + """')){
        arguments[0](el)
      }
      [...e.querySelectorAll('*')].filter(e => e.shadowRoot).map(e => traverse(e.shadowRoot))
    }
    [...document.querySelectorAll('*')].filter(e => e.shadowRoot).map(e => traverse(e.shadowRoot))
    arguments[0](null)
  """)

input = find_in_shadow_dom('#searchInput')
input.send_keys('testing')

Edit: Notes for settings page..

control = find_in_shadow_dom('[label="Open a specific page or set of pages"]')
# don't forget to scroll into view
driver.execute_script("arguments[0].scrollIntoView(true)", control)
control.click()