python - WebDriverWait not working as expected

Question

I am working with selenium to scrape some data.

There is button on the page that I am clicking say "custom_cols". This button opens up a window for me where I can select my columns.

This new window sometimes takes some time to open (around 5 seconds). So to handle this I have used

WebDriverWait 

with delay as 20 seconds. But some times it fails to select find elements on new window, even if the element is visible. This happens only once in ten times for rest of time it works properly.

I have used same function(WebDriverWait) on other places also and it is works as expected. I mean it waits till the elements gets visible and then clicks it at the moment it finds it.

My question is why elements on new window is not visible even though I am waiting for element to get visible. To add here I have tried to increase delay time but still I get that error once in a while.

My code is here

def wait_for_elem_xpath(self, delay = None, xpath = ""):
    if delay is None:
        delay = self.delay

    try:
        myElem = WebDriverWait(self.browser, delay).until(EC.presence_of_element_located((By.XPATH , xpath)))
    except TimeoutException:
        print ("xpath: Loading took too much time!")
    return myElem
select_all_performance = '//*[@id="mks"]/body/div[7]/div[2]/div/div/div/div/div[2]/div/div[2]/div[2]/div/div[1]/div[1]/section/header/div'
self.wait_for_elem_xpath(xpath = select_all_performance).click()

Answer 1

Once you wait for the element and moving forward as you are trying to invoke click() method instead of using presence_of_element_located() method you need to use element_to_be_clickable() as follows :

try:
    myElem = WebDriverWait(self.browser, delay).until(EC.element_to_be_clickable((By.XPATH , xpath)))

---

Update

As per your counter question in the comments here are the details of the three methods :

presence_of_element_located

presence_of_element_located(locator) is defined as follows :

class selenium.webdriver.support.expected_conditions.presence_of_element_located(locator)

Parameter : locator - used to find the element returns the WebElement once it is located

Description : An expectation for checking that an element is present on the DOM of a page. This does not necessarily mean that the element is visible or interactable (i.e. clickable). 

visibility_of_element_located

visibility_of_element_located(locator) is defined as follows :

class selenium.webdriver.support.expected_conditions.visibility_of_element_located(locator)

Parameter : locator -  used to find the element returns the WebElement once it is located and visible

Description : An expectation for checking that an element is present on the DOM of a page and visible. Visibility means that the element is not only displayed but also has a height and width that is greater than 0.

element_to_be_clickable

element_to_be_clickable(locator) is defined as follows :

class selenium.webdriver.support.expected_conditions.element_to_be_clickable(locator)

Parameter : locator - used to find the element returns the WebElement once it is visible, enabled and interactable (i.e. clickable).

Description : An Expectation for checking an element is visible, enabled and interactable such that you can click it.