java - Selenium: How to avoid StaleElementReferenceException when looping through a set of elements?

Question

I have a page that contains a bunch of tables. I loop through the tables in the outer loop and then loop through each row in the table in the inner loop. It all works fine. But some of the pages have a Next button. When I add code to click that after completing the page, then I start getting StaleElementReferenceException while looping through the rows of a table.

Here is the code:

WebDriverWait wait1 = new WebDriverWait(driver, 10000);
WebElement maxPage = null;
WebElement auctionsWaitingDiv = driver.findElement(By.cssSelector("div[class='Head_W']"));
if (auctionsWaitingDiv.isDisplayed() == false) return properties;

try {
    maxPage = wait1.until(ExpectedConditions.visibilityOfElementLocated(By.id("maxWA")));
} catch (TimeoutException ex) {
    return properties;
}

Integer maxPageNo = 1;
if (!maxPage.getText().isEmpty()) 
    maxPageNo = Integer.parseInt(maxPage.getText());
for (int i = 1; i <= maxPageNo; i++) {
    driver.findElement(By.cssSelector("div[id='Area_W']"));    //only look at Auctions Waiting section
    WebDriverWait wait2 = new WebDriverWait(driver, 10000);
    List<WebElement> tables = null;
    try {
        tables = wait2.until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.cssSelector("table[class='ad_tab']")));
    } catch (TimeoutException ex) {
        System.out.println("table not found in allotted time");
        return properties;
    } catch (StaleElementReferenceException ex) {
        System.out.println("returning due to StaleElementReferenceException");
        return properties;

    }
    for (WebElement table: tables) {  
        List<String> propAttributes = new ArrayList<>();

        // StaleElementReferenceException: The element reference of
        // <table class="ad_tab"> is stale; either the element is no
        // longer attached to the DOM, it is not in the current
        // frame context, or the document has been refreshed
        List<WebElement> rows = table.findElements(By.cssSelector("tr"));

        String parcelLink = "";
        for (WebElement row : rows) { 
            WebElement key = row.findElement(By.cssSelector("th"));
            WebElement val = row.findElement(By.cssSelector("td"));
            String keyVal = key.getText() + val.getText();
            propAttributes.add(keyVal);
            if (key.getText().equals("Parcel ID:")) {
                WebElement a = val.findElement(By.cssSelector("a"));
                parcelLink = a.getAttribute("href");
            }
        }
    }
    driver.findElement(By.xpath(".//*[@class='PageRight']")).click();  //click the "Next" button
}

What I don't understand is why the stale element is happening at all? The page is not changing during the loop and I've waited until all elements have been fetched. How to avoid the StaleElementReferenceException?

Edit: The last stack trace shows it is happening in this line:

List<WebElement> rows = table.findElements(By.cssSelector("tr"));

and the error message above it shows:

SEVERE: null

org.openqa.selenium.StaleElementReferenceException: The element reference of <table class="ad_tab"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed

Answer 1

The StaleElementReferenceException is thrown whenever you want to access an element reference which is not available anymore. This happens when the element is no longer attached to the DOM or if the page was updated.

The solution for this is just searching for the element again whenever this happens. You could adapt all your tests or page objects. Or you write your own RobustWebDriver and RobustWebElement which refreshes the element if a SERE is thrown.

RobustWebDriver:

public class RobustWebDriver implements WebDriver {

    private WebDriver originalWebDriver;

    public RobustWebDriver(WebDriver webDriver) {
        this.originalWebDriver = webDriver;
    }

    @Override
    public void get(String url) {
        this.originalWebDriver.get(url);
    }

    @Override
    public String getCurrentUrl() {
        return this.originalWebDriver.getCurrentUrl();
    }

    @Override
    public String getTitle() {
        return this.originalWebDriver.getTitle();
    }

    @Override
    public List<WebElement> findElements(By by) {
        List<WebElement> elements = new ArrayList<>();
        for (WebElement element : this.originalWebDriver.findElements(by)) {
            elements.add(new RobustWebElement(element, by, this));
        }
        return elements;
    }

    @Override
    public WebElement findElement(By by) {
        return new RobustWebElement(this.originalWebDriver.findElement(by), by, this);
    }

    @Override
    public String getPageSource() {
        return this.originalWebDriver.getPageSource();
    }

    @Override
    public void close() {
        this.originalWebDriver.close();

    }

    @Override
    public void quit() {
        this.originalWebDriver.quit();
    }

    @Override
    public Set<String> getWindowHandles() {
        return this.originalWebDriver.getWindowHandles();
    }

    @Override
    public String getWindowHandle() {
        return this.originalWebDriver.getWindowHandle();
    }

    @Override
    public TargetLocator switchTo() {
        return this.originalWebDriver.switchTo();
    }

    @Override
    public Navigation navigate() {
        return this.originalWebDriver.navigate();
    }

    @Override
    public Options manage() {
        return this.originalWebDriver.manage();
    }
}

RobustWebElement:

public class RobustWebElement implements WebElement {

    private WebElement originalElement;
    private RobustWebDriver driver;
    private By by;
    private static final int MAX_RETRIES = 10;

    public RobustWebElement(WebElement element, By by, RobustWebDriver driver) {
        this.originalElement = element;
        this.by = by;
        this.driver = driver;
    }

    @Override
    public void click() {
        int retries = 0;
        while (retries < MAX_RETRIES) {
            try {
                this.originalElement.click();
                return;
            } catch (StaleElementReferenceException ex) {
                refreshElement();
            }
            retries++;
        }
        throw new StaleElementReferenceException(
                String.format("Element is still stale after %s retries.", MAX_RETRIES));
    }

    @Override
    public void sendKeys(CharSequence... keysToSend) {
        int retries = 0;
        while (retries < MAX_RETRIES) {
            try {
                this.originalElement.sendKeys(keysToSend);
                return;
            } catch (StaleElementReferenceException ex) {
                refreshElement();
            }
            retries++;
        }
        throw new StaleElementReferenceException(
                String.format("Element is still stale after %s retries.", MAX_RETRIES));
    }

    // TODO add other unimplemented methods with similar logic.

    private void refreshElement() {
        this.originalElement = driver.findElement(by);
    }

And then you just need to wrap your WebDriver into the RobustWebDriver and you are ready to go:

WebDriver driver = new RobustWebDriver(new ChromeDriver());

EDIT:

Of course you need to take care of scrolling up and down by yourself.