Currently, I am tasked with a project that involves taking user input, scraping data from TripAdvisor.it based on that input, storing it in a database, and using machine learning to rearrange the data for future requests. This process aims to offer users choices tailored to their past behavior.
I am facing challenges specifically in the web-scraping phase as I encounter an error stating "Unknown Error: Element is not clickable at point (x,y);..." while trying to click on attractions on the website to save relevant data to my database.
In attempting to resolve this issue, I have explored various solutions such as implementing explicit waits and methods to ensure JavaScript has loaded completely on the webpage. However, the problem persists despite these efforts.
- Using Thread.sleep(1000) to force a wait
- Implementing a script to check if JavaScript has fully loaded
- Applying ExpectedCondition .visibilityOf(WebElement elem) instead of .elementToBeClickable(WebElement elem)
The code snippet causing the exception is as follows:
...
By locator = By.xpath("//a[contains(@class, 'attractions-attraction-overview-main-TopPOIs__name')]");
int numberOfElementsFound = getNumberOfElementsFound(driver, locator);
for (int pos = 0; pos < numberOfElementsFound; pos++) {
WebDriverWait wait = new WebDriverWait(driver, 10);
wait.until(ExpectedConditions.elementToBeClickable(getElementWithIndex(driver, locator, pos))).click();
...
}
The full error message reads:
org.openqa.selenium.WebDriverException: unknown error: Element ... is not clickable at point (360, 14). Other element would receive the click: ...
If you want to view the actual webpage causing the issue, please visit this link.
Ultimately, I suspect that incomplete JavaScript loading is the root cause of this problem. This assumption is based on my observation that the specific tag "ui_column wrap_column responsive_inline_hidden label_column" cannot be located on the page.
What steps should I take to rectify this error?