I have been experimenting with HtmlUnit to extract scores from the BBC Sports website
Upon loading the page, it initially displays Premier League scores. To view scores for other leagues, one must use a dropdown menu and click the 'Update' button (presumably triggering an ajax request).
The following code successfully retrieves updated scores:
long startTime = System.currentTimeMillis();
String titleBar = getTitleBar(page);
HtmlOption option = ukGroupDropdown.getOptionByValue(competition);
ukGroupDropdown.setSelectedAttribute(option, true);
HtmlButton updateButton = (HtmlButton)page.getElementById("filter-nav-submit");
Thread.sleep(1000); // WHY???????
HtmlPage newPage = updateButton.click();
while(titleBar.equals(getTitleBar(newPage))) {
Thread.sleep(100);
}
System.out.println("Took " + (System.currentTimeMillis() - startTime));
return getMatches(newPage);
However, if I remove the Thread.sleep command just before clicking on the update button, the 'newPage' fails to update. What could be causing this behavior? And is there a more reliable approach (similar to the titleBar loop that continuously checks the text in the title bar like "Barclays Premier League," etc.)?