Currently, I am faced with a challenge in extracting data from a JavaScript-based website using htmlunit. My approach involves calling the site through webClient.getPage, but this method is triggering an exception.
public static void main(String[] args) throws Exception {
final WebClient webClient = new WebClient();
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.setJavaScriptTimeout(10000);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
webClient.getOptions().setTimeout(10000);
final HtmlPage page1 = webClient.getPage("[my-site-url-here]");
}
The exception I am encountering is as follows:
======= EXCEPTION START ========
Exception class=[net.sourceforge.htmlunit.corejs.javascript.JavaScriptException]
com.gargoylesoftware.htmlunit.ScriptException: Error: [MobX] Proxy not available (https://[my-site-here]/static/js/8.107d4023.chunk.js#1)
Despite applying various options to handle JavaScript errors in my code snippet, including disabling proxies and utilizing cloudflare for the website, I still face this issue. Any insights on how to resolve this would be highly appreciated. Thank you.