When conducting tests using Selenium and the headless Google Chrome browser in Java, the dynamic JS content fails to load

Currently, I am in the process of testing a website that I have developed as part of a Spring Boot project using Selenium. While I can successfully test basic functionalities such as page loading and title verification, I am encountering difficulties when it comes to examining the actual content of the page.

The specific section on my page is structured as follows:

<div id="main">
    <div id="div_1"></div>
    <div id="div_2"></div>
    <div id="div_3"></div>
    <div id="div_4"></div>
</div>

Furthermore, the content is loaded through a JavaScript script:

document.addEventListener("DOMContentLoaded", function(event) {
    populateDivs()
})

To configure the WebDriver, I have incorporated certain options derived from similar inquiries (other Selenium tests are running smoothly, suggesting no conflict with the options):

final ChromeOptions options = new ChromeOptions();
options.addArguments(
    "--headless",
    "--nogpu",
    "--disable-gpu",
    "--enable-javascript",
    "--no-sandbox",
    "--disable-extensions",
    "--disable-blink-features=AutomationControlled",
    "--disable-features=NetworkService,NetworkServiceInProcess",
     "start-maximized",
     "disable-infobars"
);

In addition, I have implemented a wait statement in my test scenario to provide ample time for the content to load:

final WebDriverWait wait = new WebDriverWait(driver, Duration.ofMinutes(1L));
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("div_1_content")));

Despite multiple attempts, the issue of the content not loading persists even after the designated waiting period. Uncertain about the next course of action - could there be an error in my usage of Selenium? Might the content loading mechanism in the JS require alteration?


For further context, here is how I am approaching the page loading process in Selenium:

// Initialize driver
final ChromeOptions options = new ChromeOptions();
options.setAcceptInsecureCerts(true);
options.addArguments("--headless=new");

final URL url = new URL(http://127.0.0.1:4444/wd/hub);
final RemoteWebDriver driver= new RemoteWebDriver(url, options);


// Load page
driver.navigate().to("https://127.0.0.1:81");


// Await dynamic content retrieval
final WebDriverWait wait = new WebDriverWait(driver, Duration.ofMinutes(1L));            
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("div_1_content")));

Answer №1

Ensure you take care of the following:

  • The headless property has been deprecated for a while now. It was --headless=chrome between Chrome versions 96 & 108, and from Chrome version 109 onwards it's --headless=new

Note: What are the two headless modes?

  • --nogpu and --disable-gpu arguments are no longer relevant in modern Chrome browsers. You can remove them.
  • Modern browsers require Javascript to function properly. Therefore, remove --enable-javascript.
  • --no-sandbox is not necessary unless encountering issues running tests as a root/admin user. You can drop this argument.
  • --disable-extensions and disable-infobars arguments are no longer needed. You can drop them as well.

Network Service

Networking services in Chrome are designed to be independent of Chrome's features.

  • This section only deals with network-related features, excluding file loading, data URLs, etc...
  • Only the basic networking functionalities like http, sockets, web sockets should be included here. Any higher-level features should be built on top of these.
  • Higher level web platform and browser features should be developed outside of this code. Certain exceptions apply when these features cannot operate without some hooks in the network service. In such cases, minimal code should be added. Examples include traffic shaping for devtools, CORB blocking, and CORS.
  • Every PostTask, thread hop, and process hop should be carefully considered as they introduce delays that could impact performance. NetworkContext and NetworkService are interfaces meant for the browser only, not to be sent to the renderer.

This Use Case

NetworkService and NetworkServiceInProcess are essential configurations that shouldn't be disabled. Remove the argument:

"--disable-features=NetworkService,NetworkServiceInProcess"

By implementing the above recommendations in your test case execution, you should be ready to proceed.


Short Version

Some helpful resources:


Update

According to your comment, if dynamic content still isn't loading even after using WebDriverWait, there could be several reasons:

  • The browser may be transferring control to WebDriver before the document.readyState reaches complete. In this case, add the following code block:

    new WebDriverWait(driver, Duration.ofMinutes(1L)).until(d -> ((JavascriptExecutor)d).executeScript("return document.readyState").equals("complete"));
    
  • WebDriver, being an out-of-process library instructing the browser, does not track the real-time DOM state thoroughly. To address race conditions between the browser and instructions, ensure return jQuery.active == 0:

    public void WaitForAjax2Complete() throws InterruptedException
    {
        while (true)
        {
            if ((Boolean) ((JavascriptExecutor)driver).executeScript("return jQuery.active == 0")){
                break;
            }
            Thread.sleep(100);
        }
    }
    
  • Check if desired elements are within any iframe

  • Verify if desired elements are within any #shadow-root

Answer №2

The code provided as an example had the following:

wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("div_1_content")));

...however, there was no element with the id of div_1_content in the HTML structure:

<div id="main">
    <div id="div_1"></div>
    <div id="div_2"></div>
    <div id="div_3"></div>
    <div id="div_4"></div>
</div>

Instead of using div_1_content as the id, it should be changed to div_1. Otherwise, the specified element will not be located.


It is advisable to use --headless=new rather than just --headless (), for headless mode to mimic regular Chrome loading behavior.

About

document.addEventListener("DOMContentLoaded", function(event)
- https://developer.mozilla.org/en-US/docs/Web/API/Window/DOMContentLoaded_event

The DOMContentLoaded event triggers when the HTML document has been fully parsed, and all deferred scripts ( and ) have been downloaded and executed. It does not wait for elements like images, subframes, or async scripts to finish loading... DOMContentLoaded does not wait for stylesheets either... The load event should be used instead to detect a completely loaded page.

This indicates that the event listener may fire prematurely unless you apply this approach:

addEventListener("load", (event) => {});


By simply changing the id from div_1_content to div_1, your problem might be resolved.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Trouble arises when selecting shapes in Angular 6 with FabricJS

Currently, I am experimenting with fabricjs and drawing different shapes. Interestingly, when I draw the shapes, they appear accurately on the canvas. However, when I try to switch to selection mode, I encounter an issue where I am unable to select the ind ...

Determine if a click event in Python Selenium was successful or not

I need to implement an if statement in my code snippet below that can determine whether a button has been clicked. If the button has been clicked, I want it to output "a"; otherwise, it should output "b". element = driver.find_element(By.XPATH, '//*[c ...

Exploring the world of AngularJS for the first time

I'm currently delving into the world of AngularJS and I encountered an issue with my first example. Why is it not working as expected? Here's a look at the HTML snippet: <html ng-app> <head> <title></title> <sc ...

What is the best way to update a nested property in an object?

Consider the following object: myObject{ name: '...', label: '...', menuSettings: { rightAlignment: true, colours: [...], }, } I want to change the value of rightAlignment to fals ...

The TimeoutException in Selenium is not a callable object

When I run the code below, I encounter an issue with a TimeoutException object not being callable. driver = webdriver.Chrome(options=options) try: WebDriverWait(driver, 2).until(EC.element_to_be_clickable((By.XPATH, "//div[@id='ui-datepicker- ...

I'm having trouble loading my Google Maps widget. Everything was functioning flawlessly until I attempted to hide it using the .hide function

After successfully implementing a Google Maps widget, I encountered an issue when trying to toggle its visibility using jQuery. Despite clicking on a div element to reveal the widget, the map fails to load inside the designated container. It seems as tho ...

Pytest is not able to locate any elements on the webpage, yet the same elements can be easily found using the console

When using CSS or XPath in the console (F12), I am able to locate the element on the page. $$("span.menu-item[data-vars-category-name='Most Popular']") However, when trying to find the same elements with Selenium (pytest) using: driver.find_el ...

When attempting to access endpoints from other computers, the connection to Express.js is being refused

I have set up an Express server and React for the frontend. The express server is running on port 5000 and React on port 3000. Additionally, I am using JWT tokens for authentication. When I login to the app from the computer it is running on, everything w ...

Customize Magento pop-up close function on click event

I developed a unique module with a Magento pop-up feature. I am looking to customize the close event for the pop-up. <div onclick="Windows.close(&quot;browser_window_updatecc&quot;, event)" id="browser_window_updatecc_close" class="magento_clos ...

Issue: Child Pages not found in Nuxt RoutingDescription: When navigating

Currently working on a Nuxt application that utilizes the WordPress REST API to fetch content. While my other routes are functioning correctly, I am facing challenges with nested pages. The structure I have implemented in my Nuxt app is as follows: pages ...

Launch a web application utilizing a node.js server hosted on Firebase

I am currently developing a web application using Vue. In this app, I have integrated "pusher" for real-time multi-user communication. I have set up a node.js server on port 5000 of a specific device within my local network. The app functions smoothly with ...

A versatile function that displays a loading icon on top of a specified div

Is it possible to pass an identifier for a particular div element to a function, where the function will display a loading image that covers the entire div, and then pass the same identifier to another function to hide the loading image? I am looking to cr ...

What are the benefits of incorporating an external API with Next.js API routes?

I've recently started using Next.js and I'm curious about the purpose of export default function handler since we can directly fetch data from APIs. In my HTML code, I have the following snippet. When the submit button is clicked, the sendformDa ...

Determine the class name of an element when it is clicked on

For various reasons, I am unable to use $('.class').click or on('click', function..) Therefore, I must utilize the onclick="" event from the html element. Is there a way to determine the class of the element where the onclick event oc ...

Using GraphQL in React to access a specific field

Here is the code snippet I am working with: interface MutationProps { username: string; Mutation: any; } export const UseCustomMutation: React.FC<MutationProps> | any = (username: any, Mutation: DocumentNode ) => { const [functi ...

Is it feasible to utilize a different class method without the need to pass a method parameter?

I am facing a scenario where I need to access a method from another class that returns something. This method in the other class requires a parameter. For example: public class Class1() { public Response postResponse(String getURL, DataTable dataTabl ...

Error in JSON due to the presence of an unexpected token within the

I am facing a challenge with my PHP code, where I take a list of filenames or empty strings and store them in an array. This array is then converted to JSON and saved in a database successfully. However, the problem arises when this array is also stored wi ...

Using Java robot in conjunction with Selenium 2.41.0 WebDriver for seamless automation

Recently I updated my selenium version from 2.37.1 to 2.40.0 and encountered a new problem. I have been using selenium webdriver to automate tests in a project. One of the pages contains a plupload that triggers an upload window, which I have been handlin ...

What is the best way to incorporate currency formatting into a table using sumtr and datatables?

I have a table where I am utilizing sumtr for the table footer, and displaying all the information within datatables. My requirement is to show all the values as currency. However, I am unable to modify the values after sumtr because it won't be able ...

The verification of form is not done using an if statement

There are two forms in my code named formA and comments that I need to submit via AJAX. However, the current if and else conditions do not correctly identify the form and always trigger the alert message hello3. Here is the JavaScript function: function ...