When conducting tests using Selenium and the headless Google Chrome browser in Java, the dynamic JS content fails to load

Currently, I am in the process of testing a website that I have developed as part of a Spring Boot project using Selenium. While I can successfully test basic functionalities such as page loading and title verification, I am encountering difficulties when it comes to examining the actual content of the page.

The specific section on my page is structured as follows:

<div id="main">
    <div id="div_1"></div>
    <div id="div_2"></div>
    <div id="div_3"></div>
    <div id="div_4"></div>
</div>

Furthermore, the content is loaded through a JavaScript script:

document.addEventListener("DOMContentLoaded", function(event) {
    populateDivs()
})

To configure the WebDriver, I have incorporated certain options derived from similar inquiries (other Selenium tests are running smoothly, suggesting no conflict with the options):

final ChromeOptions options = new ChromeOptions();
options.addArguments(
    "--headless",
    "--nogpu",
    "--disable-gpu",
    "--enable-javascript",
    "--no-sandbox",
    "--disable-extensions",
    "--disable-blink-features=AutomationControlled",
    "--disable-features=NetworkService,NetworkServiceInProcess",
     "start-maximized",
     "disable-infobars"
);

In addition, I have implemented a wait statement in my test scenario to provide ample time for the content to load:

final WebDriverWait wait = new WebDriverWait(driver, Duration.ofMinutes(1L));
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("div_1_content")));

Despite multiple attempts, the issue of the content not loading persists even after the designated waiting period. Uncertain about the next course of action - could there be an error in my usage of Selenium? Might the content loading mechanism in the JS require alteration?


For further context, here is how I am approaching the page loading process in Selenium:

// Initialize driver
final ChromeOptions options = new ChromeOptions();
options.setAcceptInsecureCerts(true);
options.addArguments("--headless=new");

final URL url = new URL(http://127.0.0.1:4444/wd/hub);
final RemoteWebDriver driver= new RemoteWebDriver(url, options);


// Load page
driver.navigate().to("https://127.0.0.1:81");


// Await dynamic content retrieval
final WebDriverWait wait = new WebDriverWait(driver, Duration.ofMinutes(1L));            
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("div_1_content")));

Answer №1

Ensure you take care of the following:

  • The headless property has been deprecated for a while now. It was --headless=chrome between Chrome versions 96 & 108, and from Chrome version 109 onwards it's --headless=new

Note: What are the two headless modes?

  • --nogpu and --disable-gpu arguments are no longer relevant in modern Chrome browsers. You can remove them.
  • Modern browsers require Javascript to function properly. Therefore, remove --enable-javascript.
  • --no-sandbox is not necessary unless encountering issues running tests as a root/admin user. You can drop this argument.
  • --disable-extensions and disable-infobars arguments are no longer needed. You can drop them as well.

Network Service

Networking services in Chrome are designed to be independent of Chrome's features.

  • This section only deals with network-related features, excluding file loading, data URLs, etc...
  • Only the basic networking functionalities like http, sockets, web sockets should be included here. Any higher-level features should be built on top of these.
  • Higher level web platform and browser features should be developed outside of this code. Certain exceptions apply when these features cannot operate without some hooks in the network service. In such cases, minimal code should be added. Examples include traffic shaping for devtools, CORB blocking, and CORS.
  • Every PostTask, thread hop, and process hop should be carefully considered as they introduce delays that could impact performance. NetworkContext and NetworkService are interfaces meant for the browser only, not to be sent to the renderer.

This Use Case

NetworkService and NetworkServiceInProcess are essential configurations that shouldn't be disabled. Remove the argument:

"--disable-features=NetworkService,NetworkServiceInProcess"

By implementing the above recommendations in your test case execution, you should be ready to proceed.


Short Version

Some helpful resources:


Update

According to your comment, if dynamic content still isn't loading even after using WebDriverWait, there could be several reasons:

  • The browser may be transferring control to WebDriver before the document.readyState reaches complete. In this case, add the following code block:

    new WebDriverWait(driver, Duration.ofMinutes(1L)).until(d -> ((JavascriptExecutor)d).executeScript("return document.readyState").equals("complete"));
    
  • WebDriver, being an out-of-process library instructing the browser, does not track the real-time DOM state thoroughly. To address race conditions between the browser and instructions, ensure return jQuery.active == 0:

    public void WaitForAjax2Complete() throws InterruptedException
    {
        while (true)
        {
            if ((Boolean) ((JavascriptExecutor)driver).executeScript("return jQuery.active == 0")){
                break;
            }
            Thread.sleep(100);
        }
    }
    
  • Check if desired elements are within any iframe

  • Verify if desired elements are within any #shadow-root

Answer №2

The code provided as an example had the following:

wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("div_1_content")));

...however, there was no element with the id of div_1_content in the HTML structure:

<div id="main">
    <div id="div_1"></div>
    <div id="div_2"></div>
    <div id="div_3"></div>
    <div id="div_4"></div>
</div>

Instead of using div_1_content as the id, it should be changed to div_1. Otherwise, the specified element will not be located.


It is advisable to use --headless=new rather than just --headless (), for headless mode to mimic regular Chrome loading behavior.

About

document.addEventListener("DOMContentLoaded", function(event)
- https://developer.mozilla.org/en-US/docs/Web/API/Window/DOMContentLoaded_event

The DOMContentLoaded event triggers when the HTML document has been fully parsed, and all deferred scripts ( and ) have been downloaded and executed. It does not wait for elements like images, subframes, or async scripts to finish loading... DOMContentLoaded does not wait for stylesheets either... The load event should be used instead to detect a completely loaded page.

This indicates that the event listener may fire prematurely unless you apply this approach:

addEventListener("load", (event) => {});


By simply changing the id from div_1_content to div_1, your problem might be resolved.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Are there any more efficient methods to retrieve an object from an arrow function in TypeScript?

Trying to retrieve an object from an arrow function is posing a challenge for me, especially with the following function f: myMethod(f: data => { return { someField: data.something }; }); I am aware that for simple types, you can condense the arrow ...

Update the second mouse click functionality using jQuery

I currently have a jQuery function integrated into my WordPress portal that manages the functionality of the menu and the display of subcategories to ensure proper mobile optimization. This code, which was created by the theme editor, handles these aspects ...

Implementing image rendering functionality in Vue.js

So here's what's going on: I developed a horror movie bucket list app for my bootcamp final project. The minimum viable product received positive feedback, and I obtained my certification. However, now that I've graduated, I want to enhance ...

How can you extract elements from a JSON array into separate variables based on a specific property value within each element?

In the following JSON array, each item has a category property that determines its grouping. I need to split this array into separate JSON arrays based on the category property of each item. The goal is to extract all items with the category set to person ...

React encountered an unexpected termination of JSON input during parsing

Upon running npm install, I encountered an error that is shown in the following link: https://i.stack.imgur.com/nVvps.jpg This issue has been causing trouble for me today and I'm unsure of the reason behind it. ...

Can the selected week be highlighted along with the corresponding week number in a row?

Can we display the number of the week in a row along with the selected week, either in the toolbar or somewhere else? I attempted to utilize ToolbarComponent, but it overrides the entire header. However, I would like to keep it as is and just add informat ...

Customize the font color in Material UI to make it uniquely yours

How can I customize the default Text Color in my Material UI Theme? Using primary, secondary, and error settings are effective const styles = { a: 'red', b: 'green', ... }; createMuiTheme({ palette: { primary: { ...

The versatility of reusable Backbone components

As I search for the best way to ensure the reusability of Backbone views, I have come across various solutions but am unsure which one would best meet my needs. My goal is to create multiple widgets populated with real-time data and I require a base compon ...

Discover the xpaths and collect the text and status of checkbox and radio buttons efficiently through a looping mechanism

I am facing a challenge on a page where I need to extract the text from an xpath along with the status of the radio button or checkbox associated with the text. The elements are within a class named "c-form-group", so I wrote a loop to iterate through to e ...

Using JavaScript in Django templates: Displaying errors with a JavaScript function

Update: I recently made changes to my code, and it now looks like this: <script> function updateFunction(calibrationId) { document.getElementById(calibrationId).innerHTML = "<ul><li>" + calibrationId + "</li>" ...

Utilizing an AngularJS custom filter twice

Experimenting with a custom Angular filter example found at: https://scotch.io/tutorials/building-custom-angularjs-filters#filters-that-actually-filter, my version looks like this: <!DOCTYPE html> <html> <script src="http://ajax.googleapi ...

Error: Trying to play the Snake Game with the P5.js Library, but getting the message "(X)

During my journey of coding a snake game by following a tutorial, I encountered an issue that the instructor had not faced before. Strangely enough, I am unable to identify the root cause of this problem. To aid in troubleshooting, I meticulously commente ...

Grid items in Material UI are not positioned next to each other

I am encountering an issue with the Grid component in material-ui. The grid items are currently stacking below each other instead of beside each other, and I'm unsure of what is causing this behavior. My intention is for the grid items to stack on top ...

Typescript - Error in Parsing: Expecting an expression

I am currently working with Vue and TypeScript and have encountered a problem. How can I resolve it? Here is the code snippet in question: private setTitle(systemConfig: any) { const systemConfigParse; let obj; systemConfigParse = JSON.parse(sy ...

I need help accessing data from a REST API and displaying it in a table

How can I call all the data in "md" and display it in a table? I've tried multiple values but none seem to work. Can someone guide me on how to use the "md" value to display in a table? <script src="https://ajax.googleapis.com/ajax/libs/jquery/ ...

The inputs for Node express middleware are unclear and lack definition

I am currently exploring Node.js as a potential replacement for my existing DOT NET API. I have created middleware to enforce basic non-role authorization in my application, but I am encountering compilation problems with the function inputs. Compilation ...

The automation script fails to launch on both Chrome and Firefox using Selenium and C# but interestingly, it works perfectly on Internet Explorer

Currently, I am testing a script in Visual Studio as a part of a project. The issue I am facing is that both Chrome and Firefox browsers are not running the script and eventually timeout. Surprisingly, Internet Explorer runs the script successfully without ...

What is the best way to switch back and forth between two div elements?

I've been attempting to switch between displaying div .cam1 and div .cam2, however, I can't seem to get it to work. Here's the code snippet in question: HTML: <div class="cam1"></div> <div class="cam2"></div> CS ...

Removing a CSS Class Using Tampermonkey: A Step-by-Step Guide

I'm completely new to CSS and javascript, so please bear with me. My goal is to remove the class disable-stream from each of the div elements located under the div with the class "stream-notifications". Below is an image for reference: Even though I ...

Obtaining undefined values for req and resolvedUrl in GetServerSideProps function

In my project, I am currently using next.js version ""next": "^12.1.4"" and node version ""@types/node": "^14.14.6". I have created a function called getServerSideProps with parameters req and resolvedUrl. When the ...