Extracting data from a webpage containing a table generated through javascript by utilizing Selenium Webdriver

the source html of the page I'm attempting to extract data from

I am currently using Selenium Webdriver to scrape a web table that is generated by certain JavaScripts.

driver.get("http://xxxxx:xxxxxxxx@xxxxxx-
xxxxxx.grid.xxxxxx.com/Windchill/app/#ptc1/comp/queue.table");
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
List<WebElement> k=driver.findElements(By.xpath("//*[@id='queue.table']"));
System.out.println(k.size());
System.out.println(k.get(0).getText());

k.size() returns 1 and when I try to get text it only displays a few entries from the table

The actual table and entries - there are a total of 135 rows

After running the script, the output looks like this:


Queue Management
Loading...

Name                Type        Status      Enabled     Group       Total Entries   Waiting Entries    Severe/Failed Entries   
DeleteCompletedWorkItemsQueue     Process     Started     Enabled     Default         0                   0                      0                   
DeliveryStatusOnStartup            Process     Started     Enabled     Default         0                   0                      0                   
DTODeliverablesQueue               Process     Started     Enabled     Default         0                   0                      0                   
DTOOffPeakQueue                    Process     Started     Enabled     Default         0                   0                      0             

Loading.........

Only 25 entries from the table are displayed, while the remaining are missing. I'm puzzled by why "Loading....." is still present.

javascriptjavaseleniumxpathselenium-webdriver

Answer №1

It seems that using 

List<WebElement> k=driver.findElements(By.xpath("//*[@id='queue.table']"));
to create a list may result in including unnecessary items. A more efficient approach could be to target the nodes within <td> tags which hold the relevant values and store them in the list. Then, we can loop through the list and extract the text using either the getText() method or the getAttribute() method as shown below:

driver.get("http://xxxxx:<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="96eeeeeeeeeeeeeeeed6eeeeeeeeeeeebbeeeeeeeeeeeeb8f1e4fff2b8eeeeeeeeeeeeb8f5f9fb">[email protected]</a>/Windchill/app/#ptc1/comp/queue.table");
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
List<WebElement> k = driver.findElements(By.xpath("//*[@id='queue.table']//tr"));
System.out.println(k.size());
for (WebElement my_element : k) {
    String innerhtml = my_element.getAttribute("innerHTML");
    System.out.println("Value from Table is: " + innerhtml); 
}

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Having trouble retrieving the data from the second child object in an object returned by an API call in a

Having trouble accessing the second child object of an object generated by an API response. The weather API in question can be found at . Refer to Display.js for a detailed explanation. App.js import React,{useState} from 'react'; import Navbar ...

javascriptreactjsjsonjsxresponse

Retrieving selective data with mongoose

Attempting a seemingly simple task, yet somehow failing. The goal is to return all documents from a get request on the server, but only with specific fields populated. Here's an outline of my schema: var clientSchema = new Schema({ name:{ ...

javascriptnode.jsmongodbmongoosemongoose-schema

Using Django to Display a Line Chart with Data from a Dictionary in a Template

I have a dictionary that was generated in the views.py file. The structure of the dictionary is as follows, but it contains data for an unspecified number of enterprises: { enterprise1: {'Year': ['2014/2015', '2016/2017', &ap ...

javascriptpythonjsondjango

The installation of package.json is unsuccessful, as it will only proceed with an empty name and version

Having trouble installing the package.json file after updating to the latest version of Node on my Windows PC. When trying to run npm, an error was thrown. Any help would be greatly appreciated. Command Prompt C:\Users\Felix\Desktop\ ...

javascriptnode.js

You can eliminate the display flex property from the MuiCollapse-wrapper

Having trouble removing the display flex property from the MuiCollapse-wrapper, I did some research and found the rule name for the wrapper in this link https://material-ui.com/api/collapse/ I have been unsuccessful in overwriting the class name wrapper t ...

javascripthtmlcssreactjsmaterial-ui

Error: Parameter 'options' received multiple values in the initialization of WebDriver

An issue has arisen and it is as follows: TypeError: WebDriver.__init__() got multiple values for argument 'options' ` The piece of code causing the problem is as shown below: chrome_options = Options() chrome_options.add_argument('--headl ...

pythonpython-3.xselenium-webdriverselenium-chromedriverwebdriver

Creating watercolor effects using JavaScript

Coffee has been the only thing on my mind lately. I've been contemplating how to replicate the essence of coffee in JavaScript. Imagine being able to fill a shape on the screen as if it were infused with the rich aroma of coffee. Is there a JavaScript ...

javascriptgraphics

Looking to add a bulleted list to an HTML document with PHP?

I have integrated a PHP script within an HTML list element that enables users to edit the HTML element directly from the webpage if they enter the correct password via the variable ?pw = "password". Now, I am looking to allow users to add new list items ...

phpjavascripthtml

Step-by-step guide on incorporating a new JSON object into an array to display its elements as components on a webpage

Could I adjust the state of an array by incorporating values from a data.js file as a starting point? The process involves executing the setAllThingsArray function to add a new element, determined by the setThingsArray function based on the previous state ...

javascriptreactjs

Using scripts to populate a Google Sheet with data from Postman

I'm currently working on populating a Google sheet using apps script. The task involves receiving a JSON object from Postman via POST request. The structure of the object is as follows: { "email":"<a href="/cdn-cgi/l/email-prote ...

javascriptjsongoogle-apps-scriptpostman

java.lang.IllegalArgumentException: The string "json" is missing a forward slash ("/")

Whenever I attempt to send data to the server using the POST method, I encounter this error on the server side: java.lang.IllegalArgumentException: "json" does not contain '/'. Initially, I suspected that the date format from the jQuery UI Date ...

javajqueryajaxjsonspring

Special effects for the images动画效果。

Is there a way to add animation effects to images in the about section using this code: <div id="about" class="row section bgimg3"> <div class="col-sm-8"> <h2 style="color:black;">Want to Know More About me?</h2> ...

javascriptjqueryhtml

Column Reordering in DataTable Causes Data to be Mapped Incorrectly

I have created a JSBin where you can view the code. However, due to CORS policy restrictions, the ajax URL I used is not functioning properly. The output generated shows that the data is mapped incorrectly in the columns. Can someone please help me figure ...

javascripthtmljquerydatatables

I am unable to achieve negative X degree rotation of the image while using mousemove to rotate it

I need assistance with moving a picture in 3D. I want the offsetX of the mouse to be positive when it's past half of the picture, and negative otherwise. How can I achieve this effect for rotation degrees? This is what I have tried: $('#img ...

javascriptjquery

Encountered a React error stating: `TypeError: this.state.projects.map is not a

export default class Timeline extends Component{ state = { projects : [], }; async componentDidMount(){ const response = await api.get("/projects"); this.setState({projects: response.data}); } render(){ return ( <div className ...

javascriptreactjs

Have you ever wondered why MEAN.js applications use the #! symbol at the start of their URLs

Embarking on my first MEAN application build using MEAN.JS, I was intrigued by the structure of how mean.js organizes the application, particularly why URLs begin with /#!/. For instance, for the login page, I envisioned: http://example.com/login instea ...

javascriptangularjsurlmean-stackmeanjs

Using v-bind:class to assign an object value

I have a Vue component that displays an image grid. I want users to be able to select an image by clicking on it. When an image is selected, its style should change, and if clicked again, it should return to its unselected state. I am trying to bind the & ...

javascriptdata-bindingvue.js

Printing Elements of a Multidimensional Array in Java

I'm a little confused about why these numbers are being printed out. I thought it would just print 3, 2, 1. However, the output is: 3 0 0 0 2 0 0 0 1 Thank you so much for your assistance! :) public static void main(String[] args) { int ...

javaarraysmultidimensional-arrayprintingout

Troubleshooting why the Angular innerHTML function is failing to render the specified

I'm encountering this problem where I am receiving a string const str = '<p>Please ensure Process Model diagram represents Functions adequately (boxes that represent an activity or group of activities that produce an outcome):</p>< ...

javascriptangulartypescriptangular-sanitizer

Tips for implementing absolute positioning with dynamically loaded images using ajax

Currently, I am faced with a challenge in setting up absolute position for all images that are dynamically generated through ajax. Below are the snippets of my code: for loop... var imagediv = document.getElementById('image_layout'); //gene ...

javascriptjquery