Is there a way to analyze the webpage content once the JavaScript has been fully loaded onto the site?

Is there a way to ensure that the webpage HTML is displayed accurately after it loads the javascript? I need this in order to get a proper representation of the tables.

I have experimented with various JavaScript libraries, but so far only one seems to be effective for me as the others appear to be outdated.

System.setProperty("webdriver.chrome.driver", "D:\\Download bestanden\\chromedriver_win32\\chromedriver.exe");

    ChromeOptions options = new ChromeOptions();
    //options.addArguments("headless");
    WebDriver driver = new ChromeDriver(options);

    driver.get("https://www.flashscore.com/");
    System.out.println(driver.getTitle());

    Document doc = Jsoup.parse(driver.getPageSource());
    System.out.println(doc.select("ul.submenu.hidden li a").text());
    driver.close();
    driver.quit();
    System.out.println("Completed");

When searching for lmenu_17, I am expecting results beyond just Superlinga from Albania as an href. In the inspector, First Division Albanian cup and Super Cup are also visible. I hope to see these results reflected in the search. Any assistance on this matter would be greatly appreciated!

javascriptjavaselenium-webdriverselenium-chromedriverjsoup

Answer №1

        ChromeDriver driver = new ChromeDriver();
        driver.Navigate().GoToUrl("https://www.flashscore.com/");

        // This code runs after the page has fully loaded and scrolls to the bottom.

        string href = driver.FindElementByXPath("//*[@id='lmenu_17']/ul/li[1]/a").GetAttribute("href"); // Albanian link
        //driver.Navigate().GoToUrl(href);

        foreach (var element in driver.FindElements(By.XPath("//*[@id='lc']/div[6]/ul/li/a")))
        {
            Console.WriteLine(element.GetAttribute("href"));
        }

        driver.FindElementByXPath("//*[@id='lc']/div[6]/ul/li[12]/a").Click();
        Thread.Sleep(1000);

        foreach (var element in driver.FindElements(By.XPath("//*[@id='lc']/div[9]/ul/li/a")))
        {
            Console.WriteLine(element.GetAttribute("href"));
        }

        Console.ReadKey();

You do not need to retrieve the page source.

Check out this picture of it working. I hope this helps! Let me know if you have any questions.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Adding an object to an array of objects with the useState hook in React - A simple guide

Hello, I am currently working on a project that involves creating a dynamic test page with questions, answers, and points/grades. The goal is to generate input fields based on the number of questions selected by the admin, along with an "Add Question" butt ...

javascriptjsonreactjsreact-hooks

How to delete sections of a grid using jquery

I don't mean to be blunt, but I have organized these into a grid: $('#grid .box:eq(2)').append('<h1>P</h1>'); $('#grid .box:eq(27)').append('<h1>P</h1>'); $('#grid .box:eq(3 ...

javascriptjquery

A comprehensive guide on publishing an application to Heroku using MongoDb/Mlab

I am encountering difficulties in deploying my Node.JS application to Heroku. I suspect that the issue lies with the database, but I cannot pinpoint the exact problem. I have attempted to switch from MongoDB Atlas to Mlab in hopes of resolving the issue, h ...

javascriptnode.jsmongodbherokumlab

Pass information from one .jsp file to a JavaScript function in another .jsp file

Here is the content of OneEmployee.jsp: <%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %> <form> <td> <c:out value="${emp.getEmpid()}"/></td> <td> <input type="text" id="fname" value="<c:out v ...

javascriptjqueryhtmlajaxjsp

The header() function triggers automatic page redirection instead of waiting for the form to be submitted

When I created a form that automatically sends an email upon submission, my goal was to direct the user to a thank you page after the email is sent. In my search for a solution, I stumbled upon the header() function in php and attempted to use it with the ...

javascriptphpformsemail

What are the steps for implementing this javascript code in an HTML document?

Recently, I've been seeking advice on how to address the issue of wanting a button click to automatically select the search box on my website. Suggestions have pointed towards using Javascript for this functionality, with the following code recommend ...

javascriptjqueryhtml

The ArrowHelper in THREE.js seems to be ignoring the natural rotation provided by Euler angles

Can someone help me with setting intrinsic rotations to a THREE.ArrowHelper in THREE.js? I'm trying to work with Tait-Bryan euler angles for 3D rotations. In the snippet below, I define a unit vector for the x-axis as THREE.Vector3(1, 0, 0). Then, I ...

javascriptthree.js

A new sub-schema has been created in the JSON schema specifically tailored to a certain property

I needed to create a JSON schema with 3 fields: num, key1, and key2. The fields num and key1 are required, while the field key2 is only mandatory if key1 has a value of 'abc'. Below is the schema structure I came up with: schema = { type: &ap ...

javascriptnode.jsjsonschema

Challenge with Angular date selection component

I am currently utilizing ngbdatepicker in my form. I want to save the date separately and retrieve it as shown below. <input class="form-control ngbfield custom-form-control" required [ngClass]="{'is-invalid':f.touched && birthDay.inv ...

javascriptangular

Optical Character Recognition (OCR) tool

Does anyone have recommendations for a JavaScript OCR API that can easily be accessed via JavaScript? I'm searching for an API similar to this: upload an image along with the desired result type (e.g. numbers, objects, text, mixed, etc) and receive t ...

javascriptjsonapiocr

What scenarios call for the utilization of setScriptTimeout?

When using Selenium WebDriver, there is a method called setScriptTimeout(time, unit). The description of this method states that it Specifies the time allowed for an asynchronous script to finish executing before an error is thrown. If the timeout is set ...

javascriptseleniumselenium-webdriver

Prevent overlapping of nodes and edges in a D3 force-directed layout

Check out this fascinating example at http://bl.ocks.org/mbostock/1747543: In the demonstration, Mike illustrates how to prevent nodes from colliding with each other in a graph. I'm curious if it's feasible to also prevent collisions between no ...

javascriptalgorithmd3.jsforce-layoutgraph-layout

I've managed to mess up my code again; local storage was working fine, and now it's not. I'm completely clueless

I attempted to store the best game times in local storage by creating an empty array called gBestScores. I compared the value of the last best score to the current one and decided whether to push it into the array or not. However, despite my efforts, the ...

javascripthtml

Errors can arise in Discord.js when encountering "undefined" before the first array object in an embed

I am currently in the process of creating a Discord bot command that allows users to construct their own city. I am encountering an issue with a list command that is supposed to display all the roads and places built within the city. However, each list kee ...

javascriptnode.jsarraysjsondiscord.js

Tips for smoothly scrolling through a web page using Selenium in Python

While trying to automate scrolling down a web page with Selenium, I came across a helpful post on how to scroll using selenium webdriver in Python. It worked well initially but later caused issues within my main code, especially when parsing Twitter accoun ...

pythonseleniumselenium-webdriverscroll

Is it possible to combine the use of 'res.sendFile' and 'res.json' in the same code?

At the moment, my Express app utilizes a controller to manage routing. When a specific route is accessed, I trigger pagesController.showPlayer which sends back my index.html. This is what the controller looks like: 'use strict'; var path = requ ...

javascriptnode.jsexpress

JavaScript - Retrieve all properties and methods of an object

Can an object created through a constructor function have its keys listed using the Object.keys() method? Let's consider an example with the following code: function Foo () {} Foo.prototype.bar = 'bar'; Foo.prototype.baz = 'baz'; ...

javascriptconstructor

Assertion Error Occurs with java.lang.AssertionError: anticipated [true] but encountered [false]

I encountered an error while trying to run this code. Initially, I had 'return true' and 'return false', but later changed them. Any idea what might be causing this issue? public static boolean elementIsPresent(MobileElement element) { ...

javaseleniumappium

What could be causing this JavaScript code to use 50% of the CPU and consume a significant amount of memory in the

I'm experiencing an issue with my banner rotator code. Here is the code snippet: function ban_rot() { //First preload images // counter var i = 0; // create object imageObj = new Image(); // set image list images = new A ...

javascriptjqueryhtmlsecuritymemory

Is it possible to dynamically populate a dependent select box using Jinja variables?

I am currently using Flask with Jinja2 templates, and I need assistance in creating dependent select boxes. How can I achieve this using Jinja2 or a combination of JavaScript and Jinja2 variables? For example: On the Python side: @app.route('/&apos ...

javascriptpythonjinja2