extracting web content with selenium and javascript integration

Struggling to extract JavaScript content from a website with selenium and geckodriver, but coming up empty-handed. Below is the snippet of JavaScript code:

<div _ngcontent-c2="" class="header-wrapper">
    <div _ngcontent-c2="" class="title">Suda Office</div>
    <div _ngcontent-c2="" class="update">Jul 05 11:07 AM</div>
</div>

<div _ngcontent-c2="">
    <div _ngcontent-c2="" class="item-row title-headers">
        <div _ngcontent-c2="" class="item-col head1">Route</div>
        <div _ngcontent-c2="" class="item-col head2">Destination</div>
        <div _ngcontent-c2="" class="item-col">
            <div _ngcontent-c2="" class="head3 head3-height">ETA</div>
        </div>
    </div>

    <div _ngcontent-c2="">
        <div _ngcontent-c2="" class="alternet-color">
            <div _ngcontent-c2="" class="item-row item-eta-row">
                <div _ngcontent-c2="" class="item-col eta-route">15 T</div>
                <div _ngcontent-c2="" class="item-col eta-destination">
                    <marquee _ngcontent-c2=""> Charbagh</marquee></div>
                <div _ngcontent-c2="" class="item-col eta-col">                
                    <div _ngcontent-c2="" class="eta-display-wrapper">
                        <div _ngcontent-c2="" class="display">
                            <span _ngcontent-c2="" class="space"></span>
                            <span _ngcontent-c2="" class="currentTiming">10 min</span>
                        </div>

                    </div>
                </div>
            </div>
        </div>
    </div>
</div>

Need to retrieve data from class="item-col eta-route", class="item-col eta-destination", and class="currentTiming" in the JavaScript above. Tried using the following code without success:

from selenium import webdriver
driver = webdriver.Firefox()
driver.get(url)
a = driver.find_elements_by_class_name("item-col eta-route")

However, a=[] as output. Even attempting 

d = driver.find_elements_by_class_name("currentTiming")
results in:

[<selenium.webdriver.firefox.webelement.FirefoxWebElement (session="6b1f2344-8e8a-4f48-a29a-54610179d62f", element="38e7ce58-ea66-4461-bee7-f81ac414595b")>]

Seeking advice on how to properly extract information from the page using selenium.

javascriptpython-2.7selenium

Answer №1

There may be an issue with the class name item-col eta-route in your HTML, which could have numerous similar classes.

Instead of that, you can try using this CSS selector: 

div[_ngcontent-c2][class='item-col eta-route']

To retrieve the value of 15 T.

Implementing a webdriver wait would enhance the stability of your script. 

wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[_ngcontent-c2][class='item-col eta-route']")))
print(element.text)

For extracting the value: 

marquee_text = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[_ngcontent-c2][class='item-col eta-destination'] marquee")))
print(marquee_text.text)

Make sure to import these: 

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Passing the socket.io instance to an express route

I am currently working on developing a nodejs application that will utilize various web APIs to search for information. The goal is to send the results to the client in real-time using socket.io, with jQuery handling the front end display of the data. Wha ...

javascriptnode.jsexpresswebsocketsocket.io

Encountering a 405 error code during a Selenium login session

Attempting to implement auto login on a website using Selenium with Python is resulting in a http: 405 error, displaying "Pardon our Interruption. something about your browser made us think you were a bot". How can this issue be avoided? I prefer to observ ...

pythonpython-3.xgoogle-chromeseleniumselenium-chromedriver

How to upload multiple files using AngularJS and Laravel without using a form tag

I am seeking a solution to upload multiple files without using the form tag, utilizing AngularJS with Laravel 5.2. The code snippet below is functional for uploading a single file but fails when attempting to upload multiple files. Here is the HTML Code: ...

javascriptphpangularjslaravellaravel-5

Showing JSON data as a table in an HTML format

After receiving the JSON String from the server as a response, which can be viewed here, I've implemented the following Jquery Code: function loadCategories() { $.ajax({ type: "POST", url: "/Services/ControllerService. ...

javascriptjqueryjsonajaxasp.net-ajax

What could be causing this minimal Angular - WebTorrent configuration to fail?

The setup appears to be quite straightforward. Check out the Webtorrent usage reference here This is how I have my setup: import WebTorrent from 'webtorrent'; @Component({ selector: 'app-root', standalone: true, template: `bla` ...

javascriptangulartorrentwebtorrent

Only the initial upload file is being passed through the Apollo Express server, with the remaining files missing in action

Currently, I am utilizing the apollo-express server with GraphQL. One issue I am encountering involves a mutation where I pass files from the front-end to the back-end. Strangely, I receive the file:{} object only for the first file - for the others, I rec ...

javascriptnode.jsgraphqlapolloapollo-server

Create automatic transcripts for videos, including subtitles and captions

Are there any tools or plugins available that can automatically create a transcript of a video for website playback? For example, generating captions and subtitles in the English language. ...

javascripthtmltranscription

Set up an array data by extracting values from an array prop within a Vue component

Within my Vue component, I am dealing with an array prop called selectedSuppliers that consists of objects. My goal is to set up a data property named suppliers and initialize it with the values from selectedSuppliers. However, I do not want any modificati ...

javascriptvue.js

Is there a way to eliminate a specific input box using the jquery remove function?

I've been working on some code and have run into an issue. Currently, the remove function only works on one input box that is created with the append function. I would like it to be able to work on every input box generated through the append function ...

javascriptjqueryjquery-ui

When using the combination of Cucumber/Capybara with Angular, the test successfully passes with the Selenium driver but does not work with

I've been attempting to conduct feature tests on an Angular application, but I'm experiencing failures when using the poltergeist driver. It appears that the issue stems from the data-binding syntax being interpreted literally. For example, in th ...

angularjsseleniumcucumbercapybarapoltergeist

Exploring ways to cycle through a select dropdown in React by utilizing properties sent from the Parent Component

I'm having trouble displaying the props from a parent component in a modal, specifically in a select dropdown. How can I make it so that the dropdown dynamically shows the values from the props instead of the hardcoded 'Agent' value? What am ...

javascriptreactjsreact-hooksrxjsreact-props

Exploring the Power of Jasmine Testing with Ternary Conditions

Imagine a scenario where we are working with the following snippet of JavaScript code. object = _.isUndefined(object) ? '' : aDifferentObject.property; Is it possible to write a Jasmine test that covers both scenarios? Do we need to use two se ...

javascripttestingjasmineistanbul

Switching Perspective on Live ExpressJS Path -- Node.JS

I previously set up an express route with a template. Here's the code: app.get('/route', function route(req, res) { db.get('db_ID', function compileAndRender(err, doc) { var stream = mu.compileAndRender('theme-file.e ...

javascriptnode.jstemplatesexpressurl-routing

Removing a row from a table in a React component

I have incorporated a Table component from Material UI into my project to display data fetched from an external API. The table dynamically updates its rows based on events received through a web socket connection. One specific requirement I have is that wh ...

javascriptreactjshtml-tablematerial-ui

Changing the entire content of a webpage from the server using AJAX

I am looking to update the entire page content with the click of a button, transitioning from Words.html to SelectNumber.html This snippet is from Words.html <html> <head> <meta charset="UTF-8"> <title>Number Game< ...

javascripthtmljquerycssajax

Node.js server experiencing delays due to V8 processing constraints

REVISED I am currently running a nodeJS http server designed to handle uploads from multiple clients and process them separately. However, I have encountered an issue where the first request seems to block any subsequent requests until the first one is co ...

javascriptnode.jshttpasynchronousv8

At what point should I end the session with the webdriver?

My web scraping process involves using both Scrapy and Selenium. When I run my spider, I notice the instance of my webdriver in htop. I am unsure about when exactly I should close the webdriver in my code. Would it be best to do so after processing each l ...

pythonseleniumselenium-webdriverscrapy

Having trouble displaying values from nested JSON in a datatable

Response from server : ["{\"CLIENT\":[{\"tranche\":\"1-4\",\"prix\":\"65.96\",\"currency\":\"E\"}],\"DISTRIBUTEUR\":[{\"tranche\":\"1-4\",\"prix\ ...

javascriptjqueryangularjsjson

Most effective method for displaying modals in React

I've recently started learning ReactJS and I'm exploring the use of modal windows. However, I'm uncertain about the most appropriate approach to take. Currently, I am implementing modals using callbacks. Reactstrap Modal Dialog Component: ...

javascriptreactjsmodal-dialog

Is the JSON data not matching the file's content during validation?

After thorough testing, my JSON data appears to be functioning correctly with regular content. Here is a sample of the working JSON: Working Json { "language": "XYZ", "content": { "GEN": "this is test& ...

javascriptjson