Extracting content from a page that requires JavaScript to be rendered

I need to extract data from a dynamically rendered JavaScript page using Selenium web driver in Python3. I've tried various drivers like Firefox, Chromedriver, and PhantomJS, but I always end up with the script rather than the DOM element.

Below is a snippet of the code I'm using:

url = 'https://www.google.com/flights/explore/#explore;f=BDO;t=r-Asia-0x88d9b427c383bc81%253A0xb947211a2643e5ac;li=0;lx=2;d=2018-01-09'
driver = webdriver.Chrome("/var/chromedriver/chromedriver")
driver.implicitly_wait(20)
driver.get(url)

print(driver.page_source)

Am I overlooking something here?

javascriptpython-3.xseleniumselenium-webdriver

Answer №1

Your code block seems free of any issues. I tested your script with the following code:

from selenium import webdriver

url = 'https://www.google.com/flights/explore/#explore;f=BDO;t=r-Asia-0x88d9b427c383bc81%253A0xb947211a2643e5ac;li=0;lx=2;d=2018-01-09'
driver = webdriver.Chrome()
driver.get(url)
print(driver.page_source)

The Console Output I received was as follows:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-US">

<head>
  <meta http-equiv="content-type" content="text/html; charset=UTF-8" />
  <meta name="deals::gwt:property" content="baseUrl=/flights/explore//static/" />
  <title>Explore flights</title>
  <meta name="description" content="Explore flights" />
  <script src="https://apis.google.com/_/scs/abc-static/_/js/k=gapi.gapi.en.yoTdpQipo6s.O/m=gapi_iframes,googleapis_client,plusone/rt=j/sv=1/d=1/ed=1/am=AAE/rs=AHpOoo9_VhuRoUovwpPPf5LqLZd-dmCnxw/cb=gapi.loaded_0" async=""></script>
  <script language="javascript" type="text/javascript">
    var __JS_ILT__ = new Date();
    .
    .
    . <
    /div></div > < div aria - hidden = "true"
    style = "display: none;" > < div class = "CTPFVNB-l-j CTPFVNB-l-h" > Displayed currencies may differ from the currencies used to purchase flights.– < a href = "https://www.google.com/intl/en/googlefinance/disclaimer/"
    class = "CTPFVNB-l-k" > Disclaimer < /a></div > < /div><div aria-hidden="true" style="display: none;"><div class="CTPFVNB-l-j CTPFVNB-l-h">Showing licensed rail data. – <a href="https:/ / www.google.com / intl / en / help / legalnotices_maps.html " class="
    CTPFVNB - l - k ">Legal Notice</a></div></div><div class="
    CTPFVNB - l - i "><a class="
    CTPFVNB - l - k CTPFVNB - l - j " href="
    https: //www.google.com/intl/en/policies/">Privacy &amp; Terms</a><a class="CTPFVNB-l-k CTPFVNB-l-j" href="https://support.google.com/flights/?hl=en">Help Center</a></div></div></div><iframe id="deals" tabindex="-1" style="position: absolute; width: 0px; height: 0px; border: none; left: -1000px; top: -1000px;">
</iframe><input type="text" id="_bgInput" style="display:none;" /></body></html>

It is important to note that towards the end of the page_source, there is an iframe. In order to locate the DOM element you are seeking, you must switch to this iframe.

Answer №2

try using Helium as a wrapper for Selenium

# pip install helium
import helium, time
target_url = "https://www.vbiz.in/nseoptionchain.html"
web_browser = helium.start_chrome(target_url, headless=True)
pause_time = 5
time.sleep(pause_time)
html_content = web_browser.page_source
web_browser.close()

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Three.js Pin Placement for Clothing

I am in need of assistance! I am currently working on a simulation involving a cloth that is attached to four corners. I am attempting to reposition the pins at coordinates 0, 10, 88, 98 within a 10x10 array. My goal is to place each pin at a different pos ...

javascriptthree.js

What is the best way to retrieve JSON data in ReactJS through ExpressJS?

Exploring the realms of reactjs and expressjs, I am seeking guidance on how to extract data from reactjs and store it in a variable. My current achievement includes successfully executing res.send to display the data. app.get('*', (req, res) =& ...

javascriptnode.jsreactjsexpress

Dealing with errors in node.js

Node.js asynchronous functions typically have a callback, with some like fs.writeFile passing an err argument. fs.writeFile('message.txt', 'Hello Node', function (err) { if (err) throw err; console.log('It\'s saved!& ...

javascriptnode.jserror-handling

Activating the Mobile Menu Function when the Window is Resized

I've developed a JavaScript function that triggers on window resize. When switching between landscape and portrait orientation on mobile devices or tablets, the window dimensions change. This functionality is also useful for browser testing on desktop ...

javascriptjquery

Dynamic Data Causes Highcharts Date Formatting to Adapt

I wanted to create a line graph using Highcharts with the Date Format on x-axis set to "%b %e". For example, I expected 06/27/2014 to be displayed as Jun 17. Despite setting the date-format correctly, Highcharts seems to automatically change it when rende ...

javascriptjqueryhighcharts

Can you please explain how to indicate a modification in a JSON object with Polymer, transferring information from Javascript, and subsequently displaying child elements?

Currently, I am creating a JSON file that contains a random assortment of X's and O's. My goal is to display these elements in a grid format using a custom Polymer element. Initially, everything works fine as I can see a new grid generated each t ...

javascriptjsonobjectpolymerrerender

Modify content on a link using AngularJS

Currently, my setup looks like this: HTML Code <div ng-app="home" ng-controller="customersCtrl"> <div ng-bind-html="home" id="content">{{content}}</div> </div> JS Code angular.module('home', []).controller('c ...

javascriptangularjs

Live calculation feature for dropdown menus

I have a form where I need to calculate the total result after submission. The package and event ID should be added together, and the guest field should always be calculated (guest value * 5). The final result should be displayed in <div id="result"> ...

javascriptphpajaxcalculator

Encountering errors while attempting to render MUI components in jsdom using react testing library within a mocha test

Due to the lack of maintenance and support for React 18+ in enzyme, I am faced with the task of migrating over 1750 unit tests to work with react-testing-library and global-jsdom. This migration is necessary so that our application can continue running on ...

javascriptreactjsmaterial-uimocha.jsreact-testing-library

Is there a way to automatically redirect the main html page to a different html page upon logging in?

I have created a main page in HTML with a login box that displays a message saying "Login successful" or "Login failed" based on whether the password entered is 8 characters or more. The validation function for this works correctly, but after successfully ...

javascripthtml

What is preventing my application (docker) from connecting to the database?

Encountering a frustrating issue here: I've developed a NodeJS application with a server listening on port number 3000. The app includes simple operations such as post, put, and read, which interact with a database running in a Docker Container on por ...

javascriptnode.jspostgresqldocker

Assign specific CSS classes to elements based on the v-model value in Vue.js

Looking to create a dynamic table using Vue and trying to add a class to a row when a checkbox is checked. Each checkbox is associated with a specific value in an object of objects. View the code on Codepen. <tbody> <tr v-for="row in filter ...

javascriptvue.js

Utilizing the ng-if directive to choose the second element within an iteration of an ng-repeat loop

I am currently working on a project where I need to organize and group content by day. While the grouping function is working fine, I am facing an issue with treating the first two items in the loop differently from the rest. I have experimented with using ...

javascriptangularjs

Having trouble retrieving console logs with the Firefox webdriver

I am currently utilizing Firefox version 53, Geckodriver version 0.16, and selenium version 3.4. My aim is to capture console logs from the webdriver. from selenium import webdriver from selenium.webdriver.common.desired_capabilities import DesiredCapabil ...

python-2.7seleniumselenium-firefoxdrivergeckodriver

Understanding the process of linking JavaScript code to a database within the ASP.NET framework

I have been successfully using an ASP.NET application to connect to a SQL Server 2016 database. However, I now have a new task of incorporating Javascript into the code in order to retrieve data from the database and present it to the user. I am aware of t ...

javascriptasp.netjson.netdatabase

Does jQuery mobile lack a back button feature?

Thoughts swirling in my mind: <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> <title></title> <script src="<?php echo base_url();?>/assets/js/vendor/modern ...

javascriptjquerymobile-applicationjquery-mobile

Web Testing: Steps to remove a logged-in SSO session for WebDriver

Currently, I am conducting tests to confirm the functionality of the SSO feature we implemented on our website. The issue lies in the fact that the automated test utilizes a static test-box with pre-installed browsers, causing it to automatically launch an ...

selenium-webdriverautomated-testswebdriversingle-sign-onbrowser-history

"The power of Node JS in handling JSON data and gracefully

I'm having trouble extracting a specific part of a JSON object in Node JS. When I print the response body, the entire object is displayed correctly. However, when I try to access object.subsonic-response, it returns NaN. I've spent a lot of time ...

javascriptjsonnode.js

Extract information from a database table for presentation as simple text

I am looking to extract information from each row and display it as plain text on the same page within a paragraph. Here is an example table for reference: <table> <thead> <tr> <th class="a header">A</th ...

javascriptphpjqueryhtml

I am having trouble getting text to display properly in a jQuery Mobile dialog

I am attempting to dynamically set the header and button texts using JavaScript, but unfortunately it's not working as expected. To demonstrate the issue, I have added my code on jsfiddle: http://jsfiddle.net/tMKD3/8/. Here is the HTML code: <bod ...

javascriptjquery-mobile