Extract data from dynamically loaded tables using PhantomJS and Selenium Webdriver

Question

Extract data from dynamically loaded tables using PhantomJS and Selenium Webdriver

I've been informed by everyone that I should be able to retrieve the table content using PhantomJS when working with JavaScript. However, despite my attempts, I have not been successful.

My expectation was to obtain the table from the website

Page 1 displays correctly.

But when I attempt to navigate to page 2 by clicking on the CSS selector location, it still returns the content from page 1. What could be causing this issue?

#coding:utf-8
from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.PhantomJS()
driver.get("http://data.eastmoney.com/xg/xg/default.html")
time.sleep(2)
driver.find_element_by_css_selector("#PageCont > span.at").click()

list_cates = driver.find_element_by_css_selector("#dt_1 > tbody > tr:nth-child(1) > td:nth-child(2) > a").text
print(list_cates)

javascript selenium

Answer 1

Answer №1

The issue you are facing is that the data is not being updated immediately after the click event. You should introduce a delay to ensure the Ajax call has enough time to complete.

#coding:utf-8
from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.PhantomJS()
driver.get("http://data.eastmoney.com/xg/xg/default.html")
time.sleep(2)
driver.find_element_by_css_selector("#PageCont .next").click()

time.sleep(5)
list_cates = driver.find_element_by_css_selector("#dt_1 > tbody > tr:nth-child(1) > td:nth-child(2) > a").text
print(list_cates)
# Prints '太平鸟'

This code snippet fetches data from page 2

https://i.sstatic.net/RT32I.png

Answer 2

The issue you are facing is that the data is not being updated immediately after the click event. You should introduce a delay to ensure the Ajax call has enough time to complete.

#coding:utf-8
from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.PhantomJS()
driver.get("http://data.eastmoney.com/xg/xg/default.html")
time.sleep(2)
driver.find_element_by_css_selector("#PageCont .next").click()

time.sleep(5)
list_cates = driver.find_element_by_css_selector("#dt_1 > tbody > tr:nth-child(1) > td:nth-child(2) > a").text
print(list_cates)
# Prints '太平鸟'

This code snippet fetches data from page 2

https://i.sstatic.net/RT32I.png

Answer 3

Answer №2

Hey there, check out this code snippet by Tarun Lalwani:

 #coding:utf-8
    from selenium import webdriver
    import time
    from selenium.webdriver.support.ui import WebDriverWait

    driver = webdriver.PhantomJS()
    driver.get("http://data.eastmoney.com/xg/xg/default.html")
    time.sleep(2)
    for page_count in range(1,4):
        driver.find_element_by_id("gopage").send_keys(page_count)
        driver.find_element_by_css_selector("#PageCont > a.btn_link").click()
        time.sleep(10)
        list_cates = driver.find_element_by_css_selector("#dt_1 > tbody > tr:nth-child(1) > td:nth-child(2) > a").text
        print('retrieved ' + str(page_count) + ' items')
        print(list_cates)

View the execution result here

Moreover, trying to fetch a table within a frame using PhantomJS resulted in an error as well.

    #coding:utf-8
from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.PhantomJS()
driver.get("http://ipo.csrc.gov.cn/checkClick.action?choice=info#")
driver.find_element_by_css_selector("#type1 > a").click()

time.sleep(5)
result = driver.find_element_by_css_selector("#frame_body > table > tbody > tr:nth-child(1) > td > table > tbody > tr:nth-child(3) > td:nth-child(1)").text
print(result)

Answer 4

Hey there, check out this code snippet by Tarun Lalwani:

 #coding:utf-8
    from selenium import webdriver
    import time
    from selenium.webdriver.support.ui import WebDriverWait

    driver = webdriver.PhantomJS()
    driver.get("http://data.eastmoney.com/xg/xg/default.html")
    time.sleep(2)
    for page_count in range(1,4):
        driver.find_element_by_id("gopage").send_keys(page_count)
        driver.find_element_by_css_selector("#PageCont > a.btn_link").click()
        time.sleep(10)
        list_cates = driver.find_element_by_css_selector("#dt_1 > tbody > tr:nth-child(1) > td:nth-child(2) > a").text
        print('retrieved ' + str(page_count) + ' items')
        print(list_cates)

View the execution result here

Moreover, trying to fetch a table within a frame using PhantomJS resulted in an error as well.

    #coding:utf-8
from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.PhantomJS()
driver.get("http://ipo.csrc.gov.cn/checkClick.action?choice=info#")
driver.find_element_by_css_selector("#type1 > a").click()

time.sleep(5)
result = driver.find_element_by_css_selector("#frame_body > table > tbody > tr:nth-child(1) > td > table > tbody > tr:nth-child(3) > td:nth-child(1)").text
print(result)

Answer 5

Answer №3

I believe I have found a solution for case 1 by implementing this code: driver.find_element_by_id("gopage").clear().

However, I still require your assistance with the other case. Thank you in advance!

#encoding:utf-8
from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.PhantomJS()
driver.get("http://data.eastmoney.com/xg/xg/default.html")
time.sleep(2)
for page_count in range(1,4):
    driver.find_element_by_id("gopage").clear()
    driver.find_element_by_id("gopage").send_keys(page_count)
    driver.find_element_by_css_selector("#PageCont > a.btn_link").click()
    time.sleep(5)
    list_cates = driver.find_element_by_css_selector("#dt_1 > tbody > tr:nth-child(1) > td:nth-child(2) > a").text
    print('obtained' + str(page_count) + 'pieces')
    print(list_cates)
    driver.find_element_by_id("gopage").clear()

Answer 6

I believe I have found a solution for case 1 by implementing this code: driver.find_element_by_id("gopage").clear().

However, I still require your assistance with the other case. Thank you in advance!

#encoding:utf-8
from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.PhantomJS()
driver.get("http://data.eastmoney.com/xg/xg/default.html")
time.sleep(2)
for page_count in range(1,4):
    driver.find_element_by_id("gopage").clear()
    driver.find_element_by_id("gopage").send_keys(page_count)
    driver.find_element_by_css_selector("#PageCont > a.btn_link").click()
    time.sleep(5)
    list_cates = driver.find_element_by_css_selector("#dt_1 > tbody > tr:nth-child(1) > td:nth-child(2) > a").text
    print('obtained' + str(page_count) + 'pieces')
    print(list_cates)
    driver.find_element_by_id("gopage").clear()

Answer 7

Answer №4

By simply adding the code snippet `driver.switch_to_frame("myframe")`, I was able to find the solution for case 2.

This particular question required solving within a frame structure!

I want to express my utmost gratitude for assisting me with this issue. This is actually my first time seeking help on Stackoverflow, and I must say, I am loving it already!

#coding:utf-8
from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.PhantomJS()
driver.get("http://ipo.csrc.gov.cn/checkClick.action?choice=info#")

time.sleep(2)
driver.switch_to_frame("myframe")
result = driver.find_element_by_css_selector("#frame_body > table > tbody > tr:nth-child(1) > td > table > tbody > tr:nth-child(3) > td:nth-child(1)").text
print(result)

Answer 8

By simply adding the code snippet `driver.switch_to_frame("myframe")`, I was able to find the solution for case 2.

This particular question required solving within a frame structure!

I want to express my utmost gratitude for assisting me with this issue. This is actually my first time seeking help on Stackoverflow, and I must say, I am loving it already!

#coding:utf-8
from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.PhantomJS()
driver.get("http://ipo.csrc.gov.cn/checkClick.action?choice=info#")

time.sleep(2)
driver.switch_to_frame("myframe")
result = driver.find_element_by_css_selector("#frame_body > table > tbody > tr:nth-child(1) > td > table > tbody > tr:nth-child(3) > td:nth-child(1)").text
print(result)

Extract data from dynamically loaded tables using PhantomJS and Selenium Webdriver

Answer №1

Answer №2

Answer №3

Answer №4

Similar questions

How can a server retrieve a file uploaded using FormData and Ajax on a cross-domain upload?

Using a combination of different materials on a single mesh can lead to problems with z-index and clipping

Issue with AngularJS $http not responding to ng-click after first call

Completely different method for transmitting an array to NodeJS through AJAX

What are some ways to improve performance in JavaScript?

Unable to construct React/Next project - identified page lacking a React Component as default export (context api file)

What is the procedure for importing material UI components into the main class?

Make sure all of the inner tags are properly contained within their respective containers

What is the process for adding to a highly nested array in mongoose?

What is the best way to utilize a locator that changes based on a trailing digit within a placeholder in an xpath using Selenium with Python and pytest?

Troubleshooting VueJS, Electron, and Webpack integration with Hot Reload feature

Asynchronously load an AngularJS controller from AJAX without altering the route

Access the value of a JSON property, return null if the key is not found, all in one line of JavaScript code

Using JQuery to locate and substitute occurrences of HTML elements throughout my webpage

Utilizing Java with Selenium WebDriver to Export Webtable Data to Excel: A Step-by-Step Guide

The error message "AttributeError: 'module' does not contain a 'document' attribute" indicates that the specified module does not have a

Is there a way to fully load an entire HTML page on success using AJAX?

Using React to retrieve an object from a helper method through a GET request

What is the best way to extract value from an input text that is being repeated using ng-repeat within a Tr

Sorting the output with gulp and main-bower-files (gulp-order is not functioning)