Scraping JavaScript Content Webpages with VBA

I'm attempting to extract a table from the Drainage Services Department website. I've written the VBA code below, but it doesn't seem to be working. I suspect that the issue lies in the fact that this particular table is generated using JavaScript. Any suggestions on how to resolve this problem?

Sub DSD()
    
    Dim ie As New InternetExplorer
    Dim html As New HTMLDocument
    Dim url As String
    
    url = "https://www.dsd.gov.hk/EN/Tender_Notices/Current_Tenders/index.html"
    
    ie.Visible = False
    ie.navigate url
    
    Do While ie.readyState <> READYSTATE_COMPLETE
        DoEvents
    Loop
    
    Set html = ie.document
    
    Dim lists As IHTMLElementCollection
    Dim anchorElements As IHTMLElementCollection
    Dim ulElement As HTMLUListElement
    Dim liElement As HTMLLIElement
    Dim row As Long

    Set lists = html.getElementsByClassName("ncol-md-12 result")
    row = 1
    
    For Each ulElement In lists
        For Each liElement In ulElement.getElementsByTagName("tbody")
          Set anchorElements = liElement.getElementsByTagName("td")
          If anchorElements.Length > 0 Then
              Cells(row, 1) = anchorElements.Item(0).innerText
               row = row + 1
          End If
    Next liElement
Next ulElement
    
  
    
      
    
ie.Quit
End Sub

I'm aiming to scrape data from the above-mentioned website.

Answer №1

UPDATE Give this a try instead. I allowed some time for everything to load into the lists object that you've set up, and that seemed to do the trick.

 Sub DSD()
    
    Dim ie As New InternetExplorer
    Dim html As New HTMLDocument
    Dim url As String
    
    url = "https://www.dsd.gov.hk/EN/Tender_Notices/Current_Tenders/index.html"
    
    ie.Visible = False
    ie.navigate url
    
    Do While ie.readyState <> READYSTATE_COMPLETE
        DoEvents
    Loop
    
    Set html = ie.document
    
    Dim lists As IHTMLElementCollection
    Dim anchorElements As IHTMLElementCollection
    Dim ulElement As HTMLUListElement
    Dim liElement As HTMLLIElement
    Dim row As Long
    
    'Instead of saying "ncol-md-12 result," it's actually named "col-md-12 result"
    Set lists = html.getElementsByClassName("col-md-12 result")
    row = 1
    
    ''Application needs time to load everything into lists.
    Application.Wait (Now + TimeValue("00:00:01"))
    
    ''Or use a loop
    ''Do While i < 1000
        ''DoEvents
        ''i = i + 1
    ''Loop
    
    For Each ulElement In lists
        
        For Each liElement In ulElement.getElementsByTagName("tbody")
         
          Set anchorElements = liElement.getElementsByTagName("td")
          If anchorElements.Length > 0 Then
               Debug.Print anchorElements.Item(0).innerText
               Cells(row, 1) = anchorElements.Item(0).innerText
               row = row + 1
          End If
        Next liElement
    Next ulElement
      
ie.Quit
End Sub

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Inquiry Regarding Scrolling to Bottom of Selenium Window

Hello, I'm having an issue using Selenium to extract the title from a webpage. It seems like the elements only appear once I scroll down the page. To tackle this, I used: driver.execute_script("window.scrollTo(0,document.body.scrollHeight);" ...

I am encountering an error with an Unhandled Promise Rejection, but I am unable to determine the reason behind it

const express = require('express'); const cors = require('cors'); const massive = require('massive'); const bodyParser = require('body-parser'); const config = require('../config'); const app = express(); ...

Utilizing icons with vuetify's v-select component: a guide

In the code snippet below, I am using a v-select element to display a list of items filled from an array: <v-select v-model="myModel" :items="users" chips :readonly="!item.Active" label="Required users to f ...

Unable to populate an array with JSON elements within a for loop triggered by useEffect

In my code, there is an issue with the array candleRealTimeDataQueue not updating correctly. Below is the snippet of the problematic code: let candleCurrentJSONDataWS = null; var candleRealTimeDataQueue = []; let tempDateTime = null; let ca ...

Trouble with X-editable: Unable to view values when editing and setting values using J

When using X-editable to modify a form with data, I encounter an issue. Initially, the values are loaded from the database to the HTML, but when users try to edit by clicking on the "Edit" button, all values appear as "Empty" instead of their actual cont ...

Looking for a new slider option to replace the traditional Conveyor belt slideshow?

I have successfully used the Conveyor belt slideshow script from http://www.dynamicdrive.com/dynamicindex14/leftrightslide.htm. Now, I am looking to find another script that works similar to this one. Does anyone have any recommendations for a tool that ...

How to create a floating <Toolbar/> with ReactJS, Redux, and Material-UI

Can anyone help me figure out how to make a toolbar floatable when scrolling down using Material-UI? I tried setting textAlign:'center', position: 'fixed', top: 0, but it's resizing strangely when applied to the <Toolbar/>. ...

PHP response triggers AJAX autocomplete functionality in JavaScript

The autocomplete hints are not displaying any response for me. Here is the jQuery code that I am using: jQuery( ".newtag_new" ).autocomplete({ minLength: 0, source: function( request, response ) { jQuery.ajax({ type: 'GET ...

What are the steps to program a bot to respond to different types of media messages (such as pngs, mp4

I have been attempting to elicit a reaction from him based on the media message, but my attempts have been fruitless so far. It seems that the only time it reacts is when there is no text within the message... which complicates things. Can anyone provide s ...

Troubleshooting Images in a React Application Integrated with WordPress API

I am struggling to understand why this GET request is consistently returning a 404 error. I have thoroughly tested the URL using Postman and everything seems to be in working order for the title and excerpt, but the images are causing some issues. Does a ...

What is the best way to retrieve a response from a PHP file as an array through Ajax?

Seeking assistance in retrieving a complete address by entering the postal code in an HTML form textbox and clicking a button. The setup involves two files - one containing the ajax function and the other housing the PHP code. Uncertainty looms over whethe ...

Leveraging $http or $timeout in conjunction with $stateProvider in AngularJS

I am seeking guidance on loading a template for a specific state in Angular using $http after coming across this question on Stack Overflow: Is it possible to load a template via AJAX request for UI-Router in Angular? The documentation for ui.router demon ...

From AJAX response to React state attribute, store the JSON data

I have a component where I need to fetch freight values via ajax and store them in the state property of this class. import React, { Component } from 'react'; import Freight from './Freight'; import CreateFreightEntryModal from '. ...

Error: express is missing a closing parenthesis for the argument list

When running this code in the VS Code terminal, be sure to verify any errors that may occur. var express = require('express'); var app = express(); app.get('/', function(request, response) { response.send("hello world"); }); app.li ...

Errors and disruptions caused by SmoothScroll, ScrollMagic, and GSAP triggering glitches, jumps, and crashes

Connecting ScrollMagic with GSAP is not an issue - it works seamlessly. However, I encountered a problem when trying to implement smooth scrolling for my mouse. I added a smooth scrolling plugin to my project from this link: http://www.jqueryscript.net/ani ...

Challenges Encountered When Working with React.useState()

I am facing an issue where a new row is not appearing after clicking the button. Although the console.log output indicates that the row was added correctly to the tables variable. Another concern I have is why I can see the new row added to the table even ...

Is the useNavigate() function failing to work consistently?

Currently facing an issue while working on a MERN Web App. When logging in with an existing account, the backend API call returns user properties and a JWT Token. Upon saving it, I use the navigate function to redirect the User to the homepage. Everything ...

Encountering a node-gyp error during the deployment of a Rails 6 application with a Vue app on Heroku

I'm running into an issue when trying to deploy my Rails 6 app with Vue on Heroku. The error I'm getting is as follows: [4/4] Building fresh packages... error /tmp/build_c242c7d78580af478535f5a344ff701e/node_modules/fibers: Command failed. ...

What is the best way to implement an AppBar that fades in and out when scrolling within a div?

I'm trying to implement a Scrollable AppBar that hides on scroll down and reappears when scrolling up. Check out this image for reference To achieve this functionality, I am following the guidelines provided by Material-UI documentation export defa ...

Collecting information from JSON files

Within the cells of my calendar are placeholders for events, dates, participants, and descriptions. Now, I am looking to populate these placeholders with data previously stored using localStorage. Unfortunately, my current code is not achieving this. How ...