Scraping JavaScript Content Webpages with VBA

I'm attempting to extract a table from the Drainage Services Department website. I've written the VBA code below, but it doesn't seem to be working. I suspect that the issue lies in the fact that this particular table is generated using JavaScript. Any suggestions on how to resolve this problem?

Sub DSD()
    
    Dim ie As New InternetExplorer
    Dim html As New HTMLDocument
    Dim url As String
    
    url = "https://www.dsd.gov.hk/EN/Tender_Notices/Current_Tenders/index.html"
    
    ie.Visible = False
    ie.navigate url
    
    Do While ie.readyState <> READYSTATE_COMPLETE
        DoEvents
    Loop
    
    Set html = ie.document
    
    Dim lists As IHTMLElementCollection
    Dim anchorElements As IHTMLElementCollection
    Dim ulElement As HTMLUListElement
    Dim liElement As HTMLLIElement
    Dim row As Long

    Set lists = html.getElementsByClassName("ncol-md-12 result")
    row = 1
    
    For Each ulElement In lists
        For Each liElement In ulElement.getElementsByTagName("tbody")
          Set anchorElements = liElement.getElementsByTagName("td")
          If anchorElements.Length > 0 Then
              Cells(row, 1) = anchorElements.Item(0).innerText
               row = row + 1
          End If
    Next liElement
Next ulElement
    
  
    
      
    
ie.Quit
End Sub

I'm aiming to scrape data from the above-mentioned website.

Answer №1

UPDATE Give this a try instead. I allowed some time for everything to load into the lists object that you've set up, and that seemed to do the trick.

 Sub DSD()
    
    Dim ie As New InternetExplorer
    Dim html As New HTMLDocument
    Dim url As String
    
    url = "https://www.dsd.gov.hk/EN/Tender_Notices/Current_Tenders/index.html"
    
    ie.Visible = False
    ie.navigate url
    
    Do While ie.readyState <> READYSTATE_COMPLETE
        DoEvents
    Loop
    
    Set html = ie.document
    
    Dim lists As IHTMLElementCollection
    Dim anchorElements As IHTMLElementCollection
    Dim ulElement As HTMLUListElement
    Dim liElement As HTMLLIElement
    Dim row As Long
    
    'Instead of saying "ncol-md-12 result," it's actually named "col-md-12 result"
    Set lists = html.getElementsByClassName("col-md-12 result")
    row = 1
    
    ''Application needs time to load everything into lists.
    Application.Wait (Now + TimeValue("00:00:01"))
    
    ''Or use a loop
    ''Do While i < 1000
        ''DoEvents
        ''i = i + 1
    ''Loop
    
    For Each ulElement In lists
        
        For Each liElement In ulElement.getElementsByTagName("tbody")
         
          Set anchorElements = liElement.getElementsByTagName("td")
          If anchorElements.Length > 0 Then
               Debug.Print anchorElements.Item(0).innerText
               Cells(row, 1) = anchorElements.Item(0).innerText
               row = row + 1
          End If
        Next liElement
    Next ulElement
      
ie.Quit
End Sub

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

How to prevent mouse click events in Three.js after interacting with an HTML overlay

Encountering an issue with Three.js: I have created my own HTML user interface as a simple overlay. However, I am facing a problem where the mouse click does not reset when I interact with elements on this overlay. Specifically, when I click on the "Came ...

I encountered a response error code 500 from the development server while using my emulator

As I embark on setting up the react-native environment for development, I encounter an error when executing the command react-native run-android. root@pc:~/l3/s2/DevMobMultipltm/Wakapp# ` A series of tasks are carried out including scanning folders for sy ...

Is there a method to generate an endless carousel effect?

Hello, I am trying to create an infinite carousel effect for my images. Currently, I have a method that involves restarting the image carousel when it reaches the end by using the code snippet progress = (progress <= 0) ? 100 : 0;. However, I don't ...

Is it possible for the number returned by setTimeout() in JavaScript to be negative?

Is it possible for a number returned by the setTimeout() function in JavaScript to be negative? Currently, I've observed that the timeoutIds generated are sequentially numbered in Chrome as 1,2,3,4,5,6... While in Firefox, they start from number 4 an ...

I'm all set to launch my express js application! What are the major security concerns that I need to keep in

As a beginner in deploying express applications, I find myself lacking in knowledge about the essential security measures that need to be taken before launching a web application. Here are some key points regarding my website: 1) It is a simple website ...

Angular2 Error: Cannot have two identifiers with the same name, 'PropertyKey' is duplicated

I am currently developing an application with angular2 using angular-cli. Unfortunately, angular-in-memory-web-api was not included by default. After some research, I manually added the line "angular-in-memory-web-api": "~0.1.5" to my ...

having trouble parsing JSON data

Recently, I decided to experiment with JSON and utilized json_encode to generate a JSON object structured as shown below: [{ "timestamp": "12\/16\/2013 0:00", "curr_property": "7211", "curr_property_cost": "123", "day_pro ...

Tips for avoiding background color interference with raycaster

In my current three js scene, I have a ground, sky, and various objects. I want specific objects to change color to red when the mouse hovers over them, but not all objects should do this. Currently, everything I touch turns red, which is not what I want. ...

The problem arises when Angular's $interval function is not recognized

Despite the possibility of this being considered a duplicate, none of the related topics have provided a solution to my simple date count down directive: class Clock { constructor() { this.restrict = 'AC'; this.replace = true ...

Leveraging Selenium for extracting email addresses through character validation

Recently, I've been attempting to extract email addresses from Facebook business information pages like the one found at this link: Facebook Business Page Example However, I encountered difficulty in accurately determining the XPath needed for extrac ...

ESLint detects the error "screen not found in @testing-library/vue"

When trying to utilize @testing-library/vue with the screen method imported, I encountered an error from ESLint stating: "screen not found in @testing-library/vue". // The render function doesn't give an error but screen does import { render ...

Use JavaScript's Array.filter method to efficiently filter out duplicates without causing any UI slowdown

In a unique case I'm dealing with, certain validation logic needs to occur in the UI for specific business reasons[...]. The array could potentially contain anywhere from several tens to hundreds of thousands of items (1-400K). This frontend operation ...

Clicking on a jQuery element will reveal a list of corresponding elements

I've retrieved a list of elements from the database and displayed them in a table with a button: <a href="#" class="hiden"></a> To show and hide advanced information contained within: <div class="object></div> Here is my jQ ...

Tips for ensuring the Google Maps API script has loaded before executing a custom directive on the homepage of an Angular website

Issue - I am facing issues with Google Maps autocomplete drop-down not working on my website's main page even after parsing and loading the Google Maps API script. The problem seems to be a race condition on the main page of my website, specifically i ...

Troubleshooting MySQL Database Insertion Errors caused by Dynamic Forms

<body> <?php $con = mysqli_connect('localhost','root','','cash'); $query = "SELECT DISTINCT category FROM cash"; $result = mysqli_query($con,$query); $dropDownList = &apo ...

Can you explain the significance of npm WARN excluding symbolic link?

Could you please explain the meaning of npm WARN excluding symbolic link? Also, any advice on how to resolve this issue? ...

Typescript HashMap implementation with Lists as values

Currently delving into TypeScript, I am attempting to generate a collection in a manner akin to the following Java example: HashMap<String, List<String>> hashMap = new HashMap<String,List<String>>(); Struggling to locate any releva ...

Guide to incorporating third-party JavaScript files and functions into my Angular web app

I have been trying to integrate external code (HTML, JS, and CSS files) into my Angular web application. Within this external code, the structure of the HTML file is as follows: index.html <html> <header> </header> <body> </bo ...

Issue encountered while attempting to generate a Jquery button

I seem to be facing some difficulties as I have successfully completed this task before. Currently, I am dynamically adding HTML code: ...<td><label><input type=\"checkbox\" checked=\"checked\" class=\"Activechk fo ...

Retrieving object key value from an array using Underscore.js

Hey there, I'm facing a challenge where I need to extract the values of wave1 and wave2 from an array using underscore.js. array = [{"id":1,"name":"Monoprix", "pdv":16,"graph":[{"wave1":22,"wave2":11}]} ; I attempted the following: $scope.wave1 = a ...