Using PhantomJS to extract data from an ASP.NET website that features a dynamic combo dropdown box

Question

Using PhantomJS to extract data from an ASP.NET website that features a dynamic combo dropdown box

I am attempting to scrape this particular page that is coded in ASP.NET and contains 7 dynamic combo drop down boxes using PhantomJS v1.9.8.

The JavaScript code I am using is as follows:

var page = require('webpage').create();
console.log('Current user agent: ' + page.settings.userAgent);
page.settings.userAgent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.2 (KHTML, like Gecko) Chrome/5.0.342.3 Safari/533.2';
page.open('http://www.etcfinance.com.hk/online_appraise.aspx', function(status) {
    page.injectJs("https://code.jquery.com/jquery-latest.js", function() {
        page.evaluate(function() {
          $("#ddlArea").val('香港');
          __doPostBack('ddlArea', '');
          setTimeout(function() {
            console.log('Zone: ' + $('#ddlZone').val());
          }, 1000);
        });
        phantom.exit();
    });
});

The output stalls at :

Current user agent: Mozilla/5.0 (Macintosh; PPC Mac OS X) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.8 Safari/534.34

and does not proceed further. How can I effectively select desired values for all of those combo dropdown boxes?

The relevant part of the HTML structure looks like this:

<table width="460" bgcolor="#E0F3FF" border="0" cellpadding="3" cellspacing="0" class="content">
<tbody><tr height="20"><td></td></tr>
<tr class="insidecontent"> 
  <td style="Padding-Left:20px;Padding-Right:20px;"> 
    <div align="left"> Region : </div>
  </td>
  <td valign="top"> 
  <select name="ddlArea" onchange="javascript:setTimeout('__doPostBack(\'ddlArea\',\'\')', 0)" id="ddlArea" class="textbox" style="width:29em">
        <option selected="selected" value="">Select Region</option>
        <option value="Hong Kong">Hong Kong</option>
        <option value="Kowloon">Kowloon</option>
        <option value="New Territories/Outlying Islands">New Territories/Outlying Islands</option>

    </select>
  </td>
</tr>
... (remaining HTML structure) ...

Note: The above HTML code may contain errors.

Additionally, the reason for opting to use page.injectJS over page.includeJS is due to the latter triggering the following error:

Unsafe JavaScript attempt to access frame with URL about:blank from frame with URL file://parse.js. Domains, protocols, and ports must match.

javascript phantomjs html-select

Answer 1

Answer №1

page.injectJs does not have a callback function and can only inject local files. The desired code is not executed using this method. To include remote scripts, you should use page.includeJs.

You have two options: either download jQuery and place it in the local directory to use page.injectJs (the simpler solution) or try to make it work with remote scripts using page.includeJs. This may require running with commandline options such as --web-security=false and

--local-to-remote-url-access=true

.

Also note that jquery-latest.js is permanently fixed at version 1.11.1. If you need a newer version of jQuery, specify the actual version number.

Another issue in your script is early exiting. Using setTimeout interrupts the execution flow and page.evaluate finishes immediately. The exit call happens before the setTimeout callback. Here's a workaround:

page.evaluate(function() {
    $("#ddlArea").val('香港');
    __doPostBack('ddlArea', '');
});
setTimeout(function() {
    page.evaluate(function() {
        console.log('Zone: ' + $('#ddlZone').val());
    });
    phantom.exit();
}, 1000);

This improves the script, but you still might not see output on the console. You also need to register for the page.onConsoleMessage event.

Chained version:

var selects = [
    ['ddlArea', '香港'], 
    ['ddlZone', '...'], 
    ...
];

selects.forEach(function(sel, i){
    setTimeout(function() {
        page.evaluate(function(sel) {
            $("#"+sel[0]).val(sel[1]);
            __doPostBack(sel[0], '');
        }, sel);
    }, i * 1000);
});
setTimeout(function() {
    phantom.exit();
}, 1000 * selects.length);

A better approach would be to simulate actual clicks and utilize waitFor paired with async.js to wait until the next select is populated.

Answer 2

page.injectJs does not have a callback function and can only inject local files. The desired code is not executed using this method. To include remote scripts, you should use page.includeJs.

You have two options: either download jQuery and place it in the local directory to use page.injectJs (the simpler solution) or try to make it work with remote scripts using page.includeJs. This may require running with commandline options such as --web-security=false and

--local-to-remote-url-access=true

.

Also note that jquery-latest.js is permanently fixed at version 1.11.1. If you need a newer version of jQuery, specify the actual version number.

Another issue in your script is early exiting. Using setTimeout interrupts the execution flow and page.evaluate finishes immediately. The exit call happens before the setTimeout callback. Here's a workaround:

page.evaluate(function() {
    $("#ddlArea").val('香港');
    __doPostBack('ddlArea', '');
});
setTimeout(function() {
    page.evaluate(function() {
        console.log('Zone: ' + $('#ddlZone').val());
    });
    phantom.exit();
}, 1000);

This improves the script, but you still might not see output on the console. You also need to register for the page.onConsoleMessage event.

Chained version:

var selects = [
    ['ddlArea', '香港'], 
    ['ddlZone', '...'], 
    ...
];

selects.forEach(function(sel, i){
    setTimeout(function() {
        page.evaluate(function(sel) {
            $("#"+sel[0]).val(sel[1]);
            __doPostBack(sel[0], '');
        }, sel);
    }, i * 1000);
});
setTimeout(function() {
    phantom.exit();
}, 1000 * selects.length);

A better approach would be to simulate actual clicks and utilize waitFor paired with async.js to wait until the next select is populated.

Using PhantomJS to extract data from an ASP.NET website that features a dynamic combo dropdown box

Answer №1

Similar questions

The handleChange function fails to trigger when selecting a date using Material UI components

Can Node support setting the ECMAScript version?

"JQPlot line chart fails to extend all the way to the edge of the

The Ajax functionality seems to be malfunctioning when attempting to call a

Methods for encoding and decoding special characters using either JavaScript or jQuery

How can you set an input field to be initially read-only and then allow editing upon clicking a button using Vue.js?

What is the best way to insert hyperlinks within every cell of a table using Angular?

Creating a real-time email validation system that incorporates both syntax and domain checking

Can Angular retrieve the inner HTML content of an element?

Float and tap

Tips for maintaining the data on a page continuously updating in AngularJS

How can the node version be set globally in NVM?

Determining the type of <this> in an Object extension method using TypeScript

In my Google Sheets script, I am looking to introduce a 2-minute delay within the forEach loop for sending emails. Unfortunately, I have not had success

Encountering a VueJS error with Google Cloud Logging Winston: "TypeError: The 'original' argument must be a Function."

Confusion surrounding the concept of returning an arrow function from a Vuex storage getter

Stripping the prefix from a string using the .split function leads to an error message stating "Unexpected

Notify when the specified URL path is returned while the alert variable is present in jQuery

Error: The function $.get is not defined on line 1, character 3

What could be causing React to render only the final two elements of my array?