Scraping website page flow, including button clicks, using Javascript API

I am in search of a JavaScript API that can be used to screen scrape page flow, including button clicks. While WebDriver of Selenium would have been a good choice if it were on the server side, we specifically want the screen scraping to occur within the client browser. The pages we need to scrape are part of a transaction process (including logging into a third-party website, completing transaction steps 1 and 2, and final confirmation). Is there any JavaScript API available for this purpose?

As far as I know, both Node.js and PhantomJS do not have the capability to click buttons on the scraped page.

Thank you in advance, Abbas

javascriptseleniumscreen-scraping

Answer №1

Utilizing Webdriver as an HTTP based protocol allows for seamless communication between browsers, enabling control from one browser to another. For a comprehensive guide on this subject, I shared a tutorial a few weeks back which can be accessed here

Answer №2

Recently, I came across a tool called DalekJS which allows for capturing screenshots of web pages and interacting with elements on the page as well. It seems to have support for multiple browsers, although they need to be setup before using it.

Below is a sample code snippet taken directly from their documentation:

 test.open('http://example.com/')
 .assert.text('#header').is('Header Text')
 .assert.visible('#button')
 .click('#button')
 .done();

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Encountering a "Index of /" error when working on a basic HTML project with Tailwind

I am having trouble understanding how to properly set up and manage version control for my projects. I initially created a project using Tailwind CSS that worked fine on my local machine with an http-server plugin. However, when I tried to create a GitLab ...

javascripthtmlnode.jsgittailwind-css

Do not trigger popup if mandatory field is left blank

I am currently working on a contact form where, upon clicking the send button, a popup appears to confirm that the privacy policy has been read. The form does not submit if there are any empty required fields, but the popup still appears. I need to modify ...

javascriptjqueryhtmlcsspopup

The functionality of window.print() is not supported in Internet Explorer 11

My angular js app has a print functionality where an html page opens using the window.open method. public openNewWindow(html: string, title: string) { var popupWin = window.open('', '_blank', 'scrollbars=no,menubar=no,toolbar ...

javascripthtmlangularjsinternet-explorerprinting-web-page

The post body is incomplete and is lacking the necessary parameter "To" that

It seems there might be a configuration issue either in the header or the body. Should I make changes to the headers or is it possible that the body is misconfigured? const axios = require('axios'); const url = '/my_url'; const aut ...

javascriptnode.jsaxios

Typescript selection missing in Intellij IDEA when creating new Step definition File for Protractor and Cucumber

Currently working on: Protractor Cucumber and Typescript project Preferred IDE: Intellij IDEA Ultimate edition Any suggestions on what I might be overlooking? List of added plugins: 1. Javascript Support 2. Node.js ...

javascripttypescriptintellij-ideaprotractorcucumber

Why does my computed property become undefined during unit testing of a head() method in Vue.js with Nuxt.js?

In my Vue.js + Nuxt.js component, I have implemented a head() method: <script> export default { name: 'my-page', head() { return { title: `${this.currentPage}` }; }, ... } </script> ...

javascriptunit-testingvue.jsnuxt.js

Modifying attribute values in Angular through custom directives

Struggling to figure out the most efficient way to monitor attribute changes, preferably triggered by a keypress event and linked to the scope in the parent controller I want each individual instance of the directive to have its own 'hasFocus' p ...

javascriptangularjsangularjs-directiveangularjs-scope

Events for focusing and blurring top level windows

I'm encountering an issue with a basic frameset setup <frameset rows="100, 200"> <FRAME name="listener" src="frame1.html"> <FRAME name="content" src="http://www.someexternalurl.com/"> </frameset> Within the listener ...

javascriptjqueryevents

Modify the href link before submitting the request

Here is the code snippet that I am working with : <a class="myLink" href='http://www.website.it/'>Link</a> <script type="text/javascript"> var stringToSend = ""; $('.myLink').click(function() { string ...

javascriptjqueryhtml

Tips for categorizing items based on their names

Upon receiving a response as shown below: [ {id:1,name:"type1-something"}, {id:2,name:"something-type2"}, {id:3,name:"type3-something"}, {id:4,name:"something-type1"} ] I have an Enum with ...

javascripttypescript

Exploring the concept of reflection in Selenium using Java

I'm struggling to grasp the concept of reflection and encountering a problem. The issue lies in the second for loop where method.length is not pulling methods from another program. Here's the code snippet: public class DriverScript { publ ...

javaselenium

Firebase Functions encounters an issue with internal error handling

I am currently working on a project using Firebase Hosting and Firebase Functions. Index.js: exports.simpleFunction = functions.https.onCall((data, context) => { return data; }); Index.html: var simpleFunction = firebase.functions().httpsCallable(& ...

javascriptnode.jsfirebasegoogle-cloud-functions

React redux - unable to load store without encountering any errors

I am currently working on a project inspired by the code example in the Learning React book, which focuses on using react redux and react router for a single page application. You can find my project here, where I have tried to replicate the example. To r ...

javascriptreactjsreduxreact-redux

Assistance with Collision Detection in HTML5 Canvas using JavaScript

Attempting to create a platformer game using HTML5 and the canvas feature. I managed to implement collision detection with rectangles, but encountered issues when adding multiple rectangles. I have a function that adds new objects to an array with attribut ...

javascripthtmlcanvascollision-detection

What is the method for configuring Vue to utilize a value other than the value property in form fields?

I am facing a unique challenge. Consider this: <select name="screw[type]" v-model="form.screw.type"> <option value="My value" ><?php _e('My value', 'fiam'); ?></option> //[...] Now, in another part of my ...

javascriptvue.jsinternationalization

Tips for eliminating whitespace from an input field and then updating the field with the trimmed value

Currently, I am working on email validation where users might input empty spaces in the email field. To address this issue, I have implemented a logic to trim the input value using $trim and then re-assign it to the input field. Although everything seems ...

javascriptjqueryvalidationemailtrim

How is the height or width of the Bootstrap modal determined?

I'm having trouble utilizing Jschr's modified Bootstrap-Modal for different modal sizes. I can't seem to understand why some modals appear taller, wider, or have other specific dimensions. For more information, refer to the documentation: ...

javascriptcssajaxhtmltwitter-bootstrap

Switch between viewing outcomes retrieved from a database

I'm fairly new to working with jQuery, PHP and databases, but I have managed to successfully create a database and retrieve data using PHP. When a search term is entered, the retrieved data from the database is displayed on the results page. For exam ...

javascriptjquery

Troubleshooting issues with JavaScript's window.location.href functionality

Trying to change the URL using window.location.href doesn't seem to be working for a specific link. The current page URL is: http://localhost:37368/Office/Search/c2VhcmNoaWRzPTEyMiwxMjIsMTI0LDE1OCwzNzl8bG9jYXRpb25pZHM9MSwyfGZyb21pZHM9fHRvaWRzPQ== Af ...

javascriptwindow.location

Employ a variable in order to send json

Is it possible to insert data into a predefined JSON variable using a variable? Here are the predefined JSON variables: data(){ return { dataMON: [], dataTUE: [], dataWED: [], dataTHU: [], ...

javascriptjsonvue.js