Collecting information from websites by changing JavaScript parameters

I am currently attempting to extract intraday prices for a specific company from the website Enel Intraday. The issue I am facing is that when the data is fetched, it is spread across hundreds of pages, making it extremely time-consuming to gather all the necessary information. I have been experimenting with the Insomnia REST client in an attempt to manipulate the URL GET request or locate the JavaScript function responsible for generating the table values, however, my efforts have been futile thus far.

Upon inspecting the search button on the website, I discovered that the JavaScript function being triggered is named "searchIntraday" and requires input from a form labeled "intraday_form."

https://i.sstatic.net/dirmr.png

My goal is to acquire the following data in a single call instead of navigating through numerous tab pages. Ideally, a complete day's worth of data would resemble the following:

Time    Last Trade Price    Var %   Last Volume Type
5:40:49 PM  7.855   -2.88   570 AT
5:38:17 PM  7.855   -2.88   300 AT
...
9:00:07 AM  8.1     0.15    933,945 UT

This process involves iterating through pages 1 to 1017 for each individual day!

For assistance, I have referred to resources such as:

Article on Scraping with JavaScript

Similar Issue on Stack Overflow with Solution

https://i.sstatic.net/3XdLu.png

javascriptpython-3.xrestweb-scrapinginsomnia

Answer №1

Upon inspection, it seems that the data is not generated through javascript, but rather by loading pages. The screenshot I have provided below demonstrates the result I received when accessing the link mentioned. It is evident that the request's location corresponds to the page's location, and the HTML for the table is included in the response.

The HTML content of the response suggests that the pages are created on the server side as opposed to the client side. Unless a method is discovered to view all desired results at once, one must manually navigate through each page. However, if a specific URL can be identified, processing that singular page would suffice.

https://i.sstatic.net/bsh5M.png

Curious about its performance, I decided to run a script that processes the first 100 pages. Below is the code snippet used:

import pandas as pd
import requests

url = "https://www.borsaitaliana.it/borsa/azioni/contratti.html?isin=IT0003128367&lang=en&page="

df = pd.concat([
    pd.read_html(requests.get(url + str(page)).content)[0] 
    for page in range(100)
])

df.to_csv('enel.csv', index=False)

On my system, parsing 100 pages took approximately 1.25 minutes.

$ time python scrape.py 

real    1m16.914s
user    0m4.039s
sys 0m0.729s

Extrapolating from this, processing a single stock could take around 15 minutes. If dealing with 30 stocks of similar length, it might amount to 7.5 hours in total. Performing this task overnight could yield the desired outcome by morning.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

How can I populate an array to use in a datatable when rendering?

How can I ensure that my datatable renders with the proper data on my webpage without needing to interact with the sort button? Where should I populate the array for use in a datatable so that it occurs before the rendering? export function MyComponent() ...

javascriptreactjsasync-awaitnext.js

What is the process for passing two distinct objects in a POST request using Postman for testing a RESTful API?

I have a two-part question: firstly, how can I pass two JSON objects; secondly, how can I create an object from two different classes and pass that object to a POST method. When testing a POST method of a RESTful API using Postman, I encountered an issue. ...

jsonrestspring-bootpostman

Interacting between various components in separate files using React.js

Creating a page using React involves two Components with different functions. The first component, ProfileFill, is responsible for capturing form data. On the other hand, the second component, ProfileFillPercent, which resides in a separate file, calculate ...

javascriptreactjs

Utilizing Express-Partials in conjunction with a single layout to incorporate multiple partials

Recently, as I migrated to Node.js and ExpressJS 3.0, I noticed that partials were no longer supported. However, I stumbled upon express-partials which provided a similar feature. Upon exploring the example on their GitHub page, I came across this snippet ...

javascriptnode.jsexpresspartials

Having trouble with cc email addresses causing problems in Outlook

I'm encountering an issue where there is a '&' symbol in one of the cc email addresses. Due to the default behavior of the browser, the complete email address is not being displayed properly in Outlook. Here's an example: let to='a ...

javascripthtmloutlookmailto

What is the best way to emphasize a table row during a search rather than just the text?

I am currently using a jQuery script that searches for a specific string in a simple text input field and highlights only the string itself if it is found. However, my data is organized within a table and I am interested in knowing if it is possible to hig ...

javascriptjqueryhtml

Align the drop-down caret to the right using Bootstrap 4.1

I am currently using Bootstrap 4.1 and I have a Navbar that triggers a Modal Dialog Box with tabs and a drop down menu containing an image icon. My goal is to make the "caret" or down arrow of the drop down menu appear on the right side of the image. To a ...

javascriptjqueryhtmlcssbootstrap-4

Attempting to transmit information using Ajax to an object-oriented programming (OOP) class

Trying to send data with username, password, etc from an HTML form -> Ajax -> Instance -> OOP class file. Questioning the approach... Begins with the form on index.php <!-- Form for signing up --> <form method="post"> <div ...

javascriptphpjqueryajaxoop

Ways to verify that window.open is being invoked from a React component

In my React component, I have a set of nested components, each with its own checkbox. The state hook rulesToDownload starts as an empty array and dynamically adds or removes IDs based on checkbox selection. Upon clicking the 'download' button, t ...

javascriptreactjsdomenzymechai-enzyme

Employ the ternary operator to update the className in a React component

I'm currently developing a quiz application in React/Typescript with Bootstrap. I am aiming to have the buttons styled as bg-light initially, and then switch to bg-success if the user selects the correct answer, or bg-danger if they choose the wrong o ...

javascriptreactjsreact-bootstrap

What is the best way to import assets from an NPM-powered package in a PHP composer-based package?

First and foremost, let's clarify that this is not a question about incorporating an NPM package as a dependency of a Composer package. Direct usage of NPM or a composer plugin can easily solve that issue. If we have loaded an NPM package as a depend ...

javascriptphpnpmcomposer-phpassets

What is the process for inserting a new row into ngx-datatable using data provided by the user?

Is there a way to dynamically add a new row to an ngx-datatable based on user input? A method I currently use to add an empty row includes this code snippet: addRow() { this.rows.unshift({unique_id: '<em>empty</em>', name: '& ...

javascriptangularngx-datatable

"Bootstrap is functioning properly on my local environment, but it seems to

Utilizing the MVC framework and bootstrap has been successful for optimizing my website locally. However, when I upload it to the server, none of the CSS is being rendered. Additionally, the front page, meant to be a carousel slider, appears as a vertical ...

javascriptjquerycssasp.net-mvctwitter-bootstrap

What would be the best TypeScript target and libs to utilize in a transpiler-free Node.js project?

If I am running a TypeScript Node.js project with the command tsc && node build/index.js (where tsc builds into build/ and index.ts is in the project), what values should be used in lib and target in tsconfig.json to ensure access to the latest TypeScrip ...

javascriptnode.jstypescript

Implementing code to scroll and play multiple videos simultaneously using JavaScript and jQuery

This solution currently only works for one video, but it needs to be able to handle multiple videos (TWO OR MORE) <html> <head> <script src="https://code.jquery.com/jquery-1.8.0.min.js" integrity="sha256-jFdOCgY5bfpwZLi ...

javascriptjquery

Strategies for extracting special character codes from API data within a React functional component

I have been experimenting with a simple trivia API to improve my understanding of REACT. I have noticed that the question data is returning with special character codes, which is causing issues with react jsx. Can anyone suggest a method to easily convert ...

javascriptreactjs

Display on the terminal screen indefinitely

Recently, I delved into the world of Python and out of boredom, I decided to create a basic password generator. Below is the code snippet: import random upper = "ABCDFGHIJKLMNOPQRSTUVXYZ" lower = "abcdefghijklmnopqrstuvxwyz" numbers = ...

pythonpython-3.xpyinstaller

Executing an HTTP request with JavaScript to interact with Salesforce

Looking to connect Salesforce with Recosence, an external system. The scenario involves Recosense pushing data to Salesforce for storage. I have successfully created a post HTTP service and tested it in Postman, which generates an access token and records ...

javascript

Is it possible to assign ng-model to multiple values simultaneously?

I am currently working on incorporating an input field into my project. <input type="text" ng-model="choice.text"> Using choice.text allows me to create a new text property within the object choice. This functionality enables me to send the input c ...

javascripthtmlangularjsangular-ngmodel

Using asp.net and c# in tandem with javascript is a powerful

((LinkButton)e.Item.FindControl("my_label_name")) .Attributes.Add("onclick","javascript:myCustomFunction('" + dbValue1 + "','"+dbValue2+"')"); I implemented this code (located in my default.aspx.cs) ...

c#javascriptasp.net