JavaScript text parsing in real-time changes

I have a need to extract data from a webpage for scientific research purposes. The specific text I'm looking to extract is found within a < span > tag, but traditional HTML parsing methods won't work due to the rapid and constant updates happening, sometimes up to 10 times per second. Despite this challenge, I am aware that it can be achieved based on information from a scientific paper I came across.

The webpage where I need to gather this data from is: . Essentially, each time a paper is downloaded, a marker appears on the map indicating its location. My goal is to collect real-time data on the city/location associated with each marker as they appear, displayed beneath the map on the left side.

Questions:
1) How can I effectively parse this ever-changing text in real-time, especially considering that it's dynamically generated using Java-script code? While I have some experience with webpage parsing, handling fast-paced live text updates is new territory for me.

2) Given the importance of speed in both parsing and writing this data, which programming language would be best suited for my project? I intend to store the extracted data in an SQL database, so efficiency is key. If possible, I prefer to use Python provided there are robust libraries available for this purpose.

Thank you in advance for any guidance or recommendations you may have.

javascriptsqlparsingreal-timedata-mining

Answer №1

It appears that a JSON call is being made to retrieve map data. If you have the necessary authorization (such as a copyright notice), you can access the raw data directly by calling the specified URL instead of extracting it from the map.

$.getJSON('/ip2location/lookupMulti.php', { "rand": Math.random() }, function(data) {
    for (var i=0; i<data.length; i++) {
        var lat = data[i].lat;
        var lng = data[i].lng;
        var name = data[i].name;
    }
            // Additional code...

Keep in mind that many companies restrict frequent requests to their servers, whether it's through loading the main page or accessing lookupMulti.php. Without proper authorization, your IP address may be banned swiftly.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Iterate over the contents within the div tag

I need help with looping through the data in this specific div container. My goal is to extract row by row data from it. <div id="result" runat=server> <div id="gvResult" class="RowGroup"> <div class="Row RowBg" tabindex="99"> ...

javascriptjquery

Sending optional data in Angular routesIn Angular, you can include additional

In my project utilizing angular 5, I have a lengthy routing file: const homeRoutes: Routes = [ { path: 'home', component: HomeComponent, children: [ { path: 'registration', component: RegistrationCompone ...

javascriptangulartypescriptangular5angular-routing

What causes arrays in JavaScript to not be sorted in either ascending or descending order based on dates?

I'm attempting to organize my array of objects that contain a date property. I need to arrange the array based on ascending or descending dates. I've attempted the following approach: https://jsfiddle.net/rxaLutgn/1/ function sort_by(field, rev ...

javascriptjqueryarrays

Modify JSON date format to a shorter version using the first 2 letters of the month and the year

Looking to convert date values in a JSON array from "December 2016" format to "D16". Hoping to accomplish this using Regex, any assistance would be greatly appreciated. [["November 2016","December 2016","January 2017","February 2017","March 2017"],["tot ...

javascriptjqueryjson

Auto-scroll feature malfunctioning

My auto scroll function using jQuery isn't working, here is my CSS: #convo_mes{ text-align:left; width:98%; height:80%; background:#fff; border:1px solid #000; overflow-x:auto; } And in my JavaScript: $(".mes").click(functio ...

javascriptjquerycss

Merge two distinct JSON objects obtained through an API request using Javascript

Struggling with my currency conversion project, I'm trying to display JSON response results on my website but can't seem to make it work. The code snippet below, .then((response) => { return response.json(); }) .then((jsonRespo ...

javascriptjsonrest

Javascript menu toggle malfunctioning with sub-menus

I am in the process of creating a responsive menu for a complex website. The horizontal menu transitions into a vertical layout on smaller screens, with javascript used to toggle the sub-menu items open and closed when clicked. One issue I am facing is wit ...

javascriptjqueryhtmlcssmenu

Turning a string retrieved from the element's data attribute into a JSON format

I am running into an issue with the code snippet below. It seems that $.parseJSON() is having trouble with single and double quotes. I'm stuck on finding a solution to this problem. Any help would be greatly appreciated! <div data-x='{"a":"1" ...

javascriptjqueryhtmljson

Can we leverage map/filter/reduce functions within a promise by encapsulating the result with Promise.resolve()?

Currently, my approach to doing loops inside a promise looks like this: asyncFunc() .then(() => { return new Promise((resolve) => { for (let i = 0; i < length; i++) { // do something if (j == length - 1) { ...

javascriptnode.jspromise

Make sure to tick off the checkboxes when another checkbox is marked

When a specific condition is met, I want my checkboxes to automatically be checked through Javascript code in MVC. @if (str_item != "" && str_checkroles != "" && str_item == str_checkroles) { <script> src = "https://ajax.googl ...

javascriptrazor

Both of the radio buttons in Material-UI have been selected

I have a unique document that showcases an implementation of RadioGroup, FormControlLabel, and FormControl. Take a look at the code snippet below for reference. import React from 'react'; import PropTypes from 'prop-types'; import Radio ...

javascriptreactjsmaterial-uiredux-form

Steps to retrieve the latest value of a specific cell within the Material UI Data Grid

After updating the cell within the data grid, I encountered an issue where I could retrieve the ID and field using the prop selectedCellParams, but retrieving the modified value was proving to be challenging. In order to successfully execute the PUT reque ...

javascriptreactjsmaterial-uidatagrid

Procedures that are stored and utilize parameters

I am trying to create a Stored Procedure that takes a specific flight and date as input, and generates a customer call list with names, addresses, and phone numbers as output. The challenge I am facing is how to incorporate the input values into the quer ...

mysqlsqldatabasestored-procedures

Is there a way for me to add a new column within the 'where'

RETRIEVE *, CONCAT((',' + structure_section FROM tb_Structure WHERE CONCAT(',', parts, ',') LIKE CONCAT('%,', section_id, ',%') FOR XML PATH('')), 1, 1, '') AS updated_value from tb_Cli ...

sqlsql-serversql-server-2008select

Tips for dynamically adding an HTML element to index.html using JavaScript

https://i.sstatic.net/cAna8.png How can I use JavaScript to insert cards into a container or display them in flex? Currently, the cards are not displaying as desired. I attempted to insert data into the note class using insertAdjacentHTML in JavaScript, b ...

javascripthtml

"Patience is key when it comes to waiting for an HTTP response

Looking for a solution in AngularJS, I have a service that calls the backend to get some data. Here is how the service looks: app.factory('myService', ['$http', '$window', '$rootScope', function ($http, $window, $ro ...

javascriptangularjshttp

Not all API results are being displayed by the Nextjs API function

I am facing an issue with rendering all the returns from the API in my Next.js application. The API, which is created in Strapi, is only displaying 1 out of the 3 possible returns. I am a beginner when it comes to using APIs and I suspect that the issue li ...

javascriptapinext.jsstrapi

What is causing the continuous appearance of null in the console log?

As part of my JavaScript code, I am creating an input element that will be inserted into a div with the id "scripts" in the HTML. Initially, I add a value to this input field using JavaScript, and later I try to retrieve this value in another JavaScript fu ...

javascriptjqueryhtmljsonfunction

Is it possible for me to run two processes simultaneously on the same server - one to handle incoming requests and another to continuously poll a data storage system?

I am currently in the process of developing a server with two main functions: An API that can create, update, retrieve, and delete data in a local storage (similar to managing files on the server for a project prototype) A continuous loop that monito ...

javascriptnode.jsexpressserver

Using Restify to Serve CSS FilesLearn how to use Restify to

Currently, I am developing a node application that requires serving static files like JavaScript and CSS. In my project structure, I have the following layout: -MyProject -backend -index.js -frontEnd -index.html -js ...

javascriptnode.jsbackendrestify