Preserving the information from a webpage by utilizing casperjs for scraping and saving table data

Question

Preserving the information from a webpage by utilizing casperjs for scraping and saving table data

What is the most efficient way to preserve table data collected during a web scraping process with casperjs?

Saving it as a file after serializing into a json object.
Sending an ajax request to php and then storing it in a mysql database.

javascript mysql json phantomjs casperjs

Answer 1

Answer №1

In my approach, I opt for the second scenario:

Initial step: Retrieve information from a globalInfo variable

var globalInfo;
casper.thenOpen("www.targetpage.cl/valuableInfo", function() {
    globalInfo = this.evaluate(function(){
       var domInfo = {};
       domInfo.title = "this is the info";
       domInfo.body  = "scrap in the dom for info";
       return domInfo;
   });
});

Next step: Navigate to a webpage to save the extracted data

casper.then(function(){
   casper.thenOpen("www.mipage.com/saveIntheDBonPost.php", {
      method: 'post',
      data:{              
          'title': ''+globalInfo.title,
          'body': ''+globalInfo.body
      }
   });
});

The URL

www.mipage.com/saveIntheDBonPost.php

processes the data using the $_POST parameter and stores it in a database.

Answer 2

In my approach, I opt for the second scenario:

Initial step: Retrieve information from a globalInfo variable

var globalInfo;
casper.thenOpen("www.targetpage.cl/valuableInfo", function() {
    globalInfo = this.evaluate(function(){
       var domInfo = {};
       domInfo.title = "this is the info";
       domInfo.body  = "scrap in the dom for info";
       return domInfo;
   });
});

Next step: Navigate to a webpage to save the extracted data

casper.then(function(){
   casper.thenOpen("www.mipage.com/saveIntheDBonPost.php", {
      method: 'post',
      data:{              
          'title': ''+globalInfo.title,
          'body': ''+globalInfo.body
      }
   });
});

The URL

www.mipage.com/saveIntheDBonPost.php

processes the data using the $_POST parameter and stores it in a database.

Answer 3

Answer №2

To put it simply, think of CasperJS as a tool to gather data and then process it in another programming language. My recommendation would be to opt for the first choice - extract the data in JSON format and store it in a file for future analysis.

You can achieve this by utilizing the File System API offered by PhantomJS. Additionally, you can combine this with CasperJS's command-line interface to pass arguments to your script (such as specifying a temporary file for output).

The script to manage this process would involve:

Generating a temporary file path (e.g., using `mktemp` on Linux).
Executing your CasperJS script while passing the temporary file path as an argument.
Retrieving the data, writing it to the specified file using the File System API, and concluding the script.
Reading the file, performing tasks on the data (like storing it in a database), and deleting the temporary file afterward.

Answer 4

To put it simply, think of CasperJS as a tool to gather data and then process it in another programming language. My recommendation would be to opt for the first choice - extract the data in JSON format and store it in a file for future analysis.

You can achieve this by utilizing the File System API offered by PhantomJS. Additionally, you can combine this with CasperJS's command-line interface to pass arguments to your script (such as specifying a temporary file for output).

The script to manage this process would involve:

Generating a temporary file path (e.g., using `mktemp` on Linux).
Executing your CasperJS script while passing the temporary file path as an argument.
Retrieving the data, writing it to the specified file using the File System API, and concluding the script.
Reading the file, performing tasks on the data (like storing it in a database), and deleting the temporary file afterward.

Preserving the information from a webpage by utilizing casperjs for scraping and saving table data

Answer №1

Answer №2

Similar questions

Ways to insert a line break using ajax

The issue of onClick failing to function when paired with the addEventListener for the

What is the process of transforming an xhr request into an angular $http request?

Arranging images next to each other using CSS and HTML

Tips for toggling the appearance of like and add to cart icons

Ways to access properties beyond the top-level object

Why is it not possible to substitute table names in PDO prepared statements?

Is there a way to extract the unicode/hex representation of a symbol from HTML using JavaScript or jQuery?

Implement the Pydantic V2 JSON serialization logic within any class of your choosing

Removing duplicate values in Vue after sorting

Facing a Type Error while trying to utilize the Ember Embedded Records Mixin in conjunction with keyForAttribute

Managing Data Types in AJAX Requests

Angular implementation of a dynamic vertical full page slider similar to the one seen on www.tumblr

React is failing to display identical values for each item being mapped in the same sequence

Setting up a recurring task with a while loop in a cron job

The element "Footer" cannot be found in the file path "./components/footer/Footer"

A guide to submitting forms within Stepper components in Angular 4 Material

Transform yaml strings into JSON entities

"Is there a way to retrieve the props that have been passed down to a

The $OnChange function fails to activate when passing an object by reference