Executing background operations in Meteor.js

Question

Executing background operations in Meteor.js

Let me lay out my situation:

1. Extracting data from example.com at regular intervals
2. Storing it in a Mongodb database
3. Subscribing to this data in a Meteor App.

Since I'm still learning Meteor, here's my plan:

1. Develop a scraper script for example.com using Python or PHP.
2. Set up a cronjob to run the script at specified intervals.
3. Store the extracted data in Mongodb.

Is it feasible to accomplish this entirely using Meteor without relying on Python or PHP? How can I manage a task that needs to be executed every X minutes?

javascript meteor

Answer 1

Answer №1

There are specialized systems like percolate:synced-cron available for Meteor developers. With this, you have the ability to schedule tasks using Later.js syntax, as seen in this example from the percolate:synced-cron documentation:

SyncedCron.add({
  name: 'Crunch some important numbers for the marketing department',
  schedule: function(parser) {
    // parser object enables flexible scheduling
    return parser.text('every 2 hours');
  }, 
  job: function() {
    var numbersCrunched = CrushSomeNumbers();
    return numbersCrunched;
  }
});

Alternatively, if you prefer utilizing an OS-level cron job, you can set up an HTTP endpoint within your Meteor.js application that can be triggered using curl at a designated time.

Answer 2

There are specialized systems like percolate:synced-cron available for Meteor developers. With this, you have the ability to schedule tasks using Later.js syntax, as seen in this example from the percolate:synced-cron documentation:

SyncedCron.add({
  name: 'Crunch some important numbers for the marketing department',
  schedule: function(parser) {
    // parser object enables flexible scheduling
    return parser.text('every 2 hours');
  }, 
  job: function() {
    var numbersCrunched = CrushSomeNumbers();
    return numbersCrunched;
  }
});

Alternatively, if you prefer utilizing an OS-level cron job, you can set up an HTTP endpoint within your Meteor.js application that can be triggered using curl at a designated time.

Answer 3

Answer №2

If you're looking for a solution to schedule background jobs in Meteor, I recommend checking out the package called Steve Jobs. You can find it here.

With Steve Jobs, you are able to easily perform actions like register, replicate, and remove.

// Registering a job
Jobs.register({ 
    dataScraper: function (arg) {
        var data = getData()

        if (data) {
            this.replicate({
                in: {
                    minutes: 5
                }
            });

            this.remove(); // or use this.success(data)
        } else {
            this.reschedule({
                in: {
                    minutes: 5
                }
            })
        }
    }
})

// Scheduling the job on Meteor startup
// Using `singular` to ensure only one pending job with the same configuration
Meteor.startup(function () {
    Jobs.run("dataScraper", {
        in: {
            minutes: 5
        }, 
        singular: true
    })
})

After running the job, you have the option to store the result in the database, keep it in the jobs history, or remove it altogether based on your preference.

Answer 4

If you're looking for a solution to schedule background jobs in Meteor, I recommend checking out the package called Steve Jobs. You can find it here.

With Steve Jobs, you are able to easily perform actions like register, replicate, and remove.

// Registering a job
Jobs.register({ 
    dataScraper: function (arg) {
        var data = getData()

        if (data) {
            this.replicate({
                in: {
                    minutes: 5
                }
            });

            this.remove(); // or use this.success(data)
        } else {
            this.reschedule({
                in: {
                    minutes: 5
                }
            })
        }
    }
})

// Scheduling the job on Meteor startup
// Using `singular` to ensure only one pending job with the same configuration
Meteor.startup(function () {
    Jobs.run("dataScraper", {
        in: {
            minutes: 5
        }, 
        singular: true
    })
})

After running the job, you have the option to store the result in the database, keep it in the jobs history, or remove it altogether based on your preference.

Executing background operations in Meteor.js

Answer №1

Answer №2

Similar questions

How can I retrieve the value from req.body following an AJAX POST request on the server side with Express?

Developing a new React application with Access Control List (ACL) and encountering an issue with Casl

What is the best way to create a continuous loop of images on a never-ending

Using various colors to highlight specific sections of a letter or number

Modify the conditions of a CSS file in Bootstrap using JavaScript

Stuck with the same icon even after a successful AJAX call

Exploring the Dynamic Connection: AngularJS and MongoDB

Customize the CSS for MaterialUI's TablePagination's MenuItem

Is it possible to merge JavaScript files exclusively using encore?

What could be the reason for the inability to generate the response using response.writeHead and response.write functions?

Display a message if the local storage is empty

Retrieve the index of the item that has been selected in a dropdown list

Executing Two Distinct JQuery Api Requests

Is there a way in Python to translate JavascriptSerializer into a datetime object?

Trigger a JavaScript function using PHP and retrieve the output

The value is currently unset in the TypeScript language

The name 'withStyles' is nowhere to be found

encountering issue alerts when using the MUI TextField module alongside the select function

Is the value of the object in the array already present in another array of objects?

Display issue with React TypeScript select field