What is the best way to loop through a MongoDB collection using mongojs?

Question

What is the best way to loop through a MongoDB collection using mongojs?

In my current project, I am utilizing the mongojs library and facing an issue while attempting to iterate through all elements in a collection.

index = 0

db.keys.find({}, {uid: 1, _id: 0}).forEach((err, key) =>
    if err?
        console.log err
    else 
        console.log (++index) + " key: " + key_uid

This code snippet logs:

1 key: bB0KN
2 key: LOtOL
3 key: 51xJM
4 key: x9wFP
5 key: hcJKP
6 key: QZxnE
.
.
.
96 key: EeW6E
97 key: wqfmM
98 key: LIGHK
99 key: bjWTI
100 key: 2zNGE
101 key: F71mL

However, the logging stops prematurely. Upon checking from the terminal using MongoDB, the command

> db.keys.count()
2317381

reveals that there are significantly more keys in the database that should be returned. Any suggestions on what might be causing this unexpected behavior?

javascript mongodb coffeescript iterator mongojs

Answer 1

Answer №1

It is important to utilize the each() method instead of forEach(). When using forEach(), it will iterate over every document in the batch, which defaults to 101. With each(), it will iterate over each document in the cursor directly. Referencing the documentation:

each

This method iterates through all the documents within the cursor. Similar to {cursor.toArray} not all elements will be processed if this cursor has already been accessed. If that's the case, {cursor.rewind} can be utilized to reset the cursor. Unlike {cursor.toArray}, this method only holds a maximum of batch size elements at one time when the batch size is specified. If not, it is up to the caller to ensure that the entire result fits into memory.

http://mongodb.github.io/node-mongodb-native/api-generated/cursor.html

Sample code snippet:

// Obtain a cursor
      var cursor = collection.find();

      // Use the each method, triggers for every document
      cursor.each(function(err, item) {

        // If item is null, then the cursor is empty and closed                  
        if(item == null) {

          // Indicate that the cursor is now closed
          cursor.toArray(function(err, items) {
            assert.ok(err != null);

            // Close the database connection
            db.close();
          });
        };
      });

Answer 2

It is important to utilize the each() method instead of forEach(). When using forEach(), it will iterate over every document in the batch, which defaults to 101. With each(), it will iterate over each document in the cursor directly. Referencing the documentation:

each

This method iterates through all the documents within the cursor. Similar to {cursor.toArray} not all elements will be processed if this cursor has already been accessed. If that's the case, {cursor.rewind} can be utilized to reset the cursor. Unlike {cursor.toArray}, this method only holds a maximum of batch size elements at one time when the batch size is specified. If not, it is up to the caller to ensure that the entire result fits into memory.

http://mongodb.github.io/node-mongodb-native/api-generated/cursor.html

Sample code snippet:

// Obtain a cursor
      var cursor = collection.find();

      // Use the each method, triggers for every document
      cursor.each(function(err, item) {

        // If item is null, then the cursor is empty and closed                  
        if(item == null) {

          // Indicate that the cursor is now closed
          cursor.toArray(function(err, items) {
            assert.ok(err != null);

            // Close the database connection
            db.close();
          });
        };
      });

Answer 3

Answer №2

The reason you are only able to see the first 101 documents is because that is the default number of documents that the MongoDB driver retrieves from the server in the initial batch.

Typically, the first batch will bring back 101 documents or enough documents to exceed a size of 1 megabyte. Subsequent batches are set at 4 megabytes each.

If you want to view more documents, you can use the find method and then loop through each document.

coll.find({}, {uid:1, _id : 0}, function(err, docs){
    if (err) {
        console.log(err);
        return;
    }
    docs.forEach(function(doc, index) { 
        console.log(index + " key: " + doc.uid) 
    });
});

Answer 4

The reason you are only able to see the first 101 documents is because that is the default number of documents that the MongoDB driver retrieves from the server in the initial batch.

Typically, the first batch will bring back 101 documents or enough documents to exceed a size of 1 megabyte. Subsequent batches are set at 4 megabytes each.

If you want to view more documents, you can use the find method and then loop through each document.

coll.find({}, {uid:1, _id : 0}, function(err, docs){
    if (err) {
        console.log(err);
        return;
    }
    docs.forEach(function(doc, index) { 
        console.log(index + " key: " + doc.uid) 
    });
});

What is the best way to loop through a MongoDB collection using mongojs?

Answer №1

Answer №2

Similar questions

The automated linting process through npm script did not meet the desired expectations

How can I compare equal values and determine whether they are true or false using MongoDB aggregation?

Troubleshooting "jest + enzyme + react16: The src attribute of <img> tag is not sending

Using ReactJS to strip HTML tags from JSON response

Make sure that the click event listener is set up on the anchor element so that it also affects its children

How to use jQuery to hide radio buttons that are not checked when clicking a submit

What is the best way to compare the values in a MongoDB array collection with an array of IDs?

The function Model.find({}) is coming back as empty despite the fact that there are records present in the

Connect the B-Button to an input file using BootstrapVue

Transfer a single property from a particular object in one array to another array of objects based on a condition (JavaScript ES6)

Can Ajax be utilized to call a PHP script that contains another Ajax function?

Maintaining the proportions of images in different screen sizes when resizing

AngularJS postback method fails to trigger

Flask does not provide a direct boolean value for checkboxes

The Star Rating System fails to update accurately after hiding the Radio Buttons

Placing a v-model within a Vue.js component

Trigger an event, pause, and subsequently trigger another one within Vue

What is the best way to handle multiple responses in Ajax within a single function?

Why does JSON.parse need to be run twice - once for a string and once for an object?

Tips for integrating CSS with Material UI TableContainer