How to incorporate paginating in MongoDB operations

Question

How to incorporate paginating in MongoDB operations

It's a well-known fact that using skip for pagination can lead to performance issues, especially as the data set grows larger. One workaround is to leverage natural ordering using the _id field:

//Page 1
db.users.find().limit(pageSize);
//Get the id of the last document on this page
last_id = ...

//Page 2
users = db.users.find({'_id'> last_id}). limit(10);

The challenge I'm facing is that as a newbie to MongoDB, I'm uncertain about the best approach to retrieve this crucial last_id

javascript mongodb pagination mongodb-query

Answer 1

Answer №1

The idea being discussed can be referred to as "forward paging". One important distinction is that unlike the use of .skip() or .limit() modifiers, this method does not allow for going back to a previous page or jumping directly to a specific page without significant effort in storing "seen" or "discovered" pages. If your goal involves this type of "links to page" paging, then it is recommended to stick with the .skip() and .limit() approach despite any performance limitations.

If you are open to only "moving forward", here is the fundamental concept:

db.junk.find().limit(3)

{ "_id" : ObjectId("54c03f0c2f63310180151877"), "a" : 1, "b" : 1 }
{ "_id" : ObjectId("54c03f0c2f63310180151878"), "a" : 4, "b" : 4 }
{ "_id" : ObjectId("54c03f0c2f63310180151879"), "a" : 10, "b" : 10 }

These are the first three items on your initial page. You can iterate over the cursor using the following code snippet:

var lastSeen = null;
var cursor = db.junk.find().limit(3);

while (cursor.hasNext()) {
   var doc = cursor.next();
   printjson(doc);
   if (!cursor.hasNext())
     lastSeen = doc._id;
}

This loop iterates the cursor, performing actions, and stores the value of the present _id under lastSeen when the end of the cursor is reached:

ObjectId("54c03f0c2f63310180151879")

In subsequent iterations, you feed this stored _id value back into the query:

var cursor = db.junk.find({ "_id": { "$gt": lastSeen } }).limit(3);

while (cursor.hasNext()) {
   var doc = cursor.next();
   printjson(doc);
   if (!cursor.hasNext())
     lastSeen = doc._id;
}

{ "_id" : ObjectId("54c03f0c2f6331018015187a"), "a" : 1, "b" : 1 }
{ "_id" : ObjectId("54c03f0c2f6331018015187b"), "a" : 6, "b" : 6 }
{ "_id" : ObjectId("54c03f0c2f6331018015187c"), "a" : 7, "b" : 7 }

This process repeats until no further results can be obtained.

This serves as a basic guide for natural order paging like _id. For other scenarios, the process becomes more intricate. Review the following example:

{ "_id": 4, "rank": 3 }
{ "_id": 8, "rank": 3 }
{ "_id": 1, "rank": 3 }    
{ "_id": 3, "rank": 2 }

To split these entries into two pages sorted by rank, you must keep track of what has been "seen" and exclude those records. For the first page:

var lastSeen = null;
var seenIds = [];
var cursor = db.junk.find().sort({ "rank": -1 }).limit(2);

while (cursor.hasNext()) {
   var doc = cursor.next();
   printjson(doc);
   if ( lastSeen != null && doc.rank != lastSeen )
       seenIds = [];
   seenIds.push(doc._id);
   if (!cursor.hasNext() || lastSeen == null)
     lastSeen = doc.rank;
}

{ "_id": 4, "rank": 3 }
{ "_id": 8, "rank": 3 }

In the next iteration, you want ranks less than or equal to the last seen rank while excluding previously viewed documents. Employ the $nin operator for this purpose:

var cursor = db.junk.find(
    { "_id": { "$nin": seenIds }, "rank": "$lte": lastSeen }
).sort({ "rank": -1 }).limit(2);

while (cursor.hasNext()) {
   var doc = cursor.next();
   printjson(doc);
   if ( lastSeen != null && doc.rank != lastSeen )
       seenIds = [];
   seenIds.push(doc._id);
   if (!cursor.hasNext() || lastSeen == null)
     lastSeen = doc.rank;
}

{ "_id": 1, "rank": 3 }    
{ "_id": 3, "rank": 2 }

The number of "seenIds" retained depends on the granularity of your results where this value is likely to change. In this case, you can reset the seenIds list if the current rank differs from the lastSeen, preventing unnecessary growth.

These are the core concepts of "forward paging" for practical learning purposes.

Answer 2

The idea being discussed can be referred to as "forward paging". One important distinction is that unlike the use of .skip() or .limit() modifiers, this method does not allow for going back to a previous page or jumping directly to a specific page without significant effort in storing "seen" or "discovered" pages. If your goal involves this type of "links to page" paging, then it is recommended to stick with the .skip() and .limit() approach despite any performance limitations.

If you are open to only "moving forward", here is the fundamental concept:

db.junk.find().limit(3)

{ "_id" : ObjectId("54c03f0c2f63310180151877"), "a" : 1, "b" : 1 }
{ "_id" : ObjectId("54c03f0c2f63310180151878"), "a" : 4, "b" : 4 }
{ "_id" : ObjectId("54c03f0c2f63310180151879"), "a" : 10, "b" : 10 }

These are the first three items on your initial page. You can iterate over the cursor using the following code snippet:

var lastSeen = null;
var cursor = db.junk.find().limit(3);

while (cursor.hasNext()) {
   var doc = cursor.next();
   printjson(doc);
   if (!cursor.hasNext())
     lastSeen = doc._id;
}

This loop iterates the cursor, performing actions, and stores the value of the present _id under lastSeen when the end of the cursor is reached:

ObjectId("54c03f0c2f63310180151879")

In subsequent iterations, you feed this stored _id value back into the query:

var cursor = db.junk.find({ "_id": { "$gt": lastSeen } }).limit(3);

while (cursor.hasNext()) {
   var doc = cursor.next();
   printjson(doc);
   if (!cursor.hasNext())
     lastSeen = doc._id;
}

{ "_id" : ObjectId("54c03f0c2f6331018015187a"), "a" : 1, "b" : 1 }
{ "_id" : ObjectId("54c03f0c2f6331018015187b"), "a" : 6, "b" : 6 }
{ "_id" : ObjectId("54c03f0c2f6331018015187c"), "a" : 7, "b" : 7 }

This process repeats until no further results can be obtained.

This serves as a basic guide for natural order paging like _id. For other scenarios, the process becomes more intricate. Review the following example:

{ "_id": 4, "rank": 3 }
{ "_id": 8, "rank": 3 }
{ "_id": 1, "rank": 3 }    
{ "_id": 3, "rank": 2 }

To split these entries into two pages sorted by rank, you must keep track of what has been "seen" and exclude those records. For the first page:

var lastSeen = null;
var seenIds = [];
var cursor = db.junk.find().sort({ "rank": -1 }).limit(2);

while (cursor.hasNext()) {
   var doc = cursor.next();
   printjson(doc);
   if ( lastSeen != null && doc.rank != lastSeen )
       seenIds = [];
   seenIds.push(doc._id);
   if (!cursor.hasNext() || lastSeen == null)
     lastSeen = doc.rank;
}

{ "_id": 4, "rank": 3 }
{ "_id": 8, "rank": 3 }

In the next iteration, you want ranks less than or equal to the last seen rank while excluding previously viewed documents. Employ the $nin operator for this purpose:

var cursor = db.junk.find(
    { "_id": { "$nin": seenIds }, "rank": "$lte": lastSeen }
).sort({ "rank": -1 }).limit(2);

while (cursor.hasNext()) {
   var doc = cursor.next();
   printjson(doc);
   if ( lastSeen != null && doc.rank != lastSeen )
       seenIds = [];
   seenIds.push(doc._id);
   if (!cursor.hasNext() || lastSeen == null)
     lastSeen = doc.rank;
}

{ "_id": 1, "rank": 3 }    
{ "_id": 3, "rank": 2 }

The number of "seenIds" retained depends on the granularity of your results where this value is likely to change. In this case, you can reset the seenIds list if the current rank differs from the lastSeen, preventing unnecessary growth.

These are the core concepts of "forward paging" for practical learning purposes.

Answer 3

Answer №2

A straightforward method for incorporating pagination within MongoDB

  // Pagination
  const currentPage = parseInt(req.query.page, 10) || 1;
  const itemsPerPage = parseInt(req.query.limit, 10) || 25;
  const startingIndex = (currentPage - 1) * itemsPerPage;
  const endingIndex = currentPage * itemsPerPage;
  query = query.skip(startingIndex).limit(itemsPerPage);

Answer 4

A straightforward method for incorporating pagination within MongoDB

  // Pagination
  const currentPage = parseInt(req.query.page, 10) || 1;
  const itemsPerPage = parseInt(req.query.limit, 10) || 25;
  const startingIndex = (currentPage - 1) * itemsPerPage;
  const endingIndex = currentPage * itemsPerPage;
  query = query.skip(startingIndex).limit(itemsPerPage);

How to incorporate paginating in MongoDB operations

Answer №1

Answer №2

Similar questions

Tips for optimizing SEO by efficiently indexing data from client-side API requests

A guide to using JavaScript to retrieve a CSV file in HTML and store its contents in an array

Encountering Issues with Laravel AJAX POST Request (500 Internal Server Error)

Quickly select a stand-out team from a pool of candidates

Adjust the height automatically in extjs hbox layout with this quick guide

Ways to detect scrolling activity on the v-data-table module?

One requirement for a directive template is that it must contain only a single root element, especially when using the restrict option to E

Error encountered during Heroku deployment: "sh: 1: tailwind: not found"

Establishing a minimum date based on the date selected in the earlier datepicker

Achieving successful content loading through AJAX using the .load function

Combine 2 queries using RTK Query

The Challenge of Referencing Javascript Files (including jQuery)

the function is not correctly displaying elements on the page through JavaScript

Dealing with multiple parameters using React Router

Count the number of documents in a CommandCursor using Mongodb's count()

One-way communication between two clients using Socket.io

What is the solution to the error "modal is not defined" in Vue.js?

I am looking to tally up multiple inputs in JavaScript and then submit them to a MySQL database

The AngularJS templates' use of the ternary operator

Wait to execute until the external script in Vue.js has finished loading