Storing logs from Docker into MongoDB using Fluentd has been a fairly simple setup. However, my current struggle lies in retrieving these logs in the correct order while also implementing pagination.
The structure of a log document typically looks like this:
{
_id: ObjectId(...),
time: ISODate(...),
log: "message"
}
To display logs from newest to oldest with pagination support, my approach involves:
For the first page -
db.logs.find().sort({_id: -1}).limit(10)_idFor the next page -
db.logs.find({_id: {$lt: lastIdFromPreviousPage}).sort({_id: -1}).limit(10)and so on.
The challenge arises from the fact that according to the MongoDB documentation:
The relationship between the order of ObjectId values and generation time is not strict within a single second.
This presents an issue as it's not guaranteed that documents with lower _id values than lastIdFromPreviousPage are in the exact order they were written. Some logs from the previous page may be repeated in the result.
In comparison, MySQL's ordering by an auto_increment field ensures the results are correctly ordered. How can we achieve similar reliability in MongoDB? What approach should be followed?