My current challenge involves working with rxJS and handling bulk HTTP requests from a database containing over 1 million documents.
The code I have is relatively simple. I am pushing all the documents from the collection into an array called "allPlayers" and then making bulk HTTP requests in batches of 20 to an API. While the code is functional, I believe it's time to refactor it from its current state:
const cursor = players_db.find(query).lean().cursor();
cursor.on('data', function(player) { allPlayers.push(player); });
cursor.on('end', function() {
logger.log('warng',`S,${allPlayers.length}`);
from(allPlayers).pipe(
mergeMap(player => getPlayer(player.name, player.realm),20),
).subscribe({
next: player => console.log(`${player.name}@${player.realm}`),
error: error => console.error(error),
complete: () => console.timeEnd(`${updatePlayer.name}`),
});
});
Currently, I am using the 'find' method with 'cursor' and a 'batchSize', but based on my understanding from the length property, and the information from this related question on stack overflow: {mongoose cursor batchSize}, it seems like 'batchSize' is mainly for optimization and not for returning an array of X documents.
So, what should be my next steps and which operator in rxJS should I consider?
One approach could be to create arrays with the required length (e.g., 20) and then pass them to rxJS, similar to my previous usage. However, there might be another way to incorporate rxJS within the existing 'for promise loop'.
const players = await players_db.find(query).lean().cursor({batchSize: 10});
for (let player = await players.next(); player != null; player = await players.next()) {
// implement RxJS functionality within this loop
}
I also came across this question {Best way to query all documents from a mongodb collection in a reactive way w/out flooding RAM}, which addresses a similar issue. While I grasp the concept, I am unsure about the syntax. I'm aware that the 'cursor' variable doesn't directly provide useful data, or could it?
- The 'bufferCount' operator in rxJS seems intriguing.
- Could this solution {https://gist.github.com/wellcaffeinated/f908094998edf54dc5840c8c3ad734d3} be applicable?