When dealing with a large number of records, the performance of querying with MongoDB in a Meteor.js application may decrease due to the high volume

I have a project in meteor js with a large amount of data stored in a mongodb database, currently around 50,000 messages. I am facing issues with the performance of my application as it is taking too long to render or load the data from the database. Additionally, whenever there is any activity such as fetching messages, the app retrieves the data from the database again.

Template.messages.helper({
    linkMessages() {
        var ids = _.pluck(Meteor.user().subscription, '_id');
        var messages = Messages.find({ $or: [{ feedId: { $exists: false }, link: { $exists: true } }, { feedId: { $in: ids }, link: { $exists: true } }] }, { sort: { timestamp: 1 }, limit: Session.get("linkMessageLimit") }).fetch();
        return messages;
    }

})

Calling publication in the onCreate method:

Template.roomView.onCreated(function() {
    const self = this;
    Deps.autorun(function() {
        Meteor.subscribe('messages', Session.get('currentRoom'), Session.get('textMessageLimit'), {
            onReady() {
                isReady.messages = true;
                if (scroll.needScroll) {
                    scroll.needScroll = false;
                    if (scroll.previousMessage) {
                        Client.scrollToMessageText(scroll.previousMessage);
                    }
                } else {
                    Meteor.setTimeout(function() {
                        Client.scrollChatToBottomMsg();
                    }, 1000)
                }
            }
        });

    });
});`

The publication function on the server:

Meteor.publish('messages', function(roomId, limit) {
    check(roomId, String);
    check(limit, Match.Integer);

        let query = { $or: [{ link: {$exists: false} }, { feedId: { $exists: false } }] };

        const thresholdMessage = Messages.findOne(query, { skip: limit, sort: { timestamp: 1 } });

        if (thresholdMessage && thresholdMessage.timestamp) {
            query.timestamp = { $gt: thresholdMessage.timestamp };
        }

        return Messages.find(query, { sort: { timestamp: -1 } });

});

Answer №1

It's not advisable to let mini-mongodb store an extensive amount of data. Even though Meteor JS handles this well, it may still be slow considering factors like network traffic and bandwidth.

Whether you're dealing with unlimited scrolling or simple pagination, I recommend using pagination. I have a proven solution that works flawlessly, which you can find here with the complete code for pagination.

My pagination method is optimized for server performance. The collection publish is restricted by the limit set during subscription.

Please note that there isn't a comprehensive solution available yet for tables with search, pagination, and other advanced features. It might be best to create your own solution tailored to your specific needs.

ADDITIONAL INSIGHTS:

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Encountering a bindings issue when trying to utilize libxml-xsd within an electron application

Seeking guidance on validating an XML file against its schema within an electron application. Prior to adding the 'libxml-xsd' require statement to my angular2 service, it functioned properly. However, upon inclusion of this statement: const xs ...

Automatically sending form submission after 5 seconds

I'm facing an issue with submitting a form after the page loads and 5 seconds have passed. I've tried using setTimeout but it doesn't seem to be working. Can anyone suggest why this might be happening? jQuery and delay() also don't work ...

The computation outcome is not being displayed within the <h2> tag. Issue may be related to either

I am currently working on a percentage calculator. The calculation is functioning properly but the result is only being displayed in the input. I would like the result to also be shown in the h2. Is there a way to achieve this? $(document).on("change ke ...

What is the reason scopes remain intact even after $destroy is activated?

After creating a custom directive that generates a dialog upon click event using jQuery, I encountered an issue where the scopes were not properly cleaned up after closing the dialog. The problem is evident in the fact that 167 ChildScopes are retained, co ...

Struggling to concentrate using jQuery?

When the search icon is clicked, I want the focus to be on the input so the user can start typing right away without having to manually click on it. Although I tried using focus(), it doesn't seem to work for this particular input. $('.header__ic ...

Creating subarrays with identical strings from a given array: step-by-step guide

I currently have an array structured as follows : [ { "title": "name", "value": "" }, { "title": "version", "value": "" }, { "title": "inventory_name", "value": "" }, { "title": "inventory_version", "value": "" }, { "ti ...

How about diving into some JavaScript practice?

As I embark on my journey to teach myself JS with the help of a less-than-stellar textbook, I find myself stumped by a challenge that requires deciphering the purpose of this code: function clunk (times) { var num = times; while (num > 0) { ...

How to dismiss a JavaScript alert without refreshing the page in Codeigniter 3

I am currently working on building a form with validation using both JavaScript and Codeigniter 3.04. Here is the HTML code for my form: <form class="form-horizontal" method="POST" onsubmit="check()"> //code </form> My goal is to validate t ...

Experiencing a "non-serializable value found in the state" error specifically while utilizing redux toolkit, but this issue does not occur with traditional redux implementations

While transitioning my app to utilize Redux Toolkit, I encountered an error after switching over from createStore to configureStore: A non-serializable value was found in the state at path: `varietals.red.0`. Value:, Varietal { "color": "red", "id": " ...

Refreshing the page in Angular when switching route parameters

I am currently working on a web application that consists of multiple pages with tabs within one of them. To navigate between the tabs, I have implemented routes using the ng-view directive for the parent page and an ng-switch within a small div to display ...

Placing a list item at the beginning of an unordered list in EJS with MongoDB and Node.js using Express

What I've done: I already have knowledge on how to add an LI to UL, but it always goes to the bottom. What I'm trying to achieve: Add an LI to the top so that when my div.todos-wrapper (which has y-oveflow: hidden) hides overflow, the todos you a ...

Unable to retrieve the unprocessed value (including periods) from an HTML number input

Whenever I press a key on the <input type="number" id="n" /> and then type a ., it appears in the input but does not show up in $('#n').val(). For instance, if I type 123., the result from $('#n').val() is just 123. Is there a w ...

The $.ajax request encounters an error when trying to parse the data as JSON

I am encountering an issue where my ajax call to a JSON file fails when using dataType: "json". However, changing it to "text" allows the ajax call to succeed. See the code snippet below: $.ajax({ type: "POST", url: url, dataType: "json", ...

Text color-coded for specific hues

I am looking to change the color of a specific text within HTML/PHP/CSS/JS code. Here is an example of the text structure: @john please move the invite to the next week. My goal is to make the word "@john" appear in green, while keeping the rest of the ...

How to update or replace a complex nested object in MongoDB using Spring Data MongoDB

Hello, I am working with a specific type of document. { "id": "5a212b985735dd44089e4782", "people": [ { "personId": "5a212b985735dd44089e4783", "name": "Ronaldo", ...

"Troubleshooting: React Bootstrap Modal not appearing on screen

I am facing an issue with rendering a Bootstrap modal in my React app when a button is clicked. Here is the code snippet for my React app: import React, { Component } from 'react'; import { Grid, Row, Col, Button, Table } from 'react-boots ...

Enhance Data3 Sankey to disperse data efficiently

There are a few instances where the D3 Sankey spread feature is showcased here. However, it seems that this specific function is not included in the official D3 Sankey plugin. Is there anyone who can assist me in obtaining the code for the Spread function ...

How can I transform a CSS3D rendering plane into a floor in Three.js?

For my project, I need to create a CSS3D rendering plane that acts as a floor with a 3D cube positioned on top of the plane. I have attempted to achieve this by sharing the same scene and camera between the plane and cube in the code below. However, I face ...

Experiencing issues with organizing and displaying data using backbone.js

Experiencing some issues with sorting and displaying data using backbone.js. The collection is sorted by 'title' in the comparator function, but when rendering begins, the views of models are displayed in a different order. var TodoList = Ba ...

Displaying error message if the size of the uploaded file surpasses the maxRequestLength limit

In the web.config file, I have set the maximum request length to 10240 and execution timeout to 30: <httpRuntime maxRequestLength="10240" executionTimeout="30" /> Using valum's AjaxUpload plugin on the client side, I encountered an issue with ...