How can I handle a situation where my database is extremely large and I require all the data to be loaded on the frontend from the get

I have encountered a challenge with my MongoDB collection as it has grown significantly large, now containing 15k documents totaling nearly 15 MB in size. The website I am developing utilizes a map to display all elements, making pagination difficult. Each time I request data from my express backend, I receive an error related to the javascript heap running out of memory. You can see a screenshot of this error here.

How can I ensure that my website continues to function correctly while addressing these issues:

  1. Coping with the large database size
  2. Incorporating pagination into the design of my website
  3. Preventing backend crashes resulting from the current layout

I apologize if this question seems naive; this is my first time seeking assistance on Stack Overflow.

javascriptmongodbexpressoptimizationweb-applications

Answer №1

Instead of implementing pagination, consider loading the data incrementally and visualizing it as it comes in, similar to a loading animation.

Here's one approach: when the app is loaded, show the map without any data first, then gradually render the chunks of incoming data.

You could also explore using technologies like websocket for real-time data updates.

Try to come up with a method to hash or paginate the data, even if it means using a simple solution like organizing it based on the first character of the ID.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Is there a way to conditionally upsert a document in MongoDB only if the existing document is older than the new one?

I currently have a MongoDB collection structured like this: { "id": _id_, "doc": _some data_, "timestamp": _timestamp_ } My objective is to either add a new entry if there is no existing one with the same id, or update the current entry if it alrea ...

mongodbmongodb-queryaggregation-framework

The data from the Subscribe API call is gradually loading within the ngOnInit() function

When using Angular 8, I am experiencing slow data retrieval when making API calls in the ngOnInit() function. The issue arises when trying to pass this data as @Input from one component module to another - it initially comes through as undefined for a minu ...

javascriptangular

Standardizing the file name in Internet Explorer 11 and generating a new file using JavaScript

While attempting to upload a file and normalize its name, I encountered an issue with IE11 not supporting the 'normalize' method. After some research, I discovered the 'unorm' polyfill which resolved the normalization problem. However, ...

javascriptangularjsfileblobinternet-explorer-11

Maintaining active navigation state in JQuery/JavaScript after clicking a link: tips and tricks

While many resources discuss adding the active class to a nav link using jquery, there is less information on maintaining the active state after the nav link has been clicked. After experimenting with code from various sources, I have attempted to set ses ...

javascriptjqueryhtmlcss

What is the best way to retrieve a value from a multiple select element being utilized by asmselect?

I have a form in my code that I am validating using a JavaScript function called sub(). Once the validation is successful, I am posting the data to a PHP file using $.POST. I am able to fetch the name field using its id, but I am facing issues with fetchin ...

javascriptjquery-uijqueryjquery-plugins

Get the zip file through a RESTful web service with the help of AngularJS

Utilizing REST jersey on the server side combined with AngularJS on the client side. My task involves downloading a zip file requested by the client for a specific date range. Server-side code: //Currently, I have hardcoded a single zip file for testing ...

javascriptjavaangularjsrestjersey

Is it possible to use Gulp.js to serve src files without browserSync enabled?

Started a fresh project using Yeoman Gulp-Angular generator. I grasp the concept of BrowserSync, but I simply cannot fathom why anyone would put up with their network requests being overwhelmed by it: I am inclined to eliminate BrowserSync from my projec ...

javascriptnode.jsgulpgulp-watchbrowser-sync

Transition the object in smoothly after the slide has changed

How can I make the h4 tags fade in after the divs slide into view, while also adding the class "current" to each visible slide? Check out the example on JSFiddle here. <div class="slider"> <div class="slides"> <div class="slide ...

javascriptjquerycss

The Angular application must remain logged in until it is closed in the browser

Currently, my Angular app includes authentication functionality that is working smoothly. The only issue is that the application/session/token expires when there is no user activity and the app remains open in the browser. I am looking for a solution wher ...

javascriptangularjs

Tips for interpreting the JSON data returned by a Node server in Angular

When trying to implement a login form in my Angular frontend, I encountered an issue with reading the JSON object returned from my node.js server. Despite following the correct steps, the console displays "undefined" as if it cannot recognize the format. ...

javascriptangularjsjsonnode.js

Merging two distinct arrays of objects in JavaScript can be achieved by utilizing various methods and

I have a challenge where I need to merge two arrays of objects in a nested way. var array1=[{ PersonalID: '11', qusetionNumber: '1', value: 'Something' }, { PersonalID: '12', qusetionNumber: '2& ...

javascriptnode.jsjavascript-objects

Including an anchor element with a specified URL, alongside passing the URL as a property

Having trouble passing a URL to href using a property. I'm attempting to pass the {props.github} value to href, but it's not working as expected. I've set up a property object with a field called github like this: export const projectList ...

javascriptreactjs

Hovering over an image and trying to rotate it results in

Check out my rotating image example on hover in React here This effect utilizes scale(), rotate(), and transition properties to create an animated rotation when hovering over the parent element. Additionally, overflow: hidden is applied to the parent elem ...

javascripthtmlcssreactjshover

javascript utilizing key inputs

Is there a way I can create a function that activates when the "A" key on my keyboard is pressed, sending a signal to nupp('/TS'), and stops sending the signal when the "A" key is released? <html> <head> <script src=\"htt ...

javascriptajaxbuttonkeyboardkeydown

Ways to extract data from form inputs with the help of jQuery

My HTML form allows users to enter data and upon submission, I need to use jQuery to capture the entered values. <form class="my-form needs-validation" method="POST"> <input type="text" id="firstName" name="First Name"> <input type="tex ...

javascriptjquery

Expanding and shrinking the index within a specific circular boundary in JavaScript

I'm dealing with a circular range of ASCII values from a to z, where each letter corresponds to a number between 97 and 122. I have no issue with increasing the value within this range, but I am struggling when it comes to decreasing it. For example ...

javascriptnode.js

Styling a textbox within a gridview using JavaScript

Looking for a way to format a textbox within a gridview using JavaScript? Take a look at how the gridview columns are set up: <Columns> <asp:TemplateField> <ItemTemplate><asp:ImageButton runat="server" id="imgbtnEnterAmt" ...

javascriptc#asp.net

How can I include dynamic attributes in an HTML tag using ejs?

When using express.js + ejs, I encounter two scenarios: 1. <a href="<%= prevDisabledClass ? '' : ?page=<%=+page - 1%>%>">prev</a> Unfortunately, this code throws an error: Could not find matching close tag for "<%=". ...

expressejs

Error in Formatting Labels in CSS

I currently have a form with textboxes that include validation. Everything works smoothly when clicking the save button for the first time, but if we fill out the textboxes and then remove the text, an error message is displayed on the textboxes. You can v ...

javascriptcss

Is it possible to store numerical values with precision up to two decimal places in MongoDB?

Currently, I am immersed in a financial project that involves storing amounts for various transactions. My dilemma lies in selecting the most suitable Schema type for the amount field. Initially, my inclination was towards utilizing the Number type with ro ...

node.jsmongodbmongoose