Enhance the efficiency of iterating through a vast array of objects

Question

Enhance the efficiency of iterating through a vast array of objects

I am currently dealing with a large JSON file that is 7MB in size, containing 11000 objects. Each object consists of various items structured like this:

[  
   {  ...
      "startingTime":"Date1",
      "endingTime":"Date2"
      ...
   }
]

Within my HTML, there is a slider featuring daily Hours:Minutes.

As the user interacts with the slider, I need to update the DOM by filtering the data from the file to find:

StartingTime <= SelectedTimeOnSlider <= EndingTime

The filtered data is then utilized to display SVG shapes on a map (tubeMap + trains), ensuring that the previous shapes are removed from the DOM using remove() before drawing new ones.

The issue arises when filtering all objects each time the user slides the cursor, causing the UI to become non-fluid.

Therefore, I experimented with an alternative approach: I employed a backend to generate a new JSON in the following format:

[  
   {  
      "00:00":{  
         "data":""
      },
      "00:01":{  
         "data":""
      },
      "00:02":{  
         "data":""
      }
   }
]

This new JSON is subsequently sent to the frontend. By having the backend perform the initial filtering calculations, when the user selects a specific daytime on the slider, I no longer need to filter through all objects again in real-time; instead, I select the ObjectKey corresponding to selectedTimeOnSlider from the new format and add the data to the DOM.

However, I encountered delays while performing these computations upstream (taking more than 25 seconds from 00:00 to 02:00), which caused the frontend to wait excessively for the data.

Thus, I am seeking advice on how to enhance the performance of the filtering process or explore alternative approaches (such as indexing) to address this issue.

For context, the large JSON file is dynamic and varies based on the selected day on a calendar.

Update Below is the code snippet used for filtering the data:

let formatTrips = function (data:any) {
    var arr = [], i, j;
    for(i=0; i<24; i++) {
        for(j=0; j<60; j++) {
            arr.push((i<=9 ? "0"+i : i )+ ":" + (j<=9 ? "0"+j : j) );
        }
    }
    var start = new Date().getTime();
    var arrFormated = arr.map(time=> {
        let schedulesFiltered=[];
        let nowFormat = moment(time, 'HH:mm');
         schedulesFiltered= data.filter(elm => {
        let before = moment(elm.tp_org_r, 'HH:mm:ss'); //tp_org_r = startingTime
        let after = moment(elm.tp_des_r, 'HH:mm:ss');//tp_des_r = endingTime

        let checkifBetween = nowFormat.isBetween(before, after);

        if (checkifBetween) {
            return elm;
        }
        });
        //console.log(schedulesFiltered)
        return schedulesFiltered

    })
    var end = new Date().getTime();
    var time = end - start;
    console.log('Execution time: ' + time);

}

Additional information:

The HTTP request took 9 seconds to retrieve the file, but the main concern lies in the time-consuming filtering process involving a loop of 24*60 on an array of 11000+ object items.
In my initial approach, I load the JSON file once, saving it in a store (ngrx Angular store), and then apply filters for every change in slide value.

javascript json performance

Answer 1

Answer №1

the main objective is to display the movement of trains on a map

This is where we should begin:

Organize the timing by train. This task can be accomplished in O(n) without increasing the dataset size (it can also be done on the backend if necessary, see below for more details).

How does this help? It significantly reduces the size of the dataset that needs filtering. The 11000 events likely pertain to only a few hundred trains.

Next step, create the following initially:

A Map linking train IDs to respective trains currently displayed
A sorted array of trains based on departure time
A sorted array of trains based on arrival time
The position of the last train departed at the current slider location
The position of the last train arrived at the current slider location

Whenever the slider is adjusted, all you need to do is:

Update the departure position on the departure array, adding each train passed to the Map and DOM
Update the arrival position on the arrival array, removing any trains passed from the Map and DOM

(If the slider moves backwards, reverse the procedure).

For every currently displayed train, adjust their positions as needed.

This method will greatly increase efficiency. For example, with a typical slider step, roughly 20 trains move, 1 departs, 1 arrives, requiring only 22 updates instead of 11000.

Additional optimization suggestions written during the evolution of the question, which may still be relevant even after implementing the above tip:

The backend code used for grouping appears unoptimized as it filters the data array 120 times. A better approach would involve iterating just once over the dataset, then for each entry, allocating it to the corresponding minutes it belongs to:

   const toMinutes = date => {
     const [hours, mins] = date.split(":");
     return (+hours * 60 + +mins);
   };
   const fromMinutes = minutes => {
     const mins = minutes % 60;
     const hours = (minutes - mins) / 60;
     return hours + ":" + mins;
   };

   const START = "tp_org_r", END = "tp_des_r";
   const groupByDate = { /* "01:20": [...] */ };

   for(const elm of data) {
     const start = toMinutes(elm[START]);
     const end = toMinutes(elm[END]);
     for(let minutes = start; minutes <= end; minutes++) {
        const id = fromMinutes(minutes);
        if(!groupByDate[id]) groupByDate[id] = [];
        groupByDate[id].push(elm);
     }
 }

However, this approach results in significant duplications. Handling it on the backend would hinder performance due to increased data transmission to the frontend. Also, caching seems challenging, leading to repeated calculations for individual clients. Overall, inefficient resource utilization (server-side computation cost) outweighs potential benefits. Frontend execution may seem efficient, but building such lookup tables might not be worthwhile given the simplicity of filtering data.

Instead, send all elements to the frontend and conduct filtering there. Sorted data simplifies filtering steps. Identify start/end positions exceeding the slider time by maximum duration range consideration, eliminating numerous irrelevant entries. Subsequent adjustments post-slider movements require minimal recalculations, targeting specific ranges rather than scanning the entire set of records.

In my initial solution, real-time UI fluidity lacked when users rapidly slid the slider

Addressing user experience challenges involves:

(1) Concealing ongoing elements upon sliding initiation while displaying loading cues

(2) Deliberate delay before generating filtered outcomes post-slider cessation

This dual-trigger approach minimizes page renders, streamlining computations and enhancing UI smoothness, ensuring processing accuracy occurs with minimal user disturbance.

Answer 2

the main objective is to display the movement of trains on a map

This is where we should begin:

Organize the timing by train. This task can be accomplished in O(n) without increasing the dataset size (it can also be done on the backend if necessary, see below for more details).

How does this help? It significantly reduces the size of the dataset that needs filtering. The 11000 events likely pertain to only a few hundred trains.

Next step, create the following initially:

A Map linking train IDs to respective trains currently displayed
A sorted array of trains based on departure time
A sorted array of trains based on arrival time
The position of the last train departed at the current slider location
The position of the last train arrived at the current slider location

Whenever the slider is adjusted, all you need to do is:

Update the departure position on the departure array, adding each train passed to the Map and DOM
Update the arrival position on the arrival array, removing any trains passed from the Map and DOM

(If the slider moves backwards, reverse the procedure).

For every currently displayed train, adjust their positions as needed.

This method will greatly increase efficiency. For example, with a typical slider step, roughly 20 trains move, 1 departs, 1 arrives, requiring only 22 updates instead of 11000.

Additional optimization suggestions written during the evolution of the question, which may still be relevant even after implementing the above tip:

The backend code used for grouping appears unoptimized as it filters the data array 120 times. A better approach would involve iterating just once over the dataset, then for each entry, allocating it to the corresponding minutes it belongs to:

   const toMinutes = date => {
     const [hours, mins] = date.split(":");
     return (+hours * 60 + +mins);
   };
   const fromMinutes = minutes => {
     const mins = minutes % 60;
     const hours = (minutes - mins) / 60;
     return hours + ":" + mins;
   };

   const START = "tp_org_r", END = "tp_des_r";
   const groupByDate = { /* "01:20": [...] */ };

   for(const elm of data) {
     const start = toMinutes(elm[START]);
     const end = toMinutes(elm[END]);
     for(let minutes = start; minutes <= end; minutes++) {
        const id = fromMinutes(minutes);
        if(!groupByDate[id]) groupByDate[id] = [];
        groupByDate[id].push(elm);
     }
 }

However, this approach results in significant duplications. Handling it on the backend would hinder performance due to increased data transmission to the frontend. Also, caching seems challenging, leading to repeated calculations for individual clients. Overall, inefficient resource utilization (server-side computation cost) outweighs potential benefits. Frontend execution may seem efficient, but building such lookup tables might not be worthwhile given the simplicity of filtering data.

Instead, send all elements to the frontend and conduct filtering there. Sorted data simplifies filtering steps. Identify start/end positions exceeding the slider time by maximum duration range consideration, eliminating numerous irrelevant entries. Subsequent adjustments post-slider movements require minimal recalculations, targeting specific ranges rather than scanning the entire set of records.

In my initial solution, real-time UI fluidity lacked when users rapidly slid the slider

Addressing user experience challenges involves:

(1) Concealing ongoing elements upon sliding initiation while displaying loading cues

(2) Deliberate delay before generating filtered outcomes post-slider cessation

This dual-trigger approach minimizes page renders, streamlining computations and enhancing UI smoothness, ensuring processing accuracy occurs with minimal user disturbance.

Answer 3

Answer №2

To optimize performance, it's important to note that the main issue lies in rendering rather than filtering.

The bottleneck arises from utilizing the Date object and comparing dates. To enhance performance, consider converting your data from Date type to Number type (representing milliseconds) for efficient comparison. I've made adjustments to your code, providing a sample for reference.

Additionally, JavaScript loop performances vary significantly. To illustrate this point, I tested the summation of random 10k items using different types of loops—namely for, for-of, while, forEach, and reduce—repeating the tests 10,000 times. Here are the average loop times garnered:

For Loop, average loop time: ~10 microseconds
For-Of, average loop time: ~110 microseconds
ForEach, average loop time: ~77 microseconds
While, average loop time: ~11 microseconds
Reduce, average loop time: ~113 microseconds

You can leverage methods like Object.keys, Object.values, Array.map, and Array.filter effectively when needed.

let formatTrips = function (data) {
    var arr = [], i, j;
    for(i=0; i<24; i++) {
        for(j=0; j<60; j++) {
            arr.push(i*60*60*1000 + j*60*1000 );
        }
    }

    var start = new Date().getTime();
    const dataWithTimeStamp = data.map((d) => (
      {
        ...d,
        tp_org_r: (new Date(d.startingTime)).valueOf() % (24*60*60*1000), // just get time in miliseconds
        tp_des_r: (new Date(d.endingTime)).valueOf() % (24*60*60*1000) // just get time in miliseconds
      }
    ));

    var arrFormated = arr.map((time)=> {
        let schedulesFiltered=[];
        schedulesFiltered = dataWithTimeStamp.filter(elm => (elm.tp_org_r <= time && elm.tp_des_r >= time));
        //console.log(schedulesFiltered)
        return schedulesFiltered

    });
    var end = new Date().getTime();
    var time = end - start;
    console.log('Execution time: ' + time);
}

// Create sample data
const sampleData = [];
const date2019ToMiliseconds = (new Date("2019-01-01")).valueOf();
const miliSecondsFrom2019 = (new Date()).valueOf() - date2019ToMiliseconds;
for(let i = 0;i<10000; i++){
  const randomDate = new Date(Math.floor(date2019ToMiliseconds+Math.random()*miliSecondsFrom2019)).valueOf(); // Random Date between 2019-01-01 till now
   sampleData.push({
      "startingTime": randomDate,
      "endingTime":randomDate + (1000*60)
   })
}
formatTrips(sampleData);

<script src="https://momentjs.com/downloads/moment.min.js"></script>

Answer 4

To optimize performance, it's important to note that the main issue lies in rendering rather than filtering.

The bottleneck arises from utilizing the Date object and comparing dates. To enhance performance, consider converting your data from Date type to Number type (representing milliseconds) for efficient comparison. I've made adjustments to your code, providing a sample for reference.

Additionally, JavaScript loop performances vary significantly. To illustrate this point, I tested the summation of random 10k items using different types of loops—namely for, for-of, while, forEach, and reduce—repeating the tests 10,000 times. Here are the average loop times garnered:

For Loop, average loop time: ~10 microseconds
For-Of, average loop time: ~110 microseconds
ForEach, average loop time: ~77 microseconds
While, average loop time: ~11 microseconds
Reduce, average loop time: ~113 microseconds

You can leverage methods like Object.keys, Object.values, Array.map, and Array.filter effectively when needed.

let formatTrips = function (data) {
    var arr = [], i, j;
    for(i=0; i<24; i++) {
        for(j=0; j<60; j++) {
            arr.push(i*60*60*1000 + j*60*1000 );
        }
    }

    var start = new Date().getTime();
    const dataWithTimeStamp = data.map((d) => (
      {
        ...d,
        tp_org_r: (new Date(d.startingTime)).valueOf() % (24*60*60*1000), // just get time in miliseconds
        tp_des_r: (new Date(d.endingTime)).valueOf() % (24*60*60*1000) // just get time in miliseconds
      }
    ));

    var arrFormated = arr.map((time)=> {
        let schedulesFiltered=[];
        schedulesFiltered = dataWithTimeStamp.filter(elm => (elm.tp_org_r <= time && elm.tp_des_r >= time));
        //console.log(schedulesFiltered)
        return schedulesFiltered

    });
    var end = new Date().getTime();
    var time = end - start;
    console.log('Execution time: ' + time);
}

// Create sample data
const sampleData = [];
const date2019ToMiliseconds = (new Date("2019-01-01")).valueOf();
const miliSecondsFrom2019 = (new Date()).valueOf() - date2019ToMiliseconds;
for(let i = 0;i<10000; i++){
  const randomDate = new Date(Math.floor(date2019ToMiliseconds+Math.random()*miliSecondsFrom2019)).valueOf(); // Random Date between 2019-01-01 till now
   sampleData.push({
      "startingTime": randomDate,
      "endingTime":randomDate + (1000*60)
   })
}
formatTrips(sampleData);

<script src="https://momentjs.com/downloads/moment.min.js"></script>

Answer 5

Answer №3

To enhance efficiency, consider reverting to the initial framework and pre-computing a similar supportive structure on the user's end. For additional optimization, you might consider lazily updating the "cache" structure by predicting the user's behavior (and temporarily halting user interface actions until the cache is populated).

Answer 6

To enhance efficiency, consider reverting to the initial framework and pre-computing a similar supportive structure on the user's end. For additional optimization, you might consider lazily updating the "cache" structure by predicting the user's behavior (and temporarily halting user interface actions until the cache is populated).

Enhance the efficiency of iterating through a vast array of objects

Answer №1

Answer №2

Answer №3

Similar questions

Occasionally, wmuslider fails to load properly

Doing calculations in a GridView using JavaScript

Encountering sporadic EACCES issues while executing selenium tests in the AVA framework

Encountering difficulties hosting create-react-app build on shared server, receiving 404 errors specifically linked to the chunk.js file

Using Javscript to navigate an HTML webpage

Convert XML data to JSON format while implementing custom modifications

Discover Xml information or Json object displayed as XML tree in Html using Javascript

What is the best way to calculate the number of div elements with a specific class that are contained within parent div elements with another specific class?

Declaring module public type definitions for NPM in Typescript: A comprehensive guide

What could be the reason for the exclusion of identical Json content in the output of a REST

Utilize the circliful jQuery plugin within an Angular project

The servlet fails to generate JSON response for AJAX requests

Using @JsonView across the entire Spring MVC REST controller

Updating ngModel value dynamically from controller

The efficiency of Automapper takes a hit when operating in a multi-threaded setting

Tips for parsing through extensive JSON documents containing diverse data types

The HTML element containing a React variable is failing to render ASCII characters

`Generating JSON output containing null elements in Jersey RESTful API using JAXB`

Deleting lines from JSON object using Angular or JavaScript

React - automatically play audio when fully loaded