Is it a bad idea to filter a large JSON object of about 6MB in AngularJS?

Question

Is it a bad idea to filter a large JSON object of about 6MB in AngularJS?

I am currently working on developing a barcode scanner application. The client has provided me with a 6mb CSV file containing data for various products.

One approach I am considering is parsing the CSV file into a JSON object, extracting barcodes, and then searching for the corresponding items within a 6mb in-memory JavaScript object. Would this method negatively impact performance? Is there a size limit for the file that would affect performance?

If I were to convert this project into an Ionic app, would it further decrease performance?

While I understand that creating an API server for querying would be more efficient, do you think it is absolutely necessary in this case?

EDIT: The file is downloaded once by an Angular service and stored in memory until the app is closed. Filtering would be done on this in-memory object.

javascript angularjs json csv ionic-framework

Answer 1

Answer №1

Note: My assumption is that this code will be utilized in a single-page application scenario or any situation where it is executed repeatedly, rather than just once per page load.

Considering the average size of images nowadays, the browser should have no issue handling a 6mb file.

The real concern lies in the efficiency of looping through such a large dataset compared to optimizing it for quicker lookups. Various factors can impact performance at this stage.

JavaScript objects tend to slow down when they contain numerous keys. In my experience, performance started deteriorating around 11,000 keys or so...

Hence, I suggest breaking down the barcodes into separate categories – here's some code that groups all the 1s together, as well as the 2s and so on. This approach significantly boosts performance while handling extensive datasets.

var barcodes = {
   0: {},
   1: {},
   2: {},
   3: {},
   4: {},
   5: {},
   6: {},
   7: {},
   8: {},
   9: {}
};

function addBarcode(barcode, obj) {
   var firstNumber = barcode.charAt(0);
   barcodes[firstNumber][barcode] = obj;
}

function getBarcode(barcode) {
   var firstNumber = barcode.charAt(0);
   return barcodes[firstNumber][barcode];
}


addBarcode("1104", {
    testprop: true
});

var o = getBarcode("1104");
console.log("testProp should be true. It is ", o.testprop);

Answer 2

Note: My assumption is that this code will be utilized in a single-page application scenario or any situation where it is executed repeatedly, rather than just once per page load.

Considering the average size of images nowadays, the browser should have no issue handling a 6mb file.

The real concern lies in the efficiency of looping through such a large dataset compared to optimizing it for quicker lookups. Various factors can impact performance at this stage.

JavaScript objects tend to slow down when they contain numerous keys. In my experience, performance started deteriorating around 11,000 keys or so...

Hence, I suggest breaking down the barcodes into separate categories – here's some code that groups all the 1s together, as well as the 2s and so on. This approach significantly boosts performance while handling extensive datasets.

var barcodes = {
   0: {},
   1: {},
   2: {},
   3: {},
   4: {},
   5: {},
   6: {},
   7: {},
   8: {},
   9: {}
};

function addBarcode(barcode, obj) {
   var firstNumber = barcode.charAt(0);
   barcodes[firstNumber][barcode] = obj;
}

function getBarcode(barcode) {
   var firstNumber = barcode.charAt(0);
   return barcodes[firstNumber][barcode];
}


addBarcode("1104", {
    testprop: true
});

var o = getBarcode("1104");
console.log("testProp should be true. It is ", o.testprop);

Answer 3

Answer №2

It is advisable to organize your data before importing it into your application. Avoid directly reading the raw CSV file; instead, create a pre-sorted .json file and possibly another .json file for pre-indexed data storage.

Consider testing the performance of a simple loop as an initial evaluation step. Sometimes, a basic loop may suffice for your requirements.

If you decide to pre-index the data, you can structure an object like the example below. This object captures every 10 thousandth UPC value for efficient retrieval:

var preIndex = [
    //{"limit" : index at which value exists, value: UPC value},
    {"limit": 0, "value": 00000001234},
    {"limit": 10000, "value": 00009871234},
    {"limit": 20000, "value": 00019938234},
    ....
]

To locate your specific UPC within this preIndexed object, your code would need to compare and select the relevant limits:

 // Identify where YourUPC falls between two values in preIndex    
// preIndex[x].value < YourUPC <= preIndex[x+1].value

// Loop through a maximum of 10 thousand items to find your target item
for(var i = preIndex[x].limit; i <= preIndex[x+1].limit; i++){
     ... // if(myData[i].UPC == YourUPC)
 }

Answer 4

It is advisable to organize your data before importing it into your application. Avoid directly reading the raw CSV file; instead, create a pre-sorted .json file and possibly another .json file for pre-indexed data storage.

Consider testing the performance of a simple loop as an initial evaluation step. Sometimes, a basic loop may suffice for your requirements.

If you decide to pre-index the data, you can structure an object like the example below. This object captures every 10 thousandth UPC value for efficient retrieval:

var preIndex = [
    //{"limit" : index at which value exists, value: UPC value},
    {"limit": 0, "value": 00000001234},
    {"limit": 10000, "value": 00009871234},
    {"limit": 20000, "value": 00019938234},
    ....
]

To locate your specific UPC within this preIndexed object, your code would need to compare and select the relevant limits:

 // Identify where YourUPC falls between two values in preIndex    
// preIndex[x].value < YourUPC <= preIndex[x+1].value

// Loop through a maximum of 10 thousand items to find your target item
for(var i = preIndex[x].limit; i <= preIndex[x+1].limit; i++){
     ... // if(myData[i].UPC == YourUPC)
 }

Is it a bad idea to filter a large JSON object of about 6MB in AngularJS?

Answer №1

Answer №2

Similar questions

Error occurs when an arrow function is called before its function definition

Photos failing to load in the image slider

Tips for removing a row from a table

sending an array from one CodeIgniter view to another view via Ajax

The icon is being displayed as text instead of the actual Fontawesome icon

Error: Unable to locate module: Issue discovering 'crypto' and 'fs' modules

Checking the integrity of JSON information

What is the process for transforming <a>fai</a><b>3</b> to <fai>3</fai> with xpath within WSO2 ESB?

Exploring the effectiveness of React Hook Form using React Testing Library

Can you guide me on implementing first-person controls in react-three-fiber?

Nested Tables in JavaScript: Creating Tables within Tables

Error: Unable to access attributes of null object (specifically 'disable click')

Eliminate error class in jQuery Validate once validation is successful

Trouble with a Userscript focused on Styling

Add the item to an array to store its state

Switch between viewing outcomes retrieved from a database

How to Send C# Array as a Parameter to a JQuery Function

experiencing an excessive amount of re-renders after transferring data to a distinct component

Displaying and hiding collapsible items in Bootstrap 4: A guide to showing one item at a time

Guide to retrieving JSON data from SolarWinds Orion REST API using Java