Transforming a massive JSON object into a Blob concerns directly converting it into an ArrayBuffer or Blob to prevent exceeding the maximum string length error

Question

Transforming a massive JSON object into a Blob concerns directly converting it into an ArrayBuffer or Blob to prevent exceeding the maximum string length error

Situation:

Within my application, I am encountering the following code:

let blob = new Blob([JSON.stringify(json)], {type: "application/json"});

This code sometimes fails because the maximum string length allowed in Chrome is approximately 500MB, and the size of the json object can exceed this limit.

Inquiry:

I am seeking a method to directly convert my json variable (a POJO) into a Blob, potentially through a streaming process that converts it to an ArrayBuffer incrementally. Alternatively, I am open to any other approach that enables the conversion of a large json object into a Blob without encountering the 'maximum string length' issue.

Considerations:

The proposed solution must be functional in web browsers.
If recommending an existing library, it should not require the json object to be just an array, as handling such cases is relatively straightforward. The library should instead support arbitrarily nested JSON objects where a significant portion of data may reside deep within the structure rather than being evenly distributed across top-level keys.
I am not interested in solutions that necessitate a stream as input and yield a stream of string segments as output, like json-stream-stringify or streaming-json-stringify. My preference is for transforming an already-loaded POJO into a Blob containing the stringified JSON content.

Additional Information:

How to use JSONStream to stringify a large object - This reference shares similarities with my scenario but focuses on JSONStream, designed for Node.js rather than browser environments. Additionally, the provided solution appears to save data key-by-key rather than in a deeply nested manner. If there exists a way to achieve this functionality in a web browser, resulting in an ArrayBuffer storing the oversized JSON string for complex nested objects, that would be considered a suitable answer.
How to use streams to JSON stringify large nested objects in Node.js? - Similar to the previous link.

javascript json arraybuffer

Answer 1

Answer №1

One way to bypass this restriction is by creating the Blob in chunks using strings.

const header = 24;
const bytes = new Uint8Array((512 * 1024 * 1024) - header);
const bigStr = new TextDecoder().decode(bytes);
const arr = [];
for (let i=0; i<5; i++) {
  arr.push(bigStr);
}
console.log(new Blob(arr).size); // 2.7GB

The Blob constructor can also take other Blobs as input in its blobParts parameter, allowing us to use a simple recursive stringifier that creates a list of Blob objects instead of joining values with a DOMString separator.

This results in something like

new Blob(["{", <Blob>, ":", <Blob>, "}"]);

By following this approach, we avoid the 500MiB limit.

I made a quick modification to this implementation, but it hasn't undergone thorough testing, so you may want to verify it yourself:

// Modified JSON stringify method
// Your code here...

Answer 2

One way to bypass this restriction is by creating the Blob in chunks using strings.

const header = 24;
const bytes = new Uint8Array((512 * 1024 * 1024) - header);
const bigStr = new TextDecoder().decode(bytes);
const arr = [];
for (let i=0; i<5; i++) {
  arr.push(bigStr);
}
console.log(new Blob(arr).size); // 2.7GB

The Blob constructor can also take other Blobs as input in its blobParts parameter, allowing us to use a simple recursive stringifier that creates a list of Blob objects instead of joining values with a DOMString separator.

This results in something like

new Blob(["{", <Blob>, ":", <Blob>, "}"]);

By following this approach, we avoid the 500MiB limit.

I made a quick modification to this implementation, but it hasn't undergone thorough testing, so you may want to verify it yourself:

// Modified JSON stringify method
// Your code here...

Answer 3

Answer №2

After some collaboration with ChatGPT-4, I have come up with a potential solution that appears promising based on initial testing.

function jsonToBlob(json) {
  const textEncoder = new TextEncoder();
  const seen = new WeakSet();

  function processValue(value) {
    if(seen.has(value)) {
      throw new TypeError("Converting circular structure to JSON");
    }

    if(value && typeof value.toJSON === "function") {
      value = value.toJSON();
    }

    if(typeof value === 'object' && value !== null) {
      seen.add(value);

      const blobParts = [];
      const entries = Array.isArray(value) ? value : Object.entries(value);
      for(let i = 0; i < entries.length; i++) {
        if(Array.isArray(value)) {
          blobParts.push(processValue(entries[i]));
        } else {
          const [key, val] = entries[i];
          blobParts.push(textEncoder.encode(JSON.stringify(key) + ':'), processValue(val));
        }
        if(i !== entries.length - 1) blobParts.push(textEncoder.encode(','));
      }

      const startBracket = Array.isArray(value) ? '[' : '{';
      const endBracket = Array.isArray(value) ? ']' : '}';
      return new Blob([textEncoder.encode(startBracket), ...blobParts, textEncoder.encode(endBracket)]);
    } else if(typeof value === 'function' || typeof value === 'undefined') {
      return textEncoder.encode("null");
    } else {
      // For primitives we just convert it to string and encode
      return textEncoder.encode(JSON.stringify(value));
    }
  }

  return processValue(json);
}

✅ Test 1:

let blob = jsonToBlob([{hello:{foo:[1,2,3], a:1, bar:["a", 2, {$hi:[1,2,3, {a:3}]}]}}, 4, new Date(),, (()=>{})]);
console.log(JSON.parse(await blob.text()));

✅ Test 2:

let json = {};
for(let i = 0; i < 600000; i++) {
  json[Math.random()] = Math.random().toString().repeat(100);
}
let blob = jsonToBlob(json);
console.log(blob); // ~1 GB

I will provide updates if any issues arise during the production use of this solution.

Answer 4

After some collaboration with ChatGPT-4, I have come up with a potential solution that appears promising based on initial testing.

function jsonToBlob(json) {
  const textEncoder = new TextEncoder();
  const seen = new WeakSet();

  function processValue(value) {
    if(seen.has(value)) {
      throw new TypeError("Converting circular structure to JSON");
    }

    if(value && typeof value.toJSON === "function") {
      value = value.toJSON();
    }

    if(typeof value === 'object' && value !== null) {
      seen.add(value);

      const blobParts = [];
      const entries = Array.isArray(value) ? value : Object.entries(value);
      for(let i = 0; i < entries.length; i++) {
        if(Array.isArray(value)) {
          blobParts.push(processValue(entries[i]));
        } else {
          const [key, val] = entries[i];
          blobParts.push(textEncoder.encode(JSON.stringify(key) + ':'), processValue(val));
        }
        if(i !== entries.length - 1) blobParts.push(textEncoder.encode(','));
      }

      const startBracket = Array.isArray(value) ? '[' : '{';
      const endBracket = Array.isArray(value) ? ']' : '}';
      return new Blob([textEncoder.encode(startBracket), ...blobParts, textEncoder.encode(endBracket)]);
    } else if(typeof value === 'function' || typeof value === 'undefined') {
      return textEncoder.encode("null");
    } else {
      // For primitives we just convert it to string and encode
      return textEncoder.encode(JSON.stringify(value));
    }
  }

  return processValue(json);
}

✅ Test 1:

let blob = jsonToBlob([{hello:{foo:[1,2,3], a:1, bar:["a", 2, {$hi:[1,2,3, {a:3}]}]}}, 4, new Date(),, (()=>{})]);
console.log(JSON.parse(await blob.text()));

✅ Test 2:

let json = {};
for(let i = 0; i < 600000; i++) {
  json[Math.random()] = Math.random().toString().repeat(100);
}
let blob = jsonToBlob(json);
console.log(blob); // ~1 GB

I will provide updates if any issues arise during the production use of this solution.

Transforming a massive JSON object into a Blob concerns directly converting it into an ArrayBuffer or Blob to prevent exceeding the maximum string length error

Answer №1

Answer №2

Similar questions

Storing sound recordings with GridFS

Guide on utilizing map function in JavaScript and Vue to generate a fresh array

What is the procedure for transforming a JSON response into an array?

Retrieve the present time of an ongoing CSS3 animation

What's the best way to incorporate mouseenter() and mouseleave() into several elements sharing the same class?

The Angular route functions flawlessly in the development environment, but encounters issues when deployed to

Integrating a Json service into my ASP.Net Web Api results in an altered response format when viewing in Postman

Exploring TypeScript integration with Google Adsense featuring a personalized user interface

Please display the Bootstrap Modal first before continuing

Attempting to eliminate any dates that have already occurred

How to generate nested arrays in JSON format using MySQL data

Struggling to find a solution for changing the color of a select box when an option is chosen

Understanding how to use the 'this' variable in Vue components is essential for efficiently modifying DOM elements. Let's dive into a clarification on the usage of 'this'

Checking and contrasting dates within Javascript

Design a dynamic dropdown feature that is triggered when the chip element is clicked within the TextField component

Should I use "npm install" or "sudo npm install -g"

Leveraging the power of jQuery/javascript in conjunction with Google Forms

Include a "remember me" feature in the Stripe form

What is the most efficient way to clear the input field in Angularjs when the backspace or delete keys are pressed?

The switch statement remains unchanged for varying variables