What is the best way to store objects containing extensive binary data along with additional values?

I'm currently working on saving a JavaScript object that includes binary data along with other values. I want the output to resemble the following:

{
  "value":"xyz",
  "file1":"[FileContent]",
  "file2":"[LargeFileContent]"
}

Previously, I only dealt with JSON data without any binary content. However, as I started incorporating binary data, I encountered issues when handling large files (>1GB).

I attempted the following method: JSON.stringify or how to serialize binary data as base64 encoded JSON? This solution worked for smaller files (around 20MB). But when dealing with larger files, the result from the FileReader always turns out to be an empty string. The resulting output would appear like this:

{
   "value":"xyz:,
   "file1":"[FileContent]",
   "file2":""
}

The code responsible for reading the blobs closely resembles the one mentioned in another post:

const readFiles = async (measurements: FormData) => {
    setFiles([]); //This is where the result is stored
    let promises: Array<Promise<string>> = [];
    measurements.forEach((value) => {
      let dataBlob = value as Blob;
      console.log(dataBlob); //No issue here
      promises.push(
        new Promise((resolve, reject) => {
          const reader = new FileReader();
          reader.readAsDataURL(dataBlob);
          reader.onloadend = function () {
            resolve(reader.result as string);
          };
          reader.onerror = function (error) {
            reject(error);
          };
        })
      );
    });
    let result = await Promise.all(promises);
    console.log(result); //Empty for large file
    setFiles(result);
  };

Are there any other methods I can attempt?

Answer №1

When sharing data with other computers, it becomes necessary to create a custom binary format for transmission.

In the case of storing Blob objects alongside JSON strings, a simple schema can be devised. This schema involves storing metadata about the Blobs first, followed by the JSON string where each Blob is replaced with a UUID.

This approach circumvents the restriction on string length and allows for partial reading of the binary file using the .slice() method without converting binary data to strings excessively, thereby preventing unnecessary growth in JSON size.

Below is an example implementation showcasing this concept:

/*
 * Custom implementation for storing JSON data and Blob objects in a binary file.
 * Schema:
 *   4 bytes at the beginning = Number of blobs stored in the file
 *   Next 4 * # of blobs bytes = Size of each Blob
 *   Remaining bytes = JSON string
 *
 */
const unique_id_placeholder = "_blob_"; // <-- Modify as needed
function generateBinary(JSObject) {
  let blobIndex = 0;
  const blobsMap = new Map();
  const JSONString = JSON.stringify(JSObject, (key, value) => {
    if (value instanceof Blob) {
      if (blobsMap.has(value)) {
        return blobsMap.get(value);
      }
      blobsMap.set(value, unique_id_placeholder + (blobIndex++));
      return unique_id_placeholder + blobIndex;
    }
    return value;
  });
  const blobsArr = [...blobsMap.keys()];
  const data = [
    new Uint32Array([blobsArr.length]),
    ...blobsArr.map((blob) => new Uint32Array([blob.size])),
    ...blobsArr,
    JSONString
  ];
  return new Blob(data);
}

async function readBinary(bin) {
  const numberOfBlobs = new Uint32Array(await bin.slice(0, 4).arrayBuffer())[0];
  let cursor = 4 * (numberOfBlobs + 1);
  const blobSizes = new Uint32Array(await bin.slice(4, cursor).arrayBuffer())
  const blobs = [];
  for (let i = 0; i < numberOfBlobs; i++) {
    const blobSize = blobSizes[i];
    blobs.push(bin.slice(cursor, cursor += blobSize));
  }
  const pattern = new RegExp(`^${unique_id_placeholder}\\d+$`);
  const JSObject = JSON.parse(
    await bin.slice(cursor).text(),
    (key, value) => {
      if (typeof value !== "string" || !pattern.test(value)) {
        return value;
      }
      const index = +value.replace(unique_id_placeholder, "") - 1;
      return blobs[index];
    }
  );
  return JSObject;
}
// Demo usage
(async () => {
  const obj = {
    foo: "bar",
    file1: new Blob(["Let's pretend I'm actually binary data"]),
    // This one is 512MiB, exceeding Chrome's max string size
    // It cannot be stored as a JSON string in Chrome
    file2: new Blob([Uint8Array.from({ length: 512*1024*1024 }, () => 255)]),
  };
  const bin = generateBinary(obj);
  console.log("Stored as binary", bin);
  const deserializedObj = await readBinary(bin);
  console.log({deserializedObj});
  console.log("file1 contents:", await deserializedObj.file1.text());
})().catch(console.error);

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Capturing and saving location data without internet connection, and then displaying markers once connectivity is restored

I am currently developing a web application that will monitor the user's location. I want to ensure its reliability, so even if the user loses internet connection, the application will continue tracking their location (since GPS does not rely on inter ...

Modal Pop-ups Failing to Open on Subsequent Attempts

My table consists of buttons on the right-hand side for a menu. The first button in the menu is supposed to open a modal box upon clicking. However, the first buttons in the subsequent rows do not function as expected and fail to open the modal. Refer to t ...

Clearing all data in a form upon opening

Within a single portlet, I have organized two forms under separate tabs. What I am aiming for is that whenever I switch to tab 2, all fields within its associated form should automatically be cleared without the need for a button click. Even after attempti ...

Passing variables from AJAX response to PHP using a passthru function

In my PHP file, I have implemented a functionality where certain checkboxes are checked and upon clicking a button, a JavaScript file is triggered. The JavaScript code snippet used is as follows: $.ajax ({ type: "POST", url: "decisionExec.php", ...

Concealing the flexslider in Angular when accessing the main URL

I need to hide a div with flexslider in it on the root page using ng-hide. The issue is that the images do not load when navigating to another path. Here is how my index.html is structured: <ul> <li><a href="#/">Root</a> ...

What is the best way to combine HTML and JavaScript code within a single JavaScript file?

Is there a way to include a responsive iframe without any scroll in multiple websites by using just one line of code? I found this snippet that seems promising: <script src="testfile.js"></script> The testfile.js contains the necessary HTML a ...

When attempting to render a base64 string in an <img> tag using ReactJS, an error message ERR_INVALID_URL is displayed

I am currently working on displaying server-generated images (specifically matplotlib graphs) in a ReactJS module without needing to save the files on the server. To achieve this, I decided to use base64 to create an image string. When it comes time to sh ...

Guidelines for transferring data when a button is held down or pressed

I am looking to continuously send values while a button is pressed. Currently, a value is only sent with each click. Below is the current code: my_custom_script.js $(document).ready(function() { $('#left').mousedown(function() { var left ...

Output text in Javascript (not to the console)

I'm on the lookout for a way to welcome a user by their name when they enter it into a JavaScript application. I have tried this code snippet: function getname() { var number = document.getElementById("fname").value; alert("Hello, " + fnam ...

What should be triggered when clicking on the cancel button in Bootstrap's modal: `close()` or `dismiss()`?

Bootstrap's modal offers two methods for hiding the dialog: close(result) (Type: function) - Used to close a modal by providing a result. dismiss(reason) (Type: function) - Used to dismiss a modal and provide a reason. Should I use close when the u ...

Allowing unauthorized cross-origin requests to reach the SailsJS controller despite implementing CORS restrictions

My cors.js file has the following configuration: module.exports.cors = { allRoutes: true, origin: require('./local.js').hosts, // which is 'http://localhost' } I decided to test it by making a request from a random site, Here is ...

Discovering active path while utilizing dynamic routes - here's how!

Whenever I click on the project element in my HeadlessCMS with the URL /projects/slug-name, the projects button in my Navbar (LinkItem) does not highlight with a background color (bgColor). How can I modify this line of code: const active = path === href / ...

Unable to successfully change the Span Class

My webpage has the code snippet below, but it's not functioning as expected. I have tried two methods to change the span's class attribute, but they aren't working. Could someone please help me identify where the issue lies? :) <script l ...

Issue with DragControls in Three.js not detecting intersections

My current challenge involves setting up DragControls in my Three.js project. After researching online, I found that the process should be quite simple. The code I have written is as follows: let dragObjects = [obj] const dragControls = new DragControls(dr ...

How can we split the state between unique React forms that are interconnected?

My issue is that, based on the value of a prop in my form (specifically step), I need to display different forms. Everything works fine; the form loads the text programmatically as needed and fills in the form accordingly. However, when the value of step c ...

Error: Incompatibility - The variable is not functioning properly in node.js or express.js

Upon encountering the error message "TypeError: undefined is not a function," I called a callback with parameters null, array_reply, and threadResults. It appears that this section of code is problematic, but I am uncertain why. Your assistance in resol ...

Unable to connect with controller after deploying asp.net mvc ajax call on server

For the first time, I am encountering a new issue. Although my code runs smoothly on my localhost, when I upload it to the server, I get a bad request error. The specific error message is: 400 (Bad Request) Below is the snippet of my code: Controll ...

A guide on transferring JSON data from a Django view to Vue.js instance methods

Looking for assistance with Vue.js - I'm new to it. How can JSON data be passed from a Django view to a Vue instance (method)? Views.py def articles(request): model = News.objects.all() # getting News objects list random_generator = rando ...

What specific flag should be included in Chrome settings to prevent displaying c:/fakepath?

Is there a way to disable this security feature? I'm seeking a solution in Chrome to obtain the full file path of an uploaded or viewed file, similar to how it can be done in Internet Explorer. Despite my efforts with flags like --allow-file-access-f ...

A guide to connecting keyboard events to div elements within a React application

Currently working on a basic react project to gain some practical experience with the library. I aim to develop a simple user interface with blank spaces to be filled in by typing via keyboard input. Here's a glimpse of my progress so far: function ...