Receiving an empty string from Chrome FileReader when dealing with large files (300MB or more)

Question

Receiving an empty string from Chrome FileReader when dealing with large files (300MB or more)

Objective:

The task is to read a file from the user's file system as a base64 string in the browser
The size of these files can be up to 1.5GB

Challenge:

A script that works flawlessly on Firefox, regardless of the file size
On Chrome, the script performs well with smaller files (tested with files around 5MB)
However, when selecting a larger file (e.g., 400MB), FileReader completes without errors or exceptions but returns an empty string instead of the expected base64 string

Queries:

Is this a bug specific to Chrome?
Why does it not generate any errors or exceptions despite the issue?
What are the potential solutions or workarounds for this problem?

Note:

It's crucial to highlight that chunking is not feasible as the full base64 string needs to be sent via 'POST' to an API that does not support chunks.

Code:

'use strict';

var filePickerElement = document.getElementById('filepicker');

filePickerElement.onchange = (event) => {
  const selectedFile = event.target.files[0];
  console.log('selectedFile', selectedFile);

  readFile(selectedFile);
};

function readFile(selectedFile) {
  console.log('START READING FILE');
  const reader = new FileReader();

  reader.onload = (e) => {
    const fileBase64 = reader.result.toString();

    console.log('ONLOAD','base64', fileBase64);
    
    if (fileBase64 === '') {
      alert('Result string is EMPTY :(');
    } else {
        alert('It worked as expected :)');
    }
  };

  reader.onprogress = (e) => {
    console.log('Progress', ~~((e.loaded / e.total) * 100 ), '%');
  };

  reader.onerror = (err) => {
    console.error('Error reading the file.', err);
  };

  reader.readAsDataURL(selectedFile);
}

<!doctype html>
<html lang="en">

<head>
  <!-- Required meta tags -->
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">

  <!-- Bootstrap CSS -->
  <link href="https://cdn.jsdelivr.net/npm/<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="6d0f0202191e191f0c1d2d58435d435d">[email protected]</a>/dist/css/bootstrap.min.css" rel="stylesheet"
    integrity="sha384-wEmeIV1mKuiNpC+IOBjI7aAzPcEZeedi5yW5f2yOq55WWLwNGmvvx4Um1vskeMj0" crossorigin="anonymous">

  <title>FileReader Issue Example</title>
</head>

<body>

  <div class="container">
    <h1>FileReader Issue Example</h1>
    <div class="card">
      <div class="card-header">
        Select File:
      </div>
      <div class="card-body">
        <input type="file" id="filepicker" />
      </div>
    </div>

  </div>

  <script src="https://cdn.jsdelivr.net/npm/<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="67050808131413150617275249574957">[email protected]</a>/dist/js/bootstrap.bundle.min.js"
    integrity="sha384-p34f1UUtsS3wqzfto5wAAmdvj+osOnFyQFpp4Ua3gs/ZVWx6oOypYoCJhGGScy+8"
    crossorigin="anonymous"></script>
  <script src="main.js"></script>
</body>

</html>

javascript google-chrome filereader

Answer 1

Answer №1

Is this a chrome bug?

In response to the query posed in Chrome, FileReader API, event.target.result === "", it is important to note that this limitation is not a bug, but rather an intentional constraint within V8 (the JavaScript engine used by Chrome and other platforms). The issue stems from the inability to construct a String exceeding 512MB on 64-bit systems due to V8's heap object limitations, as explained in this commit.

Why is there neither an error nor an exception?

While creating such a large string directly does result in a RangeError, certain operations lack the expected error handling mechanisms. As outlined in FileReader::readOperation Step 3, exceptions should trigger specific actions which are absent in this case.

Detailed steps involving Uint32Array and Blob further illustrate this anomaly, indicating a potential oversight that warrants attention and correction.

I will pursue this matter by raising a relevant issue to address the absence of error handling within the FileReader interface.

How can I fix or work around this issue?

A recommended approach involves modifying your API endpoint to accept binary resources directly instead of relying on data:// URLs, the usage of which is generally discouraged.

An alternative solution for future implementation would entail sending a ReadableStream to your endpoint and performing the data:// URL conversion autonomously using a stream sourced from the Blob.

(Code snippet omitted)

For immediate resolution, consider storing chunks of base64 representations in a Blob, although caution is advised due to potential server-side constraints related to handling excessively large strings. Communication with the API maintainer is strongly suggested to mitigate any issues arising from V8's inherent limitations.

Answer 2

Is this a chrome bug?

In response to the query posed in Chrome, FileReader API, event.target.result === "", it is important to note that this limitation is not a bug, but rather an intentional constraint within V8 (the JavaScript engine used by Chrome and other platforms). The issue stems from the inability to construct a String exceeding 512MB on 64-bit systems due to V8's heap object limitations, as explained in this commit.

Why is there neither an error nor an exception?

While creating such a large string directly does result in a RangeError, certain operations lack the expected error handling mechanisms. As outlined in FileReader::readOperation Step 3, exceptions should trigger specific actions which are absent in this case.

Detailed steps involving Uint32Array and Blob further illustrate this anomaly, indicating a potential oversight that warrants attention and correction.

I will pursue this matter by raising a relevant issue to address the absence of error handling within the FileReader interface.

How can I fix or work around this issue?

A recommended approach involves modifying your API endpoint to accept binary resources directly instead of relying on data:// URLs, the usage of which is generally discouraged.

An alternative solution for future implementation would entail sending a ReadableStream to your endpoint and performing the data:// URL conversion autonomously using a stream sourced from the Blob.

(Code snippet omitted)

For immediate resolution, consider storing chunks of base64 representations in a Blob, although caution is advised due to potential server-side constraints related to handling excessively large strings. Communication with the API maintainer is strongly suggested to mitigate any issues arising from V8's inherent limitations.

Answer 3

Answer №2

Here is a clever method for converting a blob into chunks and then transforming them into base64 blobs. These base64 chunks are concatenated within a JSON blob along with some pre/suffix JSON parts.

By keeping it as a blob, the browser can efficiently manage memory allocation and even offload it to disk if necessary.

If you adjust the chunkSize to be larger, the browser prefers to store smaller blob chunks in memory (in one bucket).

// Obtaining a sample gradient file (blob)
var canvas=document.createElement("canvas"), context=canvas.getContext("2d"), gradient=context.createLinearGradient(0,0,3000,3000);canvas.width=canvas.height=3000;gradient.addColorStop(0,"red");gradient.addColorStop(1,"blue");context.fillStyle=gradient;context.fillRect(0,0,canvas.width,canvas.height);canvas.toBlob(main);

async function main (blob) {
  var fileReader = new FileReader()
  // It's recommended to add 2 so it omits == from all but the last chunk
  var chunkSize = (1 << 16) + 2 
  var position = 0
  var b64Chunks = []
  
  while (position < blob.size) {
    await new Promise(resolve => {
      fileReader.readAsDataURL(blob.slice(position, position + chunkSize))
      fileReader.onload = () => {
        const base64 = fileReader.result.split(',')[1]
        b64Chunks.push(new Blob([base64]))
        resolve()
      }
      position += chunkSize
    })
  }

  // How you combine all chunks into json is now up to you.
  // This solution just outlines what needs to be done
  // There are more automated ways, but here is a simple form
  // (just a heads-up: this new blob won't create a lot of data in memory, it will only reference other blob locations)
  const jsonData = new Blob([
    '{"data": "', ...b64Chunks, '"}'
  ], { type: 'application/json' })

  /*
  // It's strongly recommended to ask API developers 
  // to implement support for binary/file uploads (multipart-formdata)
  // Base64 is roughly ~33% bigger and handling streaming 
  // this data on the server to disk is nearly impossible 
  fetch('./upload-files-to-bad-json-only-api', {
    method: 'POST',
    body: jsonData
  })
  */
  
  // Just testing that it still functions
  //
  // new Response(jsonData).json().then(console.log)
  fetch('data:image/png;base64,' + await new Blob(b64Chunks).text()).then(response => response.blob()).then(blob => console.log(URL.createObjectURL(blob)))
}

I opted not to use

base64 += fileReader.result.split(',')[1]

and JSON.stringify since dealing with GiB of data is substantial and JSON isn't ideal for handling binary data.

Answer 4

Here is a clever method for converting a blob into chunks and then transforming them into base64 blobs. These base64 chunks are concatenated within a JSON blob along with some pre/suffix JSON parts.

By keeping it as a blob, the browser can efficiently manage memory allocation and even offload it to disk if necessary.

If you adjust the chunkSize to be larger, the browser prefers to store smaller blob chunks in memory (in one bucket).

// Obtaining a sample gradient file (blob)
var canvas=document.createElement("canvas"), context=canvas.getContext("2d"), gradient=context.createLinearGradient(0,0,3000,3000);canvas.width=canvas.height=3000;gradient.addColorStop(0,"red");gradient.addColorStop(1,"blue");context.fillStyle=gradient;context.fillRect(0,0,canvas.width,canvas.height);canvas.toBlob(main);

async function main (blob) {
  var fileReader = new FileReader()
  // It's recommended to add 2 so it omits == from all but the last chunk
  var chunkSize = (1 << 16) + 2 
  var position = 0
  var b64Chunks = []
  
  while (position < blob.size) {
    await new Promise(resolve => {
      fileReader.readAsDataURL(blob.slice(position, position + chunkSize))
      fileReader.onload = () => {
        const base64 = fileReader.result.split(',')[1]
        b64Chunks.push(new Blob([base64]))
        resolve()
      }
      position += chunkSize
    })
  }

  // How you combine all chunks into json is now up to you.
  // This solution just outlines what needs to be done
  // There are more automated ways, but here is a simple form
  // (just a heads-up: this new blob won't create a lot of data in memory, it will only reference other blob locations)
  const jsonData = new Blob([
    '{"data": "', ...b64Chunks, '"}'
  ], { type: 'application/json' })

  /*
  // It's strongly recommended to ask API developers 
  // to implement support for binary/file uploads (multipart-formdata)
  // Base64 is roughly ~33% bigger and handling streaming 
  // this data on the server to disk is nearly impossible 
  fetch('./upload-files-to-bad-json-only-api', {
    method: 'POST',
    body: jsonData
  })
  */
  
  // Just testing that it still functions
  //
  // new Response(jsonData).json().then(console.log)
  fetch('data:image/png;base64,' + await new Blob(b64Chunks).text()).then(response => response.blob()).then(blob => console.log(URL.createObjectURL(blob)))
}

I opted not to use

base64 += fileReader.result.split(',')[1]

and JSON.stringify since dealing with GiB of data is substantial and JSON isn't ideal for handling binary data.

Receiving an empty string from Chrome FileReader when dealing with large files (300MB or more)

Answer №1

Answer №2

Similar questions

What steps do I need to take in order to create functions that are

What is the best way to save a string for future use in Angular after receiving it from a POST request API?

How can you make a button in Vue only visible after all API calls have successfully retrieved the data?

Is it illegal to escape quotes when using an image source attribute and onerror event in HTML: `<img src="x" onerror="alert("hello")" />`?

Having trouble retrieving the table value from an HTML document?

Having trouble getting Three.js JSON models to cast shadows properly?

Javascript issue: SyntaxError - A numerical value is required after the decimal point

Implementing interactive dropdown menus to trigger specific actions

Endless loop caused by Angular UI-router's promise resolving

Display additional inputs using the PHP Foreach Loop depending on the selection made

Tips for transferring an array variable into a div element using AJAX

Dealing with Request Disconnection in a Node.js and Express Application

D3.js Issue: The <g> element's transform attribute is expecting a number, but instead received "translate(NaN,NaN)"

Crafting artistic shapes using the Canny Edge Detection technique on Canvas

The ajax function nestled within a nested forEach loop completes its execution once the inner callback function is triggered

Keep a close eye on your Vue app to make sure it's

Activate fancybox when clicked, triggering numerous ajax requests

What are the advantages of choosing express.js over Ruby on Sinatra?

Deleting a row from a table in AngularJS can be accomplished by following these steps

The HTTP request is being executed twice for some reason unknown to me