Receiving an empty string from Chrome FileReader when dealing with large files (300MB or more)

Question

Receiving an empty string from Chrome FileReader when dealing with large files (300MB or more)

Objective:

The task is to read a file from the user's file system as a base64 string in the browser
The size of these files can be up to 1.5GB

Challenge:

A script that works flawlessly on Firefox, regardless of the file size
On Chrome, the script performs well with smaller files (tested with files around 5MB)
However, when selecting a larger file (e.g., 400MB), FileReader completes without errors or exceptions but returns an empty string instead of the expected base64 string

Queries:

Is this a bug specific to Chrome?
Why does it not generate any errors or exceptions despite the issue?
What are the potential solutions or workarounds for this problem?

Note:

It's crucial to highlight that chunking is not feasible as the full base64 string needs to be sent via 'POST' to an API that does not support chunks.

Code:

'use strict';

var filePickerElement = document.getElementById('filepicker');

filePickerElement.onchange = (event) => {
  const selectedFile = event.target.files[0];
  console.log('selectedFile', selectedFile);

  readFile(selectedFile);
};

function readFile(selectedFile) {
  console.log('START READING FILE');
  const reader = new FileReader();

  reader.onload = (e) => {
    const fileBase64 = reader.result.toString();

    console.log('ONLOAD','base64', fileBase64);
    
    if (fileBase64 === '') {
      alert('Result string is EMPTY :(');
    } else {
        alert('It worked as expected :)');
    }
  };

  reader.onprogress = (e) => {
    console.log('Progress', ~~((e.loaded / e.total) * 100 ), '%');
  };

  reader.onerror = (err) => {
    console.error('Error reading the file.', err);
  };

  reader.readAsDataURL(selectedFile);
}

<!doctype html>
<html lang="en">

<head>
  <!-- Required meta tags -->
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">

  <!-- Bootstrap CSS -->
  <link href="https://cdn.jsdelivr.net/npm/<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="6d0f0202191e191f0c1d2d58435d435d">[email protected]</a>/dist/css/bootstrap.min.css" rel="stylesheet"
    integrity="sha384-wEmeIV1mKuiNpC+IOBjI7aAzPcEZeedi5yW5f2yOq55WWLwNGmvvx4Um1vskeMj0" crossorigin="anonymous">

  <title>FileReader Issue Example</title>
</head>

<body>

  <div class="container">
    <h1>FileReader Issue Example</h1>
    <div class="card">
      <div class="card-header">
        Select File:
      </div>
      <div class="card-body">
        <input type="file" id="filepicker" />
      </div>
    </div>

  </div>

  <script src="https://cdn.jsdelivr.net/npm/<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="67050808131413150617275249574957">[email protected]</a>/dist/js/bootstrap.bundle.min.js"
    integrity="sha384-p34f1UUtsS3wqzfto5wAAmdvj+osOnFyQFpp4Ua3gs/ZVWx6oOypYoCJhGGScy+8"
    crossorigin="anonymous"></script>
  <script src="main.js"></script>
</body>

</html>

javascript google-chrome filereader

Answer 1

Answer №1

Is this a chrome bug?

In response to the query posed in Chrome, FileReader API, event.target.result === "", it is important to note that this limitation is not a bug, but rather an intentional constraint within V8 (the JavaScript engine used by Chrome and other platforms). The issue stems from the inability to construct a String exceeding 512MB on 64-bit systems due to V8's heap object limitations, as explained in this commit.

Why is there neither an error nor an exception?

While creating such a large string directly does result in a RangeError, certain operations lack the expected error handling mechanisms. As outlined in FileReader::readOperation Step 3, exceptions should trigger specific actions which are absent in this case.

Detailed steps involving Uint32Array and Blob further illustrate this anomaly, indicating a potential oversight that warrants attention and correction.

I will pursue this matter by raising a relevant issue to address the absence of error handling within the FileReader interface.

How can I fix or work around this issue?

A recommended approach involves modifying your API endpoint to accept binary resources directly instead of relying on data:// URLs, the usage of which is generally discouraged.

An alternative solution for future implementation would entail sending a ReadableStream to your endpoint and performing the data:// URL conversion autonomously using a stream sourced from the Blob.

(Code snippet omitted)

For immediate resolution, consider storing chunks of base64 representations in a Blob, although caution is advised due to potential server-side constraints related to handling excessively large strings. Communication with the API maintainer is strongly suggested to mitigate any issues arising from V8's inherent limitations.

Answer 2

Is this a chrome bug?

In response to the query posed in Chrome, FileReader API, event.target.result === "", it is important to note that this limitation is not a bug, but rather an intentional constraint within V8 (the JavaScript engine used by Chrome and other platforms). The issue stems from the inability to construct a String exceeding 512MB on 64-bit systems due to V8's heap object limitations, as explained in this commit.

Why is there neither an error nor an exception?

While creating such a large string directly does result in a RangeError, certain operations lack the expected error handling mechanisms. As outlined in FileReader::readOperation Step 3, exceptions should trigger specific actions which are absent in this case.

Detailed steps involving Uint32Array and Blob further illustrate this anomaly, indicating a potential oversight that warrants attention and correction.

I will pursue this matter by raising a relevant issue to address the absence of error handling within the FileReader interface.

How can I fix or work around this issue?

A recommended approach involves modifying your API endpoint to accept binary resources directly instead of relying on data:// URLs, the usage of which is generally discouraged.

An alternative solution for future implementation would entail sending a ReadableStream to your endpoint and performing the data:// URL conversion autonomously using a stream sourced from the Blob.

(Code snippet omitted)

For immediate resolution, consider storing chunks of base64 representations in a Blob, although caution is advised due to potential server-side constraints related to handling excessively large strings. Communication with the API maintainer is strongly suggested to mitigate any issues arising from V8's inherent limitations.

Answer 3

Answer №2

Here is a clever method for converting a blob into chunks and then transforming them into base64 blobs. These base64 chunks are concatenated within a JSON blob along with some pre/suffix JSON parts.

By keeping it as a blob, the browser can efficiently manage memory allocation and even offload it to disk if necessary.

If you adjust the chunkSize to be larger, the browser prefers to store smaller blob chunks in memory (in one bucket).

// Obtaining a sample gradient file (blob)
var canvas=document.createElement("canvas"), context=canvas.getContext("2d"), gradient=context.createLinearGradient(0,0,3000,3000);canvas.width=canvas.height=3000;gradient.addColorStop(0,"red");gradient.addColorStop(1,"blue");context.fillStyle=gradient;context.fillRect(0,0,canvas.width,canvas.height);canvas.toBlob(main);

async function main (blob) {
  var fileReader = new FileReader()
  // It's recommended to add 2 so it omits == from all but the last chunk
  var chunkSize = (1 << 16) + 2 
  var position = 0
  var b64Chunks = []
  
  while (position < blob.size) {
    await new Promise(resolve => {
      fileReader.readAsDataURL(blob.slice(position, position + chunkSize))
      fileReader.onload = () => {
        const base64 = fileReader.result.split(',')[1]
        b64Chunks.push(new Blob([base64]))
        resolve()
      }
      position += chunkSize
    })
  }

  // How you combine all chunks into json is now up to you.
  // This solution just outlines what needs to be done
  // There are more automated ways, but here is a simple form
  // (just a heads-up: this new blob won't create a lot of data in memory, it will only reference other blob locations)
  const jsonData = new Blob([
    '{"data": "', ...b64Chunks, '"}'
  ], { type: 'application/json' })

  /*
  // It's strongly recommended to ask API developers 
  // to implement support for binary/file uploads (multipart-formdata)
  // Base64 is roughly ~33% bigger and handling streaming 
  // this data on the server to disk is nearly impossible 
  fetch('./upload-files-to-bad-json-only-api', {
    method: 'POST',
    body: jsonData
  })
  */
  
  // Just testing that it still functions
  //
  // new Response(jsonData).json().then(console.log)
  fetch('data:image/png;base64,' + await new Blob(b64Chunks).text()).then(response => response.blob()).then(blob => console.log(URL.createObjectURL(blob)))
}

I opted not to use

base64 += fileReader.result.split(',')[1]

and JSON.stringify since dealing with GiB of data is substantial and JSON isn't ideal for handling binary data.

Answer 4

Here is a clever method for converting a blob into chunks and then transforming them into base64 blobs. These base64 chunks are concatenated within a JSON blob along with some pre/suffix JSON parts.

By keeping it as a blob, the browser can efficiently manage memory allocation and even offload it to disk if necessary.

If you adjust the chunkSize to be larger, the browser prefers to store smaller blob chunks in memory (in one bucket).

// Obtaining a sample gradient file (blob)
var canvas=document.createElement("canvas"), context=canvas.getContext("2d"), gradient=context.createLinearGradient(0,0,3000,3000);canvas.width=canvas.height=3000;gradient.addColorStop(0,"red");gradient.addColorStop(1,"blue");context.fillStyle=gradient;context.fillRect(0,0,canvas.width,canvas.height);canvas.toBlob(main);

async function main (blob) {
  var fileReader = new FileReader()
  // It's recommended to add 2 so it omits == from all but the last chunk
  var chunkSize = (1 << 16) + 2 
  var position = 0
  var b64Chunks = []
  
  while (position < blob.size) {
    await new Promise(resolve => {
      fileReader.readAsDataURL(blob.slice(position, position + chunkSize))
      fileReader.onload = () => {
        const base64 = fileReader.result.split(',')[1]
        b64Chunks.push(new Blob([base64]))
        resolve()
      }
      position += chunkSize
    })
  }

  // How you combine all chunks into json is now up to you.
  // This solution just outlines what needs to be done
  // There are more automated ways, but here is a simple form
  // (just a heads-up: this new blob won't create a lot of data in memory, it will only reference other blob locations)
  const jsonData = new Blob([
    '{"data": "', ...b64Chunks, '"}'
  ], { type: 'application/json' })

  /*
  // It's strongly recommended to ask API developers 
  // to implement support for binary/file uploads (multipart-formdata)
  // Base64 is roughly ~33% bigger and handling streaming 
  // this data on the server to disk is nearly impossible 
  fetch('./upload-files-to-bad-json-only-api', {
    method: 'POST',
    body: jsonData
  })
  */
  
  // Just testing that it still functions
  //
  // new Response(jsonData).json().then(console.log)
  fetch('data:image/png;base64,' + await new Blob(b64Chunks).text()).then(response => response.blob()).then(blob => console.log(URL.createObjectURL(blob)))
}

I opted not to use

base64 += fileReader.result.split(',')[1]

and JSON.stringify since dealing with GiB of data is substantial and JSON isn't ideal for handling binary data.

Receiving an empty string from Chrome FileReader when dealing with large files (300MB or more)

Answer №1

Answer №2

Similar questions

When going through an array using jquery, only the final object is returned

Is there a way to leverage JavaScript to click on one div and modify the settings of another div simultaneously?

Creating a dynamic user interface in Angular 6 that successfully tracks changes without reliance on the parent

Tips for accessing the HTML content enclosed within a specific HTML tag using Python with Selenium

Error Encountered when Attempting to Retry AI SDK on Vercel's

ChartJS does not function properly when included in a .php file that is loaded using jQuery's .load() method (

Utilizing AngularJS to bind form fields with select boxes to enable synchronized data. Modifying the selection in the dropdown should dynamically

Select a Date: Input for Date Selection

The hydration error in next js is causing this code to malfunction

The function window.location.reload(true) is failing to properly refresh the page

Tips on enabling JS tooltips in Shadow DOM

Issue encountered while performing an Upsert operation on Pinecone using Node.js

The dynamic data graph generated by HIGHCHARTS Areaspline is not as effective as expected

The Google reCaptcha reply was "Uncaught (in promise) null"

The image will remain static and will not alternate between hidden and visible states

Is it possible to change the name of a PHP file using the value entered in an input

Issues with JQuery `.click()` event

SyntaxError: Unexpected symbol

JavaScript: Filtering an object array pushed from a MySQL query

Which specific jQuery functionality was made visible?