How to use JavaScript to retrieve extensive data streams (exceeding 1GB) from the internet

Question

How to use JavaScript to retrieve extensive data streams (exceeding 1GB) from the internet

I'm interested in finding out if there is a way to stream data directly from JavaScript to the browser's download manager.

With WebRTC, I am able to stream large amounts of data (files over 1GB) from one browser to another. On the receiving end, I save all this data into memory as an arraybuffer, essentially maintaining it as chunks. My goal is to allow users to download this data.

Issue : Blob objects have a size limit of around 600MB (varies by browser), preventing me from reconstructing the file from the chunks. Is there a method to stream these chunks directly for the browser to handle the downloads?

javascript blob webrtc

Answer 1

Answer №1

When looking to retrieve a large file blob from an API or URL, utilizing streamsaver is the way to go.

npm install streamsaver

Following the installation, proceed with the following steps:

import { createWriteStream } from 'streamsaver';

export const downloadFile = (url, fileName) => {
  return fetch(url).then(res => {
    const fileStream = createWriteStream(fileName);
    const writer = fileStream.getWriter();
    if (res.body.pipeTo) {
      writer.releaseLock();
      return res.body.pipeTo(fileStream);
    }

    const reader = res.body.getReader();
    const pump = () =>
      reader
        .read()
        .then(({ value, done }) => (done ? writer.close() : writer.write(value).then(pump)));

    return pump();
  });
};

To use this functionality, simply follow these steps:

const url = "http://urltobigfile";
const fileName = "bigfile.zip";

downloadFile(url, fileName).then(() => { alert('done'); });

Answer 2

When looking to retrieve a large file blob from an API or URL, utilizing streamsaver is the way to go.

npm install streamsaver

Following the installation, proceed with the following steps:

import { createWriteStream } from 'streamsaver';

export const downloadFile = (url, fileName) => {
  return fetch(url).then(res => {
    const fileStream = createWriteStream(fileName);
    const writer = fileStream.getWriter();
    if (res.body.pipeTo) {
      writer.releaseLock();
      return res.body.pipeTo(fileStream);
    }

    const reader = res.body.getReader();
    const pump = () =>
      reader
        .read()
        .then(({ value, done }) => (done ? writer.close() : writer.write(value).then(pump)));

    return pump();
  });
};

To use this functionality, simply follow these steps:

const url = "http://urltobigfile";
const fileName = "bigfile.zip";

downloadFile(url, fileName).then(() => { alert('done'); });

Answer 3

Answer №2

After taking advice from @guest271314, I incorporated StreamSaver.js into my current project and I was able to successfully download files larger than 1GB on Chrome. The documentation stated that it should support files up to 15GB, but unfortunately my browser crashed before reaching that limit (the maximum file size I managed to download was around 4GB).

Note I: In an attempt to bypass the Blob max size restriction, I also experimented with manually adding data to the href field of an <a></a> element, but this method failed when trying to download files of approximately 600MB...

Note II: Surprisingly, the conventional approach using createObjectURL function worked flawlessly on Firefox for files up to 4GB !!

Answer 4

After taking advice from @guest271314, I incorporated StreamSaver.js into my current project and I was able to successfully download files larger than 1GB on Chrome. The documentation stated that it should support files up to 15GB, but unfortunately my browser crashed before reaching that limit (the maximum file size I managed to download was around 4GB).

Note I: In an attempt to bypass the Blob max size restriction, I also experimented with manually adding data to the href field of an <a></a> element, but this method failed when trying to download files of approximately 600MB...

Note II: Surprisingly, the conventional approach using createObjectURL function worked flawlessly on Firefox for files up to 4GB !!

How to use JavaScript to retrieve extensive data streams (exceeding 1GB) from the internet

Answer №1

Answer №2

Similar questions

NodeJS npm module installed globally is unable to run the main/bin JavaScript file using node command

Effortlessly navigate between Formik Fields with automated tabbing

Examining Resolver Functionality within NestJS

Is there a way to automatically execute this function when the React component is loaded, without the need for a button click?

Patiently waiting for the component variable to be assigned through subscription

Refresh the screen after 10 seconds

Vue.js computed property fails to initiate

WordPress AJAX code encountered a http400 Bad Request

You can only set headers once during the initial request; any additional attempts to set headers will result in an

Resetting the selected value in an Angular2 select element after the user makes a change

What is the process for exporting a plugin from dayjs() in JavaScript?

Integrating external JavaScript widgets into the web application in real-time

Invoke a method on a child component once an event has been emitted

Loading animation reminiscent of a whirlpool, similar to the movement

Error: The meteor package encountered a SyntaxError due to an unexpected reserved word 'export'

HTML table row content should be aligned to the left side

PHP Form encountering error due to JSON decoding following an AJAX request

Is it possible to add animation to an SVG filter to create a captivating rotating planet with color effects?

Obtain identical socket event in two separate useEffect hooks

Having trouble extracting parameters from the router listen event in react-router?