How to use JavaScript to retrieve extensive data streams (exceeding 1GB) from the internet

Question

How to use JavaScript to retrieve extensive data streams (exceeding 1GB) from the internet

I'm interested in finding out if there is a way to stream data directly from JavaScript to the browser's download manager.

With WebRTC, I am able to stream large amounts of data (files over 1GB) from one browser to another. On the receiving end, I save all this data into memory as an arraybuffer, essentially maintaining it as chunks. My goal is to allow users to download this data.

Issue : Blob objects have a size limit of around 600MB (varies by browser), preventing me from reconstructing the file from the chunks. Is there a method to stream these chunks directly for the browser to handle the downloads?

javascript blob webrtc

Answer 1

Answer №1

When looking to retrieve a large file blob from an API or URL, utilizing streamsaver is the way to go.

npm install streamsaver

Following the installation, proceed with the following steps:

import { createWriteStream } from 'streamsaver';

export const downloadFile = (url, fileName) => {
  return fetch(url).then(res => {
    const fileStream = createWriteStream(fileName);
    const writer = fileStream.getWriter();
    if (res.body.pipeTo) {
      writer.releaseLock();
      return res.body.pipeTo(fileStream);
    }

    const reader = res.body.getReader();
    const pump = () =>
      reader
        .read()
        .then(({ value, done }) => (done ? writer.close() : writer.write(value).then(pump)));

    return pump();
  });
};

To use this functionality, simply follow these steps:

const url = "http://urltobigfile";
const fileName = "bigfile.zip";

downloadFile(url, fileName).then(() => { alert('done'); });

Answer 2

When looking to retrieve a large file blob from an API or URL, utilizing streamsaver is the way to go.

npm install streamsaver

Following the installation, proceed with the following steps:

import { createWriteStream } from 'streamsaver';

export const downloadFile = (url, fileName) => {
  return fetch(url).then(res => {
    const fileStream = createWriteStream(fileName);
    const writer = fileStream.getWriter();
    if (res.body.pipeTo) {
      writer.releaseLock();
      return res.body.pipeTo(fileStream);
    }

    const reader = res.body.getReader();
    const pump = () =>
      reader
        .read()
        .then(({ value, done }) => (done ? writer.close() : writer.write(value).then(pump)));

    return pump();
  });
};

To use this functionality, simply follow these steps:

const url = "http://urltobigfile";
const fileName = "bigfile.zip";

downloadFile(url, fileName).then(() => { alert('done'); });

Answer 3

Answer №2

After taking advice from @guest271314, I incorporated StreamSaver.js into my current project and I was able to successfully download files larger than 1GB on Chrome. The documentation stated that it should support files up to 15GB, but unfortunately my browser crashed before reaching that limit (the maximum file size I managed to download was around 4GB).

Note I: In an attempt to bypass the Blob max size restriction, I also experimented with manually adding data to the href field of an <a></a> element, but this method failed when trying to download files of approximately 600MB...

Note II: Surprisingly, the conventional approach using createObjectURL function worked flawlessly on Firefox for files up to 4GB !!

Answer 4

After taking advice from @guest271314, I incorporated StreamSaver.js into my current project and I was able to successfully download files larger than 1GB on Chrome. The documentation stated that it should support files up to 15GB, but unfortunately my browser crashed before reaching that limit (the maximum file size I managed to download was around 4GB).

Note I: In an attempt to bypass the Blob max size restriction, I also experimented with manually adding data to the href field of an <a></a> element, but this method failed when trying to download files of approximately 600MB...

Note II: Surprisingly, the conventional approach using createObjectURL function worked flawlessly on Firefox for files up to 4GB !!

How to use JavaScript to retrieve extensive data streams (exceeding 1GB) from the internet

Answer №1

Answer №2

Similar questions

How can you notice when a DOM element is deleted from the page?

Steps to initiate my selenium-standalone server: Execute the command "selenium-standalone start"

Click on the input to add or remove a value

Having difficulty grasping the concept of toggleClass and jQuery selectors

Angular does not delay for promises to settle

Unable to produce scrolling animation using JavaScript

I am facing difficulties in installing the necessary node modules for my Angular project

Generating a safe POST connection with express.js

JavaScript and CSS animations offer dynamic and engaging ways to bring

Setting the current time in an Animation using ThreeJS

Utilizing a combination of Mongo, Mongoose, Multer, and FS for deleting images

This error message 'React Native _this2.refs.myinput.focus is not a function' indicates that

Is there a way for me to extract text from a leaflet popup in order to generate the complete URL for an AJAX request?

Incorporating dynamic images into Laravel using Angular

Optimizing JavaScript performance by packaging files with Browserify

The direction to the Excel document for conversion into JSON

Changing buffer from base64 to UTF-8 encoding in Node.js

Identify HTML elements within a textarea or text input using JavaScript while also accommodating the common special characters ">" and "<"

Building an Express API using ES6 script: A Step-by-Step Guide

Switching my Selenium code to HtmlUnit: A Step-by-Step Guide