Bun: Execute the file one line at a time

Question

Bun: Execute the file one line at a time

When working with NodeJs, I often use the fs module and the readline interface to read large text files line by line. However, I'm looking for a similar solution using Bun instead.

I attempted to read the file with this code snippet:

const input = Bun.file("input.txt");

Unfortunately, this method behaves like fs.readFile, which does not meet my needs for handling extremely large files.

Here is an example using Nodejs:

const readline = require("readline");
const fs = require("fs");


const input = fs.createReadStream("file.txt", {
  encoding: "utf16le",
});
const rl = readline.createInterface({ input });

rl.on("line", (line) => {
  // process each line here
})

javascript bun

Answer 1

Answer №1

As mentioned by @GrafiCode in the comments

const file = Bun.file("foo.txt");

This code snippet does not directly read the file from the disk, instead it creates a BunFile object which can be read in various ways, one of them being through streaming

await file.stream(); // contents as ReadableStream

For more information, you can visit this link

A demonstration on how to read a file line by line

async function readFileLineByLine(filePath, onLineRead, onFinish) {
  const file = Bun.file(filePath);

  const stream = await file.stream();
  const decoder = new TextDecoder();

  let remainingData = "";

  for await (const chunk of stream) {
    const str = decoder.decode(chunk);

    remainingData += str; // Append the chunk to the remaining data

    // Split the remaining data by newline character
    let lines = remainingData.split(/\r?\n/);
    // Loop through each line, except the last one
    while (lines.length > 1) {
      // Remove the first line from the array and pass it to the callback
      onLineRead(lines.shift());
    }
    // Update the remaining data with the last incomplete line
    remainingData = lines[0];
  }

  return onFinish();
}

function onLineRead(line) {
  console.log("Line read: " + line);
}

function onFinish() {
  console.log("File read successfully");
}

readFileLineByLine("2.txt", onLineRead, onFinish);

Answer 2

As mentioned by @GrafiCode in the comments

const file = Bun.file("foo.txt");

This code snippet does not directly read the file from the disk, instead it creates a BunFile object which can be read in various ways, one of them being through streaming

await file.stream(); // contents as ReadableStream

For more information, you can visit this link

A demonstration on how to read a file line by line

async function readFileLineByLine(filePath, onLineRead, onFinish) {
  const file = Bun.file(filePath);

  const stream = await file.stream();
  const decoder = new TextDecoder();

  let remainingData = "";

  for await (const chunk of stream) {
    const str = decoder.decode(chunk);

    remainingData += str; // Append the chunk to the remaining data

    // Split the remaining data by newline character
    let lines = remainingData.split(/\r?\n/);
    // Loop through each line, except the last one
    while (lines.length > 1) {
      // Remove the first line from the array and pass it to the callback
      onLineRead(lines.shift());
    }
    // Update the remaining data with the last incomplete line
    remainingData = lines[0];
  }

  return onFinish();
}

function onLineRead(line) {
  console.log("Line read: " + line);
}

function onFinish() {
  console.log("File read successfully");
}

readFileLineByLine("2.txt", onLineRead, onFinish);

Answer 3

Answer №2

NodeJS versus BunJS

In addition to Sarkar's response, another option is to utilize the node:fs library (which is fully compatible with bun).

There is an existing discussion on this topic in a node environment: Dan Dascalescu responds to Alex C's inquiry (Read a file one line at a time in node.js?).

Performance Evaluation

There are noticeable discrepancies in terms of performance. I conducted a comparison between Sarkar's bunjs solution using bun:stream and Dan Dascalescu's nodejs solution using node:stream. Additionally, two scenarios were tested where files were read as bulk and then split line by line (node:bulk and bun:bulk). The following are the results of the analysis.

Versions used: node v20.11.1, bun v1.1.6

📋 Results Run 1 (file: 5 MB)

  ⏱ node:stream = 24 ms
  ⏱ node:bulk = 4 ms
  ⏱ bun:stream = 8 ms
  ⏱ bun:bulk = 3 ms

📋 Results Run 2 (file: 368 MB)

  ⏱ node:stream = 470 ms
  ⏱ node:bulk = 204 ms
  ⏱ bun:stream = 716 ms
  ⏱ bun:bulk = 727 ms

📋 Results Run 3 (file: 5.5 GB)

  ⏱ node:stream = 18057 ms
  ⏱ node:bulk = 40004 ms
  ⏱ bun:stream = 25658 ms
  ⏱ bun:bulk = 35215 ms

📋 Results Run 4 (file: 12.8 GB)

  ⏱ node:stream = 46784 ms
  ⏱ bun:stream = 44367 ms

Conclusive Remarks

Unsurprisingly, stream-based file reading proves to be more efficient for larger files (approximately over 1GB), while bulk file reading exhibits better efficiency for smaller files (allowing for adaptability based on file size). When comparing node and bun solutions, it seems that bun's solution is notably slower than the node:fs solution (particularly evident for files > 100MB) up until files reach around 10GB in size. For larger files though, bun's performance starts catching up to node:fs.

This comparison could be further expanded upon (considering additional file sizes, impact of file structure - long lines vs short lines, etc.) in order to develop a smart switch mechanism for utilizing the most effective method based on specific file sizes.

Answer 4

NodeJS versus BunJS

In addition to Sarkar's response, another option is to utilize the node:fs library (which is fully compatible with bun).

There is an existing discussion on this topic in a node environment: Dan Dascalescu responds to Alex C's inquiry (Read a file one line at a time in node.js?).

Performance Evaluation

There are noticeable discrepancies in terms of performance. I conducted a comparison between Sarkar's bunjs solution using bun:stream and Dan Dascalescu's nodejs solution using node:stream. Additionally, two scenarios were tested where files were read as bulk and then split line by line (node:bulk and bun:bulk). The following are the results of the analysis.

Versions used: node v20.11.1, bun v1.1.6

📋 Results Run 1 (file: 5 MB)

  ⏱ node:stream = 24 ms
  ⏱ node:bulk = 4 ms
  ⏱ bun:stream = 8 ms
  ⏱ bun:bulk = 3 ms

📋 Results Run 2 (file: 368 MB)

  ⏱ node:stream = 470 ms
  ⏱ node:bulk = 204 ms
  ⏱ bun:stream = 716 ms
  ⏱ bun:bulk = 727 ms

📋 Results Run 3 (file: 5.5 GB)

  ⏱ node:stream = 18057 ms
  ⏱ node:bulk = 40004 ms
  ⏱ bun:stream = 25658 ms
  ⏱ bun:bulk = 35215 ms

📋 Results Run 4 (file: 12.8 GB)

  ⏱ node:stream = 46784 ms
  ⏱ bun:stream = 44367 ms

Conclusive Remarks

Unsurprisingly, stream-based file reading proves to be more efficient for larger files (approximately over 1GB), while bulk file reading exhibits better efficiency for smaller files (allowing for adaptability based on file size). When comparing node and bun solutions, it seems that bun's solution is notably slower than the node:fs solution (particularly evident for files > 100MB) up until files reach around 10GB in size. For larger files though, bun's performance starts catching up to node:fs.

This comparison could be further expanded upon (considering additional file sizes, impact of file structure - long lines vs short lines, etc.) in order to develop a smart switch mechanism for utilizing the most effective method based on specific file sizes.

Answer 5

Answer №3

I conducted my own performance tests.

Technique 1: Utilizing {createReadStream} from 'node:fs'

#!esn -i
import {createReadStream} from 'node:fs'

async function* readLines(filePath: string) {
    const stream = createReadStream(filePath, {
        encoding: 'utf-8',
    })

    let leftover = ''
    for await (const piece of stream) {
        let lines = (leftover + piece).split(/\r?\n/)
        leftover = lines.pop()!

        for(const line of lines) {
            yield line
        }
    }

    if(leftover) {
        yield leftover
    }
}


let totalLines = 0;
for await(const line of readLines(`${__dirname}/data.txt`)) {
    ++totalLines;
}
console.log(totalLines)

Technique 2: Using Bun.file

#!esn -i

async function* readLines(filePath: string) {
    const reader = Bun.file(filePath).stream().pipeThrough(new TextDecoderStream('utf-8')).getReader()

    let leftover = ''
    while(true) {
        const {value, done} = await reader.read()
        if(done) break
        let lines = (leftover + value).split(/\r?\n/)
        leftover = lines.pop()!

        for(const line of lines) {
            yield line
        }
    }

    if(leftover) {
        yield leftover
    }
}


let totalLines = 0
for await(const line of readLines(`${__dirname}/data.txt`)) {
    ++totalLines
}
console.log(totalLines)

Performance Test Results

#!esn
import {$} from 'bun'

await $`hyperfine --warmup 3 ${[
    `bun run ${__dirname}/node-read-stream.ts`,
    `bun run ${__dirname}/bun-stream.ts`,
]}`

Outcomes:

Test 1: bun run .../bench/node-read-stream.ts
  Time (mean ± σ):      89.4 ms ±   1.5 ms    [User: 90.0 ms, System: 55.1 ms]
  Range (min … max):    87.1 ms …  94.5 ms    32 runs

Test 2: bun run .../bench/bun-stream.ts
  Time (mean ± σ):      83.8 ms ±   1.2 ms    [User: 93.9 ms, System: 48.5 ms]
  Range (min … max):    82.2 ms …  87.6 ms    34 runs

Summary
  'bun run .../bench/bun-stream.ts' outperformed
    'bun run .../bench/node-read-stream.ts' by a margin of
    1.07 ± 0.02 in terms of speed.

The dataset comprises 99,991 lines of lorem ipsum.

Bun 1.1.25-canary.18+98a709fb1 on WSL environment.

Too Long; Didn't Read: Bun.file (Technique 2) showed slightly better performance.

Answer 6

I conducted my own performance tests.

Technique 1: Utilizing {createReadStream} from 'node:fs'

#!esn -i
import {createReadStream} from 'node:fs'

async function* readLines(filePath: string) {
    const stream = createReadStream(filePath, {
        encoding: 'utf-8',
    })

    let leftover = ''
    for await (const piece of stream) {
        let lines = (leftover + piece).split(/\r?\n/)
        leftover = lines.pop()!

        for(const line of lines) {
            yield line
        }
    }

    if(leftover) {
        yield leftover
    }
}


let totalLines = 0;
for await(const line of readLines(`${__dirname}/data.txt`)) {
    ++totalLines;
}
console.log(totalLines)

Technique 2: Using Bun.file

#!esn -i

async function* readLines(filePath: string) {
    const reader = Bun.file(filePath).stream().pipeThrough(new TextDecoderStream('utf-8')).getReader()

    let leftover = ''
    while(true) {
        const {value, done} = await reader.read()
        if(done) break
        let lines = (leftover + value).split(/\r?\n/)
        leftover = lines.pop()!

        for(const line of lines) {
            yield line
        }
    }

    if(leftover) {
        yield leftover
    }
}


let totalLines = 0
for await(const line of readLines(`${__dirname}/data.txt`)) {
    ++totalLines
}
console.log(totalLines)

Performance Test Results

#!esn
import {$} from 'bun'

await $`hyperfine --warmup 3 ${[
    `bun run ${__dirname}/node-read-stream.ts`,
    `bun run ${__dirname}/bun-stream.ts`,
]}`

Outcomes:

Test 1: bun run .../bench/node-read-stream.ts
  Time (mean ± σ):      89.4 ms ±   1.5 ms    [User: 90.0 ms, System: 55.1 ms]
  Range (min … max):    87.1 ms …  94.5 ms    32 runs

Test 2: bun run .../bench/bun-stream.ts
  Time (mean ± σ):      83.8 ms ±   1.2 ms    [User: 93.9 ms, System: 48.5 ms]
  Range (min … max):    82.2 ms …  87.6 ms    34 runs

Summary
  'bun run .../bench/bun-stream.ts' outperformed
    'bun run .../bench/node-read-stream.ts' by a margin of
    1.07 ± 0.02 in terms of speed.

The dataset comprises 99,991 lines of lorem ipsum.

Bun 1.1.25-canary.18+98a709fb1 on WSL environment.

Too Long; Didn't Read: Bun.file (Technique 2) showed slightly better performance.

Bun: Execute the file one line at a time

Answer №1

A demonstration on how to read a file line by line

Answer №2

Answer №3

Technique 1: Utilizing {createReadStream} from 'node:fs'

Technique 2: Using Bun.file

Performance Test Results

Similar questions

Display Default Image in Vue.js/Nuxt.js when Image Not Found

What does the JS Array Filter function return?

JavaScript text spacing

Setting default date and time for Bootstrap datetimepicker only in the expanded calendar view

How come this 2-dimensional array is adding elements unexpectedly?

Using conditional subscriptions in Meteor at the template level

Improving an unspecified JavaScript function

What happens when ES6 async/await interacts with Observables and streams during failures?

Make sure that the TextBox OnTextChanged event in ASP.NET triggers a "setTimeout" function before the OnClick event is fired

Implement a feature on React using hooks that detects when a user clicks

How to iterate through two arrays using AngularJS ng-repeat

Runtime hydration error triggered by React Interweave Library

endless refreshing material ui accordion

The essential guide to creating a top-notch design system with Material UI

Tips for canceling an http request in angularjs when the requesting controller has exited its scope

PHP versus JavaScript: A Comparison of XML Parsing Performance

Subscriber client successfully activated and operational via the command line interface

Adding QML code into a Jade file

displaying li tag depending on the index value within angularjs

Clicking on the input triggers the appearance of a border using the OnClick function