How can we manage JSON data in chunks during a progress event?

Question

How can we manage JSON data in chunks during a progress event?

My dilemma involves dealing with a server request that may bring back a substantial JSON list (around 100K records, approximately 50 Mb) of points to be presented on a canvas using D3js. To ensure interactivity and save memory, I am keen on rendering them as they stream in. Here's what I've done so far:

Firstly, I activated Chunked transfer encoding at the server end + Additionally, this is what I experimented with on the client side:

d3.json('?json=qDefects&operationid=' + opid) // my request 
  .on("load", function (json) {
    draw(json); // although it works, there's a significant delay that I want to eliminate...
  })
  .on("progress", function (json) {
   draw(json); // unfortunately, this fails: json isn't accessible at this point
  })
 .get();

I'm curious if there's a way to manage the JSON data in portions as it loads. Would restructuring the JSON data be beneficial? Currently, it's a single array setup like this:

[{"x":1, "y":2},{"x":2, "y":3}, // chunk 1
...
{"x":6845, "y":239426},{"x":51235, "y":234762}] // last chunk

Do you think splitting the points into smaller arrays would be advantageous?

javascript json d3.js xmlhttprequest chunked-encoding

Answer 1

Answer №1

Check out the provided fiddle: http://jsfiddle.net/Q5Jag/12412/

Although it is true that you cannot modify the progress event directly, a simple workaround is to use an external variable. By implementing the code below, you can reprocess the string and send it to d3:

var x = ''
d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    x = d.responseText
    x = "ehllo" + x;
    console.log(x)
  })
  .on("load", function() {
    console.log("done")
  })
  .get()

You have the flexibility to assign responseText to the variable x and manipulate x according to your requirements.

Answer 2

Check out the provided fiddle: http://jsfiddle.net/Q5Jag/12412/

Although it is true that you cannot modify the progress event directly, a simple workaround is to use an external variable. By implementing the code below, you can reprocess the string and send it to d3:

var x = ''
d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    x = d.responseText
    x = "ehllo" + x;
    console.log(x)
  })
  .on("load", function() {
    console.log("done")
  })
  .get()

You have the flexibility to assign responseText to the variable x and manipulate x according to your requirements.

Answer 3

Answer №2

After considering the suggestions from earlier responses, I have come up with the following solution:

function loadProgress(callback) {
  let start = 0;
  return function (event) {
    let str = event.responseText.substr(start); 
    let i = str.indexOf("{");                  
    let j = str.lastIndexOf("}");
    str = "[" + str.substr(i, j) + "]";
    let data = JSON.parse(str);
    callback(data);
    start = start + j + 1;
  }
}

d3.json('?json=qDefects&operationid=' + opid)
  .on("progress", loadProgress(draw));

This setup works effectively for my situation where there are no nested {}. I have ensured that the server provides chunks corresponding to each record of my request, resulting in matching {} in the responseText.

Although this approach still generates a lengthy and potentially unnecessary responseText, as well as an extra json parse step, I am able to manage it at the moment.

Answer 4

After considering the suggestions from earlier responses, I have come up with the following solution:

function loadProgress(callback) {
  let start = 0;
  return function (event) {
    let str = event.responseText.substr(start); 
    let i = str.indexOf("{");                  
    let j = str.lastIndexOf("}");
    str = "[" + str.substr(i, j) + "]";
    let data = JSON.parse(str);
    callback(data);
    start = start + j + 1;
  }
}

d3.json('?json=qDefects&operationid=' + opid)
  .on("progress", loadProgress(draw));

This setup works effectively for my situation where there are no nested {}. I have ensured that the server provides chunks corresponding to each record of my request, resulting in matching {} in the responseText.

Although this approach still generates a lengthy and potentially unnecessary responseText, as well as an extra json parse step, I am able to manage it at the moment.

Answer 5

Answer №3

In short: The JSON cannot be manipulated using the progress event.

To begin with, it is likely that you are using d3.request (D3 v3 and v4) instead of d3.fetch (D3 v5). It is important to note this distinction because both micro libraries have a method with the same name, d3.json. However, in the former, d3.json is an XMLHttpRequest while in the latter, it is a Promise.

More importantly, this appears to be (unfortunately) an XY problem. You mentioned wanting to "draw them as they arrive in order to favor interactivity and spare memory," but the issue is that even if you could manipulate the data as it arrives (which you cannot, as explained below), D3 will only start rendering after the XHR or Promise has finished downloading the data. This means that with 50MB of data, the user would face a blank page for several seconds. Therefore, it might be beneficial to reconsider the size of the data file and the overall data visualization approach.

Regarding your initial question:

The progress event is primarily used for monitoring progress. As per the W3 Consortium:

This specification defines an event interface — ProgressEvent — that can be used for measuring progress. (emphasis mine)

We can confirm this through the following example (using an array containing the objects you provided repeated multiple times). While we can access the loaded JSON using srcElement.response, it is not possible to modify it:

d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    console.log(d.srcElement.response)
  })
  .on("load", function() {
    console.log("done")
  })
  .get()

<script src="https://d3js.org/d3.v4.min.js"></script>

For instance, in this attempt to modify the string within the progress event, no changes occur:

d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    d.srcElement.response[0] = "foo";
    console.log("First character is: " + d.srcElement.response[0])
  })
  .on("load", function(data) {
    console.log("JSON:" + JSON.stringify(data))
  })
  .get()

<script src="https://d3js.org/d3.v4.min.js"></script>

Answer 6

In short: The JSON cannot be manipulated using the progress event.

To begin with, it is likely that you are using d3.request (D3 v3 and v4) instead of d3.fetch (D3 v5). It is important to note this distinction because both micro libraries have a method with the same name, d3.json. However, in the former, d3.json is an XMLHttpRequest while in the latter, it is a Promise.

More importantly, this appears to be (unfortunately) an XY problem. You mentioned wanting to "draw them as they arrive in order to favor interactivity and spare memory," but the issue is that even if you could manipulate the data as it arrives (which you cannot, as explained below), D3 will only start rendering after the XHR or Promise has finished downloading the data. This means that with 50MB of data, the user would face a blank page for several seconds. Therefore, it might be beneficial to reconsider the size of the data file and the overall data visualization approach.

Regarding your initial question:

The progress event is primarily used for monitoring progress. As per the W3 Consortium:

This specification defines an event interface — ProgressEvent — that can be used for measuring progress. (emphasis mine)

We can confirm this through the following example (using an array containing the objects you provided repeated multiple times). While we can access the loaded JSON using srcElement.response, it is not possible to modify it:

d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    console.log(d.srcElement.response)
  })
  .on("load", function() {
    console.log("done")
  })
  .get()

<script src="https://d3js.org/d3.v4.min.js"></script>

For instance, in this attempt to modify the string within the progress event, no changes occur:

d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    d.srcElement.response[0] = "foo";
    console.log("First character is: " + d.srcElement.response[0])
  })
  .on("load", function(data) {
    console.log("JSON:" + JSON.stringify(data))
  })
  .get()

<script src="https://d3js.org/d3.v4.min.js"></script>

How can we manage JSON data in chunks during a progress event?

Answer №1

Answer №2

Answer №3

Similar questions

Utilize only certain JSON properties within JavaScript

Utilizing Jquery to automatically scroll to a specific div on page load by setting an

Prevent ESLint from linting files with non-standard extensions

Achieve the central element while scrolling on the window

Parsley JS - Personalized Validation for Ensuring selected Items meet Minimum Value Requirements

Building a DOM element using jQuery

Remove any instances of null from an array

Use vue.js to add a block of content after every sixth iteration in a loop

The challenge of Cross-Origin Resource Sharing with AjaxSubmit compared to a traditional Ajax request

"Retrieve real-time information from the server directly on the client side

Expandable Grid Sections in React MUI

I'm currently working on incorporating an edit feature into the create form by utilizing server actions in the latest version of Next.js, version 14

Learning to parse a JSON file in Java

How can I access the data variables from a JSON request within a function?

Extracting text from an HTML file and passing it to an Express.js server: A beginner

Incorporating JSON data seamlessly into a visually appealing Highcharts pie chart

Revamping Legacy React Native Projects with the Redux Toolkit

jqGrid display/hide columns options dialogue box

What steps need to be taken to set up Node.js to accommodate requests from external sources beyond just localhost?

Having trouble with creating a new Next.js app using the latest version with npx?