How can we manage JSON data in chunks during a progress event?

Question

How can we manage JSON data in chunks during a progress event?

My dilemma involves dealing with a server request that may bring back a substantial JSON list (around 100K records, approximately 50 Mb) of points to be presented on a canvas using D3js. To ensure interactivity and save memory, I am keen on rendering them as they stream in. Here's what I've done so far:

Firstly, I activated Chunked transfer encoding at the server end + Additionally, this is what I experimented with on the client side:

d3.json('?json=qDefects&operationid=' + opid) // my request 
  .on("load", function (json) {
    draw(json); // although it works, there's a significant delay that I want to eliminate...
  })
  .on("progress", function (json) {
   draw(json); // unfortunately, this fails: json isn't accessible at this point
  })
 .get();

I'm curious if there's a way to manage the JSON data in portions as it loads. Would restructuring the JSON data be beneficial? Currently, it's a single array setup like this:

[{"x":1, "y":2},{"x":2, "y":3}, // chunk 1
...
{"x":6845, "y":239426},{"x":51235, "y":234762}] // last chunk

Do you think splitting the points into smaller arrays would be advantageous?

javascript json d3.js xmlhttprequest chunked-encoding

Answer 1

Answer №1

Check out the provided fiddle: http://jsfiddle.net/Q5Jag/12412/

Although it is true that you cannot modify the progress event directly, a simple workaround is to use an external variable. By implementing the code below, you can reprocess the string and send it to d3:

var x = ''
d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    x = d.responseText
    x = "ehllo" + x;
    console.log(x)
  })
  .on("load", function() {
    console.log("done")
  })
  .get()

You have the flexibility to assign responseText to the variable x and manipulate x according to your requirements.

Answer 2

Check out the provided fiddle: http://jsfiddle.net/Q5Jag/12412/

Although it is true that you cannot modify the progress event directly, a simple workaround is to use an external variable. By implementing the code below, you can reprocess the string and send it to d3:

var x = ''
d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    x = d.responseText
    x = "ehllo" + x;
    console.log(x)
  })
  .on("load", function() {
    console.log("done")
  })
  .get()

You have the flexibility to assign responseText to the variable x and manipulate x according to your requirements.

Answer 3

Answer №2

After considering the suggestions from earlier responses, I have come up with the following solution:

function loadProgress(callback) {
  let start = 0;
  return function (event) {
    let str = event.responseText.substr(start); 
    let i = str.indexOf("{");                  
    let j = str.lastIndexOf("}");
    str = "[" + str.substr(i, j) + "]";
    let data = JSON.parse(str);
    callback(data);
    start = start + j + 1;
  }
}

d3.json('?json=qDefects&operationid=' + opid)
  .on("progress", loadProgress(draw));

This setup works effectively for my situation where there are no nested {}. I have ensured that the server provides chunks corresponding to each record of my request, resulting in matching {} in the responseText.

Although this approach still generates a lengthy and potentially unnecessary responseText, as well as an extra json parse step, I am able to manage it at the moment.

Answer 4

After considering the suggestions from earlier responses, I have come up with the following solution:

function loadProgress(callback) {
  let start = 0;
  return function (event) {
    let str = event.responseText.substr(start); 
    let i = str.indexOf("{");                  
    let j = str.lastIndexOf("}");
    str = "[" + str.substr(i, j) + "]";
    let data = JSON.parse(str);
    callback(data);
    start = start + j + 1;
  }
}

d3.json('?json=qDefects&operationid=' + opid)
  .on("progress", loadProgress(draw));

This setup works effectively for my situation where there are no nested {}. I have ensured that the server provides chunks corresponding to each record of my request, resulting in matching {} in the responseText.

Although this approach still generates a lengthy and potentially unnecessary responseText, as well as an extra json parse step, I am able to manage it at the moment.

Answer 5

Answer №3

In short: The JSON cannot be manipulated using the progress event.

To begin with, it is likely that you are using d3.request (D3 v3 and v4) instead of d3.fetch (D3 v5). It is important to note this distinction because both micro libraries have a method with the same name, d3.json. However, in the former, d3.json is an XMLHttpRequest while in the latter, it is a Promise.

More importantly, this appears to be (unfortunately) an XY problem. You mentioned wanting to "draw them as they arrive in order to favor interactivity and spare memory," but the issue is that even if you could manipulate the data as it arrives (which you cannot, as explained below), D3 will only start rendering after the XHR or Promise has finished downloading the data. This means that with 50MB of data, the user would face a blank page for several seconds. Therefore, it might be beneficial to reconsider the size of the data file and the overall data visualization approach.

Regarding your initial question:

The progress event is primarily used for monitoring progress. As per the W3 Consortium:

This specification defines an event interface — ProgressEvent — that can be used for measuring progress. (emphasis mine)

We can confirm this through the following example (using an array containing the objects you provided repeated multiple times). While we can access the loaded JSON using srcElement.response, it is not possible to modify it:

d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    console.log(d.srcElement.response)
  })
  .on("load", function() {
    console.log("done")
  })
  .get()

<script src="https://d3js.org/d3.v4.min.js"></script>

For instance, in this attempt to modify the string within the progress event, no changes occur:

d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    d.srcElement.response[0] = "foo";
    console.log("First character is: " + d.srcElement.response[0])
  })
  .on("load", function(data) {
    console.log("JSON:" + JSON.stringify(data))
  })
  .get()

<script src="https://d3js.org/d3.v4.min.js"></script>

Answer 6

In short: The JSON cannot be manipulated using the progress event.

To begin with, it is likely that you are using d3.request (D3 v3 and v4) instead of d3.fetch (D3 v5). It is important to note this distinction because both micro libraries have a method with the same name, d3.json. However, in the former, d3.json is an XMLHttpRequest while in the latter, it is a Promise.

More importantly, this appears to be (unfortunately) an XY problem. You mentioned wanting to "draw them as they arrive in order to favor interactivity and spare memory," but the issue is that even if you could manipulate the data as it arrives (which you cannot, as explained below), D3 will only start rendering after the XHR or Promise has finished downloading the data. This means that with 50MB of data, the user would face a blank page for several seconds. Therefore, it might be beneficial to reconsider the size of the data file and the overall data visualization approach.

Regarding your initial question:

The progress event is primarily used for monitoring progress. As per the W3 Consortium:

This specification defines an event interface — ProgressEvent — that can be used for measuring progress. (emphasis mine)

We can confirm this through the following example (using an array containing the objects you provided repeated multiple times). While we can access the loaded JSON using srcElement.response, it is not possible to modify it:

d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    console.log(d.srcElement.response)
  })
  .on("load", function() {
    console.log("done")
  })
  .get()

<script src="https://d3js.org/d3.v4.min.js"></script>

For instance, in this attempt to modify the string within the progress event, no changes occur:

d3.json("https://api.myjson.com/bins/1d7yoi")
  .on("progress", function(d) {
    d.srcElement.response[0] = "foo";
    console.log("First character is: " + d.srcElement.response[0])
  })
  .on("load", function(data) {
    console.log("JSON:" + JSON.stringify(data))
  })
  .get()

<script src="https://d3js.org/d3.v4.min.js"></script>

How can we manage JSON data in chunks during a progress event?

Answer №1

Answer №2

Answer №3

Similar questions

How jQuery stops the submission of a form

Troubleshooting Android: Issues with Data Integration in MySQL Database Without any Error Notifications

Glitches and sudden jumps occur when using the useMediaQuery function in Material UI

Obtaining template attributes in CKEditor: A guide

How to toggle the visibility of an element in an array using AngularJS and JavaScript

Base64 Encoded Key for Implementing CURL Rest API

Having trouble with implementing the .addclass function in a dice roller project

Maximizing the potential of NestJS apps with Docker

Managing two separate instances with swiper.js

Using jQuery to deactivate buttons upon submission or clicking within forms and hyperlinks

Automatically preselect the checkbox inputs with the API and update their status when they are interacted with

Troubleshooting a Messaging Error between Background and Another Script

Theme.breakpoints.down not being acknowledged by MUI breakpoints

Exploring the possibilities of node-webkit: node-odbc encounters a setback

Is there no "on" function available in the Node readline module?

Is it possible to manipulate the attribute of an object using Object.defineProperty when a value is passed as a function parameter?

Accessing a specific item from a detailed JSON structure using PHP

Is this loader going through a triple loop?

Numerous Google Gauges featuring varying sets of options

Tips for inserting Weather Underground forecast icons