Selecting the quartile value for every individual data point

Question

Selecting the quartile value for every individual data point

I am currently graphing the sentiment values of tweets over a span of 10 years. The CSV file contains three columns as outlined below.

Successfully, I managed to plot each value by date. However, upon attempting to create an area graph, I encountered an issue where each date had multiple values.

This complication arises from each data point being derived from a single tweet, resulting in one x point having multiple y values.

In order to address this dilemma, I attempted to choose the quartile value of each date or select the largest or smallest y value for clarity. An example is provided below.

https://i.sstatic.net/xxLQi.png

For instance, January 8 exhibits multiple y values (textblob).

My objective is to generate an area graph while selecting either the largest value or the 2nd quartile value of each point.

How can I effectively pick the points?

The intention is to utilize the points in the following code as x/y coordinates for line or area graphs.

  function* vlinedrawing(data){
         for(let i;i<data.length;i++){
            if( i%500==0) yield svg.node();
            let px = margin+xscale(data[i].date)
            let py = height-margin-yscale(data[i].vader)
            paths.append('path')
            .attr('x',px)
            .attr('y',py)   
     }     
            yield svg.node()  
         }

The complete code can be found at the following link.

https://jsfiddle.net/soonk/uh5djax4/2/

Thank you in advance. ( The reason why it is a generator is that I'm going to visualize the graph in animated way)

javascript d3.js data-visualization

Answer 1

Answer №1

To determine the 2nd quartile, you have a couple of options. One way is to utilize d3.quantile as follows:

d3.quantile(dataArray, 0.5);

Alternatively, since the 2nd quartile is essentially the median, you can simply use:

d3.median(dataArray);

Using d3.quantile provides more flexibility, allowing you to adjust the specified quartile by changing the p value.

If we consider your data without parsing dates (making it suitable for unique values using a Set), here's a potential solution:

const aggregatedData = [...new Set(data.map(function(d) {
    return d.date
}))].map(function(d) {
    return {
      date: parser(d),
      textblob: d3.quantile(data.filter(function(e) {
        return e.date === d
      }).map(function(e) {
        return e.textblob
      }), 0.5)
    }
});

This code serves as a guide and may not be optimized due to nested loops. You are encouraged to enhance its performance.

For demonstration purposes, please find below the script:

var parser = d3.timeParse("%m/%d/%y");

d3.csv('https://raw.githubusercontent.com/jotnajoa/Javascript/master/tweetdata.csv', row).then(function(data) {

  const aggregatedData = [...new Set(data.map(function(d) {
    return d.date
  }))].map(function(d) {
    return {
      date: parser(d),
      textblob: d3.quantile(data.filter(function(e) {
        return e.date === d
      }).map(function(e) {
        return e.textblob
      }), 0.5)
    }
  });

  console.log(aggregatedData)

});

function row(d) {
  d.vader = +d.vader;
  d.textblob = +d.textblob;
  return d;
}

<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.min.js"></script>

Answer 2

To determine the 2nd quartile, you have a couple of options. One way is to utilize d3.quantile as follows:

d3.quantile(dataArray, 0.5);

Alternatively, since the 2nd quartile is essentially the median, you can simply use:

d3.median(dataArray);

Using d3.quantile provides more flexibility, allowing you to adjust the specified quartile by changing the p value.

If we consider your data without parsing dates (making it suitable for unique values using a Set), here's a potential solution:

const aggregatedData = [...new Set(data.map(function(d) {
    return d.date
}))].map(function(d) {
    return {
      date: parser(d),
      textblob: d3.quantile(data.filter(function(e) {
        return e.date === d
      }).map(function(e) {
        return e.textblob
      }), 0.5)
    }
});

This code serves as a guide and may not be optimized due to nested loops. You are encouraged to enhance its performance.

For demonstration purposes, please find below the script:

var parser = d3.timeParse("%m/%d/%y");

d3.csv('https://raw.githubusercontent.com/jotnajoa/Javascript/master/tweetdata.csv', row).then(function(data) {

  const aggregatedData = [...new Set(data.map(function(d) {
    return d.date
  }))].map(function(d) {
    return {
      date: parser(d),
      textblob: d3.quantile(data.filter(function(e) {
        return e.date === d
      }).map(function(e) {
        return e.textblob
      }), 0.5)
    }
  });

  console.log(aggregatedData)

});

function row(d) {
  d.vader = +d.vader;
  d.textblob = +d.textblob;
  return d;
}

<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.min.js"></script>

Selecting the quartile value for every individual data point

Answer №1

Similar questions

Please use Shift + Enter feature only when using desktop devices

Send the form via ajax

The issue with IE-9 is that it is mistakenly picking up the placeholder value as

retrieve object from s3 using angular and open it as a pdf

The sticky navigation and scroll to top features both function perfectly on their own, but when used simultaneously, they do not work

Capturing page titles accurately for timeonsite tracker in a single-page Angular app is challenging when navigating to other pages

Looking to retrieve the AssetLoadedFunc properties in the LoadAssets function? Wondering if you should use TypeScript or JavaScript

Tips for presenting HTML source code with appropriate tag coloring, style, and indentation similar to that found in editors

Discover the method for inserting a title attribute value into a span element

Sorting data in a Footable table by an unfinished line

Why is it that despite passing all the unit tests successfully, the function fails when used in its actual context with the same parameters?

Accessing index.html via file:// from Vue-cli template

I'm attempting to integrate the map function into my React Redux application, but after implementing useDispatch, I'm encountering an error message in the console

When a user clicks on a specific element's id, the image will rotate accordingly

What exactly is the purpose of measuring "First Load JS" in the @next/bundle-analyzer tool?

Trouble with Bootstrap Collapse feature not collapsing on its own

Can you provide a way to directly calculate the total length of all arrays within an array of objects?

Unable to directly assign a variable within the subscribe() function

Learn how to showcase the current date by utilizing JavaScript arrays and the getDate method!

Highlighting menu elements when landing on a page and making another menu element bold upon clicking in WordPress