What is the process ShaderToy uses to load sound files directly into a texture?

I'm attempting to replicate the functionality of shadertoy in passing audio frequency and waveform into a shader using three.js.

In this specific example, it appears that IQ is converting audio data into an image which is then utilized as a texture in the shader. How can I generate this audio texture in Javascript?

Just to clarify, I don't require assistance with loading the texture uniform into the shader. My challenge lies in creating the audio texture from an audio file.

var texture = new THREE.Texture();

shader.uniforms = {
     iChannel0:  { type: 't', value: texture }
};

I assume I'll need to encode audio data into the texture, but I am uncertain about how to proceed with this process.

Answer №1

To extract audio data using the Web Audio API, simply create an analyser node

const audioCtx = new window.AudioContext();
const analyzerNode = audioCtx.createAnalyser();

Next, set up a buffer to store the data obtained

const numberOfSamples = analyzerNode.frequencyBinCount;
const audioDataArray = new Uint8Array(numberOfSamples);

In your rendering loop, retrieve the data and transfer it into a texture

analyzerNode.getByteFrequencyData(audioDataArray);
...
gl.texImage2D(gl.TEXTURE_2D, 0, gl.LUMINANCE, numberOfSamples, 1, 0,
              gl.LUMINANCE, gl.UNSIGNED_BYTE, audioDataArray);

If you are using three.js, consider utilizing a DataTexture

This summarizes the process briefly. However, keep in mind that audio data should be from the same domain to avoid CORS challenges. For obtaining data from an audio stream like an <audio> element, utilize

const sourceNode = audioCtx.createMediaElementSource(audioElement);

Note that this method may not work in mobile Chrome or Safari due to existing bugs.

Check out this functional sample

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

unable to retrieve the .id attribute from a resource

When attempting to access $scope.mySlot.id, it appears to be undefined. $scope.removeMe = function() { var shouldRemove = confirm('Are you sure you want to remove yourself from this field trip?'); ...

Storing array data locally within an AngularJS application for seamless sharing between two separate applications

I need assistance with persisting and sharing array data var queries = []; between two separate AngularJS applications. One application is for the front end, while the other is for the admin side. Both applications cause the entire page to load when access ...

Issue arises after injecting HTML element into the DOM due to memory constraints and display inconsistencies

Upon executing the following script in Firefox... var d = $("<div class='test'></div>"); d.hide(); $("body").prepend(d); d.show(); ...and examining the HTML, the inserted element will have the style attribute: style="display: blo ...

Adjust the header image as you scroll

Currently, I have a static image in the header of the page. I'm looking to have the image change to a different one when half the page has been scrolled. Do I need to utilize JavaScript for this functionality or is it achievable with CSS alone? bo ...

Internet Explorer encounters errors when processing LESS code

Currently experimenting with the combination of LESS and respond.js to facilitate the development of a new website. Both LESS and respond are incredibly useful tools. However, encountered several issues with LESS in Internet Explorer. Initially, while in ...

Is there a way to retrieve YouTube URLs through programming automation?

I'm currently working on a project to automatically retrieve YouTube URLs and incorporate a download button feature. I found a tutorial suggesting the use of 'ytplayer.config.args.url_encoded_fmt_stream.map.split(",");' After attempting to ...

Incorporate 'Additional features' into the Navbar when adjusting window size

When the window is resized, I want to display a collapsed 'More options' button that will collapse all hidden <li> elements. Here is an example: <li id="menu_more_container" class="dropdown" style="display: none; ...

Converting JSON data into a JavaScript array and storing it in a variable

Currently, I am delving into the world of JavaScript and my instructor has assigned a task that involves utilizing information from a JSON file within our JavaScript code. The issue I'm facing is deciphering how to effectively convert the JSON data i ...

Angular with PHP Integration Success

I'm navigating the realms of Angular and PHP, understanding that Angular is client-side while PHP operates on the server. I'm curious if there's a way to merge these two technologies, as it's all quite unfamiliar territory for me. For ...

Flatpickr will refresh the list of days once a day is selected, causing any modifications made using onDayCreate to be reverted

After creating a Vue.js component that displays a customized calendar with different background colors for days, I encountered a problem. Whenever I select a day, all my customizations are lost because Flatpickr redraws the DOM elements for the days. How c ...

Incorrect behavior of responsive background images

For the first time, I am working on a school project that requires me to create a responsive website. I am following the "mobile first" approach. On stackoverflow, I came across a helpful way of making background images responsive: background-image: url(. ...

Angular 2 Update RC5: Router Provider Not Found

I'm encountering an issue with the latest Angular 2 RC5 router (router version is RC1). Below is the error log from the dev console: EXCEPTION: Error in /templates/app.component.html:2:0 ORIGINAL EXCEPTION: No provider for Router! This is how my ap ...

Can someone assist me in creating a clickable link that opens a menu in HTML when clicked?

I have been attempting for the past few days to open the megamenu by clicking on a link, but despite my efforts, I have not been successful. After reviewing some code, I discovered a clue in the CSS. It seems that setting the visibility value to visible wi ...

Leveraging the power of ES6 syntax in Node scripts with Babel

My npm CLI tool utilizes ES6 syntax from BabelJS, specifically arrow functions. Within the entry point of my tool, I'm using the following require: require('babel-core/register'); var program = require('./modules/program.js'); I ...

React Component: Issue with toggle functionality in child component not meeting expectations

I'm currently working on a checkbox user interface, where I need to pass a toggle function down to a list component. The issue I'm facing is that while I can check the checkbox successfully, I'm unable to uncheck it for some reason. Below i ...

Matching objects in an array based on a specific property using the key of another object

To display information in the UI based on matching values between the data.examples array object's name.value property and the keys in the wcObject.notCoveredList, we need to create a logic. If there is a match, all corresponding values from wcObject ...

What is the best way to identify the position of an element in an array given my specific circumstances

I am trying to utilize the ng-repeat directive in order to display an array. Here is what I have: <div ng-repeat="stu in school"> <div>{{stu.name}}</div> <div>{{stu.grade}}</div> </div> JavaScript $scope.scho ...

Retrieving an image from an input file and setting it as the background of a div element

I am currently developing a whack the mole game using Vanilla JS, and I'm attempting to allow players to choose their own target by uploading an image file. However, despite days of searching for a solution, I have not been successful. HTML: <inpu ...

Create a unique functionality by assigning multiple event handlers to a single event

I am looking to add a JavaScript function to an event that already has a handler function. The new function should complement the existing one rather than replace it. For instance: There is a function named exFunction() that is currently linked to docume ...

Customize the shadow array values in Material UI themes by utilizing the createMuiTheme function to override default

Currently, I have a customized theme for my toolkit where I am utilizing createMuiTheme to override various properties like palette, fonts, etc. One particular challenge I am facing is in minimizing the shadow array within the theme object which contains 2 ...