Creating Augmented Reality Web Applications using Three.js integrated with MediaPipe

My friend and I are embarking on a project to create an AR web app for a University Workshop. Our goal is to track the hand movements and overlay a 3D Object on the wrist. However, we are facing two primary challenges:

  • How can we utilize the webcam in three.js as part of the scene?
  • How can we accurately detect the wrist landmark and place an object on it?

We appreciate any suggestions or guidance you may have.

So far, we have successfully implemented hand detection by using the webcam streaming video as input. However, when trying to integrate this with a scene and camera in Three.js, the components seem to operate independently.

UPDATE: While our existing code functions as expected, there seems to be an issue specifically related to Three.js:

import {
  HandLandmarker,
  FilesetResolver
} from "https://cdn.jsdelivr.net/npm/@mediapipe/<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="ed998c9e869ec09b849e848283adddc3dcddc3dd">[email protected]</a>";
import * as THREE from 'https://cdn.skypack.dev/<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="582c302a3d3d186876696b6a766a">[email protected]</a>';

// Rest of the code has been left unchanged...

The error message currently displayed in the console is as follows:

three.js:2744 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'center')
    at Sphere.copy (three.js:2744:29)
    at Frustum.intersectsObject (three.js:7224:15)
    at projectObject (three.js:15269:47)
    at projectObject (three.js:15292:7)
    at WebGLRenderer.render (three.js:15181:5)
    at render (script.js:163:14)
    at initializeThreeJs (script.js:167:3)
    at HTMLButtonElement.toggleWebcam (script.js:46:5)
javascriptthree.jsaugmented-realitymediapipe

Answer №1

Begin by ensuring that your x,y,z coordinates are clean and do not contain any additional information from MediaPipe. Here is an example:

coordinates = [
   {x:15, y:20, z:10},
   {x:25, y:30, z:35},
   {x:40, y:45, z:50}
]

Once you have the x,y,z information, you can use it as needed. For instance, to position an object at those points:

let geometry = new THREE.SphereGeometry( 0.5, 32, 32 )
let material = new THREE.MeshBasicMaterial( {color: 0xffff00} )
let sphere = new THREE.Mesh( geometry, material )
scene.add( sphere )

sphere.position.set(coordinates[0].x, coordinates[0].y, coordinates[0].z)

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

No element found with the specified exportAs value of "ngForm" on the <form> tag

I am currently experimenting with a template driven form in Angular, but I encountered an error stating **There is no directive with “exportAs” set to “ngForm"** I have made sure to import FormsModule and ReactiveFormsModule in app.module.ts as well ...

javascriptangularangular-formsangular-test

Using CSS to position an element relative/absolute within text inline

Need help aligning caret icons next to dynamically populated text in a navbar menu with dropdown tabs at any viewport size. Referring to positioning similar to the green carets shown here: https://i.stack.imgur.com/4XM7x.png Check out the code snippet bel ...

javascripthtmlcssflexbox

Using Formik: When the initial value is changed, the props.value will be updated

After the initial props are updated, it is important to also update the values in the forms accordingly. export default withFormik({ mapPropsToValues: (props: Props) => { return ( { id: props.initialData.id, ...

javascriptformsreactjstypescriptformik

Ensure the correct file extension is chosen when selecting a file for the 'file_field' and display any error messages using Ruby on Rails

I am currently using Ruby on Rails 3 and Prototype, and I need to be able to check the file extension when a file is selected with file_field. I only want to allow files with extensions of .doc or .pdf, any other extensions should display an error. In my ...

javascriptruby-on-railsrubyajaxprototype

Clicking an element to uncover more information

Currently, I am working on solving the second question within this series of problems. The task involves creating a functionality where clicking on a legislator's name displays additional information about them. You can view my progress so far by visi ...

javascriptjqueryapiget

"Provide information, then open a new webpage by clicking on a button. Perform various actions with the form by using type

Is there a way to quickly submit form entries to my database and then automatically redirect to a new page? My current situation requires that submitted information is stored in the DB before directing users to a thank you page, all in one seamless click! ...

javascriptphphtmlforms

A guide on enhancing Autocomplete Tag-it plugin with custom tags

Looking to expand the tags in the 'sampleTags' variable within this code snippet $(function () { var sampleTags = ['c++', 'java', 'php', 'coldfusion', 'javascript', 'asp', &apo ...

javascriptjqueryajaxjsontag-it

Enhancing the $routeProvider templateUrl request with extra data

I am working on adding headers to my request for fetching templates to filter out non-AngularJS requests and return an error message for them. It seems like the only way to achieve this is by using $http, but I feel like it would be a waste to recreate a ...

javascriptangularjsxmlhttprequestangularjs-routingng-view

Browsing a container with JavaScript

I am attempting to display one div at a time and scroll through them repeatedly. I found and modified a Fiddle that works as intended, but when I try to implement it on my own test page, the divs do not scroll as expected. Here is the Fiddle example: http ...

javascriptjqueryhtml

Monitoring the progress of file uploads within itemView

In the process of developing an application using Marionette/Backbone, I have successfully implemented file uploads over an AJAX call. Now, I am looking for a way to provide users with feedback on when they can select the uploaded file and proceed with mod ...

javascriptajaxbackbone.jsmarionette

``Is there a way to redirect users when they click outside the modal-content in Bootstrap

I have a modal on my website that currently redirects to the homepage when the close button is clicked. However, I also want it to redirect to the homepage when clicking outside of the bootstrap modal, specifically targeting the ".modal-content" class. I ...

javascriptjquerytwitter-bootstrapbootstrap-4

Is there a way to customize the ripple effect color on a Button?

As I'm exploring the API (v3.9.2), I have noticed a reference to TouchRippleProps for ButtonBase on https://material-ui.com/api/button-base/ The code for my button is as follows: <Button variant="text" size={"large"} ...

javascriptreactjsmaterial-ui

Encountering issues with posting data on a straightforward Node.js form using routes

I'm having trouble getting a simple form to post within my NodeJS/ExpressJS app. The code in my app.js file (referred to as server.js) is as follows: var express = require('express'); var fs = require('fs'); var path = require(&a ...

javascriptnode.jsexpressroutesform-submit

Implement lazy loading functionality in Django to dynamically load data as the user scrolls

Currently, I am developing a web application using Django to showcase the interface and how content is loaded. In my database, there could potentially be thousands of entries, and I am looking for a way to load only a certain number at a time to minimize ...

javascriptpythonajaxdjango

Adding static files to your HTML page with node.js

This is not a question about using express.static() In my application, I have multiple pages that share the same JS and CSS dependencies. Instead of adding <script> or <link> tags to every single page, I'm looking for a way to include the ...

javascriptphphtmlcssnode.js

Configure Jest to run tests for Lit using TypeScript with Microbundle

Today, I began the process of migrating our web components to lit-html. However, when attempting to run the initial test, an error message was encountered: SyntaxError: Cannot use import statement outside a module > 1 | import {LitElement, html, css, s ...

javascripttypescriptjestjsts-jestlit

Stop the user from navigating away from the current page if any changes have not been saved, all within React without relying on

My project includes a panel that opens as a modal from right to left. This panel contains a left navigation menu with various options that, when clicked, display different components. These components primarily consist of forms, and my goal is to alert th ...

javascriptreactjsfrontend

Dynamic Creation of a jQuery Selector

Dear, I'm facing an issue while trying to load data dynamically into a table using JSON format. The data is coming from a PHP file. However, despite my attempts, the table remains empty and no data is being displayed. function fetchTableData() { ...

javascriptjqueryjson

Extracting a specific variable value from a response by sending a request with CURL command and manipulating the data using JavaScript

Here is my Curl request: curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \ -X GET "https://api-cloud.browserstack.com/app-automate/sessions/22dbfb187486090d974a11ac91t65722988e0705.json" This is the Response I received: { "automati ...

javascriptjsonjenkinscurlgroovy

I'm looking to automate a system response whenever a user inputs something - how can I achieve this using NodeJS and Socket

I'm looking to create a feature where, if a user enters a specific command, the socket.io will respond with a predefined message. For example: Input: !hi Output: hello! If anyone knows how to achieve this, I'd appreciate some guidance, tha ...

javascriptnode.jssocket.io