Changing z-coordinate from perspective to orthographic camera in three.js

Question

Changing z-coordinate from perspective to orthographic camera in three.js

In my project, I am trying to blend perspective objects (which appear smaller as they move farther away) with orthographic objects (which maintain the same size regardless of distance). The perspective objects are a part of the rendered "world," while the orthographic objects serve as adornments like labels or icons. Unlike traditional HUD elements, I want these orthographic objects to be rendered within the world, meaning they can be obscured by other world objects, such as a plane passing in front of a label.

My approach involves using one renderer but two scenes - one with a PerspectiveCamera and the other with an OrthographicCamera. By rendering them sequentially without clearing the z-buffer (setting the renderer's autoClear property to false), I aim to achieve this effect. However, the challenge lies in synchronizing the placement of objects in each scene so that an object in one scene is assigned a proper z-position relative to objects in the opposite scene, both in front and behind it.

To tackle this problem, I have designated the perspective scene as the "leading" scene, where all object coordinates (perspective and orthographic) are set based on this scene. Perspective objects use these coordinates directly and are rendered accordingly with the perspective camera. Orthographic objects' coordinates are transformed to match those in the orthographic scene, then rendered using the orthographic camera. This transformation involves projecting the coordinates from the perspective scene onto the perspective camera's view pane and back onto the orthogonal scene with the orthographic camera:

position.project(perspectiveCamera).unproject(orthographicCamera);

However, despite my best efforts, this method is not yielding the desired results. The orthographic objects consistently render in front of the perspective objects, even when they should be positioned between them. For instance, consider an instance where a blue circle should appear behind a red square but in front of a green square (which it currently does not):

// Inserted code snippets here

The challenge persists, even though conceptually positioning orthographic objects before each render could potentially solve the issue.

UPDATE:

To address this problem, I have proposed a current solution in the form of an answer below. Despite this mitigation, the performance may not fully align with expectations compared to the orthographic camera itself. Therefore, any insights into why the orthographic camera behaves unexpectedly or alternative solutions would be greatly appreciated.

javascript three.js camera projection coordinate-transformation

Answer 1

Answer №1

You are very close to achieving the expected result. Don't forget to update the camera matrices for the project and unproject operations to work correctly:

pCam.updateMatrixWorld ( false );
oCam.updateMatrixWorld ( false );
circle.position.project(pCam).unproject(oCam);

Explanation:

In a rendering process, each mesh in the scene is typically transformed by the model matrix, view matrix, and projection matrix.

Projection matrix:
Describes the mapping from 3D points in the scene to 2D points on the viewport. Transforms from view space to clip space, then to normalized device coordinates (NDC) ranging from (-1, -1, -1) to (1, 1, 1) by dividing with the w component of clip coordinates.
View matrix:
Defines the direction and position from which the scene is viewed. Transforms the world space to the view (eye) space. In the viewport coordinate system, X-axis left, Y-axis up, Z-axis out of view (Note: in a right-hand system, Z-Axis is the cross product of X-Axis and Y-Axis).
Model matrix:
Specifies the location, orientation, and relative size of a mesh in the scene. Transform vertex positions from mesh to world space.

The depth value determines if a fragment is drawn "behind" or "before" another fragment. For orthographic projection, Z-coordinate of view space is linearly mapped to depth value, but in perspective projection it's non-linear.

Depth value calculation:

float ndc_depth = clip_space_pos.z / clip_space_pos.w;
float depth = (((farZ-nearZ) * ndc_depth) + nearZ + farZ) / 2.0;

Projection matrix transforms 3D points in scene to 2D points on the viewport, converting eye space to clip space then NDC by dividing clip coordinates' w component.

For Orthographic Projection, eye space coordinates are directly mapped to NDC.

Orthographic Projection

Z-component calculated using the linear function:

z_ndc = z_eye * -2/(f-n) - (f+n)/(f-n)

Perspective Projection

Z-component calculated using the rational function:

z_ndc = ( -z_eye * (f+n)/(f-n) - 2*f*n/(f-n) ) / -z_eye

Ensure the Z-coordinate of the circle in orthographic projection lies between depths of objects in perspective projection. Depth value formula is depth = z_ndc * 0.5 + 0.5, calculations can be done using NDC instead of depth values.

Normalized device coordinates found using the project method of THREE.PerspectiveCamera. The unproject converts NDC to view space then world space.

Example:

// JavaScript code snippet
var renderer, pScene, oScene, pCam, oCam, frontPlane, backPlane, circle;
// Initialization, rendering, and animation functions...
init();
animate();

// CSS style snippet
html,body {
    height: 100%;
    width: 100%;
    margin: 0;
    overflow: hidden;
}
// Load Three.js library
<script src="https://threejs.org/build/three.min.js"></script>

OpenGL - Mouse coordinates to Space coordinates

How to render depth linearly in modern OpenGL with gl_FragCoord.z in fragment shader?

Answer 2

You are very close to achieving the expected result. Don't forget to update the camera matrices for the project and unproject operations to work correctly:

pCam.updateMatrixWorld ( false );
oCam.updateMatrixWorld ( false );
circle.position.project(pCam).unproject(oCam);

Explanation:

In a rendering process, each mesh in the scene is typically transformed by the model matrix, view matrix, and projection matrix.

Projection matrix:
Describes the mapping from 3D points in the scene to 2D points on the viewport. Transforms from view space to clip space, then to normalized device coordinates (NDC) ranging from (-1, -1, -1) to (1, 1, 1) by dividing with the w component of clip coordinates.
View matrix:
Defines the direction and position from which the scene is viewed. Transforms the world space to the view (eye) space. In the viewport coordinate system, X-axis left, Y-axis up, Z-axis out of view (Note: in a right-hand system, Z-Axis is the cross product of X-Axis and Y-Axis).
Model matrix:
Specifies the location, orientation, and relative size of a mesh in the scene. Transform vertex positions from mesh to world space.

The depth value determines if a fragment is drawn "behind" or "before" another fragment. For orthographic projection, Z-coordinate of view space is linearly mapped to depth value, but in perspective projection it's non-linear.

Depth value calculation:

float ndc_depth = clip_space_pos.z / clip_space_pos.w;
float depth = (((farZ-nearZ) * ndc_depth) + nearZ + farZ) / 2.0;

Projection matrix transforms 3D points in scene to 2D points on the viewport, converting eye space to clip space then NDC by dividing clip coordinates' w component.

For Orthographic Projection, eye space coordinates are directly mapped to NDC.

Orthographic Projection

Z-component calculated using the linear function:

z_ndc = z_eye * -2/(f-n) - (f+n)/(f-n)

Perspective Projection

Z-component calculated using the rational function:

z_ndc = ( -z_eye * (f+n)/(f-n) - 2*f*n/(f-n) ) / -z_eye

Ensure the Z-coordinate of the circle in orthographic projection lies between depths of objects in perspective projection. Depth value formula is depth = z_ndc * 0.5 + 0.5, calculations can be done using NDC instead of depth values.

Normalized device coordinates found using the project method of THREE.PerspectiveCamera. The unproject converts NDC to view space then world space.

Example:

// JavaScript code snippet
var renderer, pScene, oScene, pCam, oCam, frontPlane, backPlane, circle;
// Initialization, rendering, and animation functions...
init();
animate();

// CSS style snippet
html,body {
    height: 100%;
    width: 100%;
    margin: 0;
    overflow: hidden;
}
// Load Three.js library
<script src="https://threejs.org/build/three.min.js"></script>

OpenGL - Mouse coordinates to Space coordinates

How to render depth linearly in modern OpenGL with gl_FragCoord.z in fragment shader?

Answer 3

Answer №2

I have discovered a unique method that utilizes only the perspective camera to adjust the size of adornments based on their distance from the camera. While it is akin to a solution mentioned in another response to a similar query, my approach has its own distinct characteristics. My specific challenge involves not just maintaining consistent adornment sizes regardless of distance, but also exerting precise control over their dimensions on screen.

To ensure they are scaled to an exact size rather than any arbitrary dimension, I implement a function for calculating on-screen size as detailed in this referenced answer. By computing the position of both ends of a vector with a known on-screen length and evaluating the projection distance to the screen, I can determine the precise scaling factor:

var widthVector = new THREE.Vector3( 100, 0, 0 );
widthVector.applyEuler(pCam.rotation);

var baseX = getScreenPosition(circle, pCam).x;
circle.position.add(widthVector);
var referenceX = getScreenPosition(circle, pCam).x;
circle.position.sub(widthVector);

var scale = 100 / (referenceX - baseX);
circle.scale.set(scale, scale, scale);

An issue with this approach is that while it typically yields accurate results, occasional rounding errors may cause adornments to render incorrectly.

Answer 4

I have discovered a unique method that utilizes only the perspective camera to adjust the size of adornments based on their distance from the camera. While it is akin to a solution mentioned in another response to a similar query, my approach has its own distinct characteristics. My specific challenge involves not just maintaining consistent adornment sizes regardless of distance, but also exerting precise control over their dimensions on screen.

To ensure they are scaled to an exact size rather than any arbitrary dimension, I implement a function for calculating on-screen size as detailed in this referenced answer. By computing the position of both ends of a vector with a known on-screen length and evaluating the projection distance to the screen, I can determine the precise scaling factor:

var widthVector = new THREE.Vector3( 100, 0, 0 );
widthVector.applyEuler(pCam.rotation);

var baseX = getScreenPosition(circle, pCam).x;
circle.position.add(widthVector);
var referenceX = getScreenPosition(circle, pCam).x;
circle.position.sub(widthVector);

var scale = 100 / (referenceX - baseX);
circle.scale.set(scale, scale, scale);

An issue with this approach is that while it typically yields accurate results, occasional rounding errors may cause adornments to render incorrectly.

Changing z-coordinate from perspective to orthographic camera in three.js

Answer №1

Explanation:

Orthographic Projection

Perspective Projection

Answer №2

Similar questions

The module 'three/examples/jsm/controls/OrbitControls' or its corresponding type declarations could not be located

Replace a portion of text with a RxJS countdown timer

Can an additional height be added to the equalizer to increase its height?

Introducing React JSX via a nifty bookmarklet

Creating a distinct folder for every upload in Node.js using multer and shortid

Output the JSON array containing [object Object] and [object Object]

Creating a slider plugin with Right-to-Left (RTL) support

Dealing with bodyParser errors in Express.js

Displaying imported JSON data in a tabular format

Tips for creating text that adjusts to the size of a div element

Guide to correctly setting up the b-table component in vue-test-utils

Retrieve the outcome of an Ajax request

Prevent submission until all form fields are filled with data

Arrange the array in chronological order based on the month and year

Is there a way to identify the top five elements that are most frequently occurring in the array?

Switching between sockets on a rotational basis

What are the best practices for managing live notifications with WebSocket technology?

Error: The function `map` is not available on the variable `filtredItems

Combining switch statements from various classes

Important notice: It is not possible to assign refs to function components. Any attempt to do so will result in failure. If you intended to assign a ref, consider