3D-Aware Kinematic Projection Overview

• 3D-aware kinematic projection is a strategy that reconstructs 3D structures from 2D inputs to more accurately capture motion and articulated dynamics.
• It leverages intermediate representations—such as spheres, kinematic chain spaces, or physics-based simulators—to simplify and regularize kinematic relationships.
• This approach enhances motion fidelity in applications like 360° video coding, monocular pose estimation, and articulated object manipulation, while facing challenges in depth approximation and complexity.

3D-aware kinematic projection denotes a family of methods that treat motion, articulation, or mechanical state in a geometry-aware intermediate representation rather than directly in the observed or coded 2D domain. Across video coding, human motion simulation, articulated manipulation, monocular pose estimation, image-to-video generation, and structured scene reasoning, the common pattern is to map an observation or control signal into a representation in which kinematic regularities are easier to express—such as a sphere, an articulated rigid-body simulator, a unified textual kinematic description, a kinematic chain space, a differentiable forward-kinematics pipeline, or a hierarchical 3D scene graph—and then map the result back into the target domain (Li et al., 2017, Siyao et al., 31 Jul 2025, Wandt et al., 2017, Kundu et al., 2020, Zhang et al., 8 Sep 2025, Li et al., 7 Jun 2026).

1. Definition and conceptual scope

In the strictest sense, 3D-aware kinematic projection is a projection strategy in which a system first reconstructs or parameterizes an underlying 3D structure, formulates motion or articulation there, and only then re-expresses the result in image, latent, simulation, or coding coordinates. In "Projection based advanced motion model for cubic mapping for 360-degree video" (Li et al., 2017), this takes the explicit form of mapping pixels from an unfolded cube to a 3D cube, then to a sphere, using the assumption that “most of the motions in the sphere are uniform,” and finally projecting the result back to the cube map. In "Half-Physics: Enabling Kinematic 3D Human Model with Physical Interactions" (Siyao et al., 31 Jul 2025), the analogous operation is to project a discrete SMPL-X pose trajectory into a physics simulation by converting framewise pose differences into continuous-time velocities. In "Zero-shot 3D-Aware Trajectory-Guided image-to-video generation via Test-Time Training" (Zhang et al., 8 Sep 2025), a 2D trajectory is converted into depth-scaled affine transforms and masks that are consistent with monocular scene depth.

A broader usage appears in works where the “projection” is representational rather than purely geometric. "A Kinematic Chain Space for Monocular Motion Capture" (Wandt et al., 2017) projects joint observations into a Kinematic Chain Space (KCS) built from bone vectors; "Kinematic-Structure-Preserved Representation for Unsupervised 3D Human Pose Estimation" (Kundu et al., 2020) imposes three differentiable transformations—forward-kinematics, camera-projection and spatial-map transformation; "Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs" (Xia et al., 2023) projects unified textual kinematic knowledge into 3D waypoints; and "PhysGraph: A Physics-aware 3D Scene Graph for Perception and Reasoning" (Li et al., 7 Jun 2026) maps RGB-D observations into a hierarchical graph with object geometry, articulation, and material properties.

This suggests that the term is best understood as an architectural principle rather than a single algorithmic template. The invariant feature is that the intermediate space is chosen so that kinematic relations are simpler, more structured, or more physically grounded than in the original observation space.

2. Core geometric mechanism

The canonical geometric formulation appears in cube-mapped 360-degree video coding. The problem is that a translational motion vector in the unfolded cube does not correspond to a physically uniform displacement on the sphere. The method in (Li et al., 2017) therefore first maps a pixel with unfolded-cube coordinates $(x_u,y_u)$ to face-dependent cube coordinates $(x_c,y_c,z_c)$, then projects that point to a sphere point $(x_s,y_s,z_s)$ subject to the sphere constraint

$x_s^2 + y_s^2 + z_s^2 = faceWidth^2/4$

and the collinearity constraint

$$\frac{x_s}{x_c} = \frac{y_s}{y_c} = \frac{z_s}{z_c}.$$

From these, the paper derives

$$x_s = \frac{faceWidth/2}{\sqrt{1+\frac{y_c^2}{x_c^2}+\frac{z_c^2}{x_c^2}}},\qquad y_s = y_c \times \frac{x_s}{x_c},\qquad z_s = z_c \times \frac{x_s}{x_c}.$$

The kinematic step then preserves sphere-domain offsets relative to a block center. If the current block center, the corresponding reference center, and another current pixel project to $(x_{s0},y_{s0},z_{s0})$, $(x_{s1},y_{s1},z_{s1})$, and $(x_{s2},y_{s2},z_{s2})$, the corresponding reference-sphere point is computed by

$$x_{s3} = x_{s1} - x_{s0} + x_{s2},\qquad y_{s3} = y_{s1} - y_{s0} + y_{s2},\qquad z_{s3} = z_{s1} - z_{s0} + z_{s2}.$$

After this, the result is mapped back to the 3D cube and then to unfolded cube coordinates. Fractional positions are supported using DCT-based interpolation filtering to $(x_c,y_c,z_c)$0-pixel precision, and the model is integrated into HEVC/HM through merge and AMVP motion-vector prediction (Li et al., 2017).

A different but related geometric front-end appears in Zo3T (Zhang et al., 8 Sep 2025). Here, monocular depth from the initial frame $(x_c,y_c,z_c)$1 is used to derive a perspective-aware scale factor

$(x_c,y_c,z_c)$2

where $(x_c,y_c,z_c)$3 is the depth at object center $(x_c,y_c,z_c)$4. The per-frame affine transform is

$(x_c,y_c,z_c)$5

which maps the initial box $(x_c,y_c,z_c)$6 to a depth-scaled projected box $(x_c,y_c,z_c)$7. The associated binary mask is

$(x_c,y_c,z_c)$8

The paper frames this as a 3D-aware kinematic projection because it converts a user’s 2D trajectory into a perspective-correct per-frame motion prior before latent optimization begins (Zhang et al., 8 Sep 2025).

A plausible implication is that geometric projection modules of this type are especially valuable when the observation domain contains systematic distortion—spherical projection in 360-degree video or perspective scale change in trajectory-guided video generation—that a purely planar control signal cannot encode.

3. Projection into kinematic or physical state spaces

Not all 3D-aware kinematic projection is purely geometric. Several systems project motion into a state space where dynamics, articulation, or topology become explicit.

In Half-Physics (Siyao et al., 31 Jul 2025), the body is converted from SMPL-X into an articulated rigid-body surrogate with 55 rigid parts. Each part $(x_c,y_c,z_c)$9 has position $(x_s,y_s,z_s)$0 and orientation $(x_s,y_s,z_s)$1, with the root aligned to global translation and rotation and the remaining parts aligned to the 54 joint angles. The core move is to convert a kinematic motion sequence

$(x_s,y_s,z_s)$2

into simulator-executable velocities. The global linear velocity is obtained by finite differencing,

$(x_s,y_s,z_s)$3

while angular velocities are computed by quaternion angular differences via SLERP / “angdiff.” The simulator then advances one step of size $(x_s,y_s,z_s)$4, yielding updated body and object states. The overall mapping is written as

$(x_s,y_s,z_s)$5

If no collision occurs, the body follows the assigned velocities and reproduces the original kinematic motion up to numerical rounding; if contact occurs, the simulator resolves it physically. Passive Joint Stiffness Compensation (PJSC) provides a corrective torque only when collision causes deviation from the prescribed trajectory (Siyao et al., 31 Jul 2025).

In KCS-based monocular motion capture (Wandt et al., 2017), the projection target is a kinematic-chain representation built from bone vectors rather than joint positions. A bone is defined by

$(x_s,y_s,z_s)$6

all bones are stacked into $(x_s,y_s,z_s)$7, and joint positions can be recovered by $(x_s,y_s,z_s)$8. The structural result is that

$(x_s,y_s,z_s)$9

has diagonal entries $x_s^2 + y_s^2 + z_s^2 = faceWidth^2/4$0, and the paper proves that the nuclear norm of $x_s^2 + y_s^2 + z_s^2 = faceWidth^2/4$1 is invariant for any bone configuration of the same person. This yields the relaxed optimization

$x_s^2 + y_s^2 + z_s^2 = faceWidth^2/4$2

The projection into KCS makes bone-length consistency, connectivity, and translation invariance directly accessible to optimization (Wandt et al., 2017).

In monocular video-based 3D object detection (Brazil et al., 2020), the projection is temporal and kinematic. Per-frame 3D detections are fused by a Kalman filter whose tracked state is

$x_s^2 + y_s^2 + z_s^2 = faceWidth^2/4$3

Objects are assumed to move along the direction of their orientation, so the state transition advances position by a scalar speed projected along orientation, followed by ego-motion compensation: $x_s^2 + y_s^2 + z_s^2 = faceWidth^2/4$4 This is 3D-aware because camera motion and object motion are explicitly separated (Brazil et al., 2020).

4. Differentiable projection pipelines for pose estimation

A separate lineage uses differentiable projection as a training bottleneck for 3D pose estimation. The defining idea is that pose should be encoded in physically meaningful kinematic variables, projected into image-compatible form, and supervised through reconstruction or reprojection consistency.

In the kinematic-structure-preserved unsupervised framework (Kundu et al., 2020), the pose encoder outputs local kinematic parameters $x_s^2 + y_s^2 + z_s^2 = faceWidth^2/4$5 and camera parameters $x_s^2 + y_s^2 + z_s^2 = faceWidth^2/4$6, while the appearance encoder outputs foreground appearance features $x_s^2 + y_s^2 + z_s^2 = faceWidth^2/4$7. The architecture then applies three consecutive differentiable transformations: forward kinematics $x_s^2 + y_s^2 + z_s^2 = faceWidth^2/4$8, camera projection $x_s^2 + y_s^2 + z_s^2 = faceWidth^2/4$9, and spatial-map transformation $$\frac{x_s}{x_c} = \frac{y_s}{y_c} = \frac{z_s}{z_c}.$$0, summarized by

$$\frac{x_s}{x_c} = \frac{y_s}{y_c} = \frac{z_s}{z_c}.$$1

The local kinematic representation uses one global trunk-to-hip-line angle and 13 local unit vectors, so $$\frac{x_s}{x_c} = \frac{y_s}{y_c} = \frac{z_s}{z_c}.$$2. Joint positions are reconstructed recursively by

$$\frac{x_s}{x_c} = \frac{y_s}{y_c} = \frac{z_s}{z_c}.$$3

which enforces skeletal connectivity and fixed bone-length ratios in canonical scale. Camera projection gives

$$\frac{x_s}{x_c} = \frac{y_s}{y_c} = \frac{z_s}{z_c}.$$4

and the 2D joints are converted into heat maps and affinity maps. The paper treats this pipeline as a suitable bottleneck stimulating pose disentanglement and enabling learning from in-the-wild videos without paired or unpaired weak supervisions beyond minimal skeletal prior knowledge (Kundu et al., 2020).

SPGNet (Wang et al., 2022) uses a supervised reprojection loop. The network predicts global position and relative 3D joint coordinates,

$$\frac{x_s}{x_c} = \frac{y_s}{y_c} = \frac{z_s}{z_c}.$$5

then reprojects the estimated 3D pose back to 2D key points along with spatial adjustments. The overall loss combines 3D regression, 2D reprojection, and a bone-length constraint: $$\frac{x_s}{x_c} = \frac{y_s}{y_c} = \frac{z_s}{z_c}.$$6 The kinematic term penalizes deviations from constant human bone length, which the paper identifies as a plausibility prior in monocular lifting (Wang et al., 2022).

These works differ in supervision regime, but they share an identical structural move: pose is not learned as an unconstrained latent vector. Instead, it is passed through a kinematically interpretable 3D representation and a projection operator before image-space consistency is enforced.

5. Articulated objects, language interfaces, and structured scene representations

For articulated objects, 3D-aware kinematic projection often means converting structural knowledge about joints and parts into executable 3D control signals or reconstruction variables.

In kinematic-aware prompting (Xia et al., 2023), the parser represents an articulated object as a unified textual kinematic description $$\frac{x_s}{x_c} = \frac{y_s}{y_c} = \frac{z_s}{z_c}.$$7 containing geometric-linked parts, kinematic joints, and contact point, serialized into a structured .xml format. The planner operates hierarchically: $$\frac{x_s}{x_c} = \frac{y_s}{y_c} = \frac{z_s}{z_c}.$$8 where $$\frac{x_s}{x_c} = \frac{y_s}{y_c} = \frac{z_s}{z_c}.$$9 is the instruction, $$x_s = \frac{faceWidth/2}{\sqrt{1+\frac{y_c^2}{x_c^2}+\frac{z_c^2}{x_c^2}}},\qquad y_s = y_c \times \frac{x_s}{x_c},\qquad z_s = z_c \times \frac{x_s}{x_c}.$$0 the abstract manipulation sequence, and $$x_s = \frac{faceWidth/2}{\sqrt{1+\frac{y_c^2}{x_c^2}+\frac{z_c^2}{x_c^2}}},\qquad y_s = y_c \times \frac{x_s}{x_c},\qquad z_s = z_c \times \frac{x_s}{x_c}.$$1 the sequence of 3D manipulation waypoints. The planner is explicitly designed to reason over coordinate and joint orientation, and then to emit waypoints composed from the action primitives move, grasp, release, clockwise rotate, and anti-clockwise rotate. The paper reports evaluation on 48 instances across 16 distinct categories, including zero-shot generalization to 8 unseen articulated object categories, and real-world experiments on 7 object categories with a Franka Panda robot (Xia et al., 2023).

KineDiff3D (Xu et al., 20 Oct 2025) addresses category-level articulated reconstruction from single-view partial point clouds by learning a shared latent prior over geometry, segmentation, and joints. Its Kinematic-Aware VAE encodes SDFs, joint angles $$x_s = \frac{faceWidth/2}{\sqrt{1+\frac{y_c^2}{x_c^2}+\frac{z_c^2}{x_c^2}}},\qquad y_s = y_c \times \frac{x_s}{x_c},\qquad z_s = z_c \times \frac{x_s}{x_c}.$$2, and part segmentation labels $$x_s = \frac{faceWidth/2}{\sqrt{1+\frac{y_c^2}{x_c^2}+\frac{z_c^2}{x_c^2}}},\qquad y_s = y_c \times \frac{x_s}{x_c},\qquad z_s = z_c \times \frac{x_s}{x_c}.$$3 into a latent code $$x_s = \frac{faceWidth/2}{\sqrt{1+\frac{y_c^2}{x_c^2}+\frac{z_c^2}{x_c^2}}},\qquad y_s = y_c \times \frac{x_s}{x_c},\qquad z_s = z_c \times \frac{x_s}{x_c}.$$4. The training loss is

$$x_s = \frac{faceWidth/2}{\sqrt{1+\frac{y_c^2}{x_c^2}+\frac{z_c^2}{x_c^2}}},\qquad y_s = y_c \times \frac{x_s}{x_c},\qquad z_s = z_c \times \frac{x_s}{x_c}.$$5

A diffusion model estimates base pose and joint parameters, and a second latent diffusion model reconstructs the kinematic-aware latent code from partial observations. Refinement is performed by minimizing

$$x_s = \frac{faceWidth/2}{\sqrt{1+\frac{y_c^2}{x_c^2}+\frac{z_c^2}{x_c^2}}},\qquad y_s = y_c \times \frac{x_s}{x_c},\qquad z_s = z_c \times \frac{x_s}{x_c}.$$6

while preserving articulation constraints (Xu et al., 20 Oct 2025).

PhysGraph (Li et al., 7 Jun 2026) pushes the same principle to scene scale. From RGB-D observations

$$x_s = \frac{faceWidth/2}{\sqrt{1+\frac{y_c^2}{x_c^2}+\frac{z_c^2}{x_c^2}}},\qquad y_s = y_c \times \frac{x_s}{x_c},\qquad z_s = z_c \times \frac{x_s}{x_c}.$$7

it reconstructs object-centric 3D geometry with 3D Gaussian Splatting, associates instances across views using geometric and semantic similarity,

$$x_s = \frac{faceWidth/2}{\sqrt{1+\frac{y_c^2}{x_c^2}+\frac{z_c^2}{x_c^2}}},\qquad y_s = y_c \times \frac{x_s}{x_c},\qquad z_s = z_c \times \frac{x_s}{x_c}.$$8

decomposes objects into functional parts, and infers articulation type, articulation position, and material/physical properties $$x_s = \frac{faceWidth/2}{\sqrt{1+\frac{y_c^2}{x_c^2}+\frac{z_c^2}{x_c^2}}},\qquad y_s = y_c \times \frac{x_s}{x_c},\qquad z_s = z_c \times \frac{x_s}{x_c}.$$9. The final graph contains object nodes, part nodes, spatial and functional edges, and is used for constraint-aware 3D affordance prediction and real-to-sim transfer. On 30 articulation-rich scenes from SceneFun3D, the paper reports 96.04% joint-type accuracy, minimum distance 6.33 cm, and orientation error 5.84° for articulation evaluation (Li et al., 7 Jun 2026).

A concise comparison of these representational forms is useful:

Domain	Projection target	Resulting control or inference object
Articulated manipulation (Xia et al., 2023)	Unified textual kinematic description	3D manipulation waypoints
Articulated reconstruction (Xu et al., 20 Oct 2025)	Kinematic-aware latent code	Full articulated shape and joint state
Scene understanding (Li et al., 7 Jun 2026)	Hierarchical 3D scene graph	Articulation, material, affordance, simulation

This suggests that language, latent generative models, and scene graphs can all function as projection spaces, provided they preserve joint structure, contact information, or part-level articulation.

6. Mechanical and projective foundations, performance patterns, and limitations

A mathematically explicit precursor to this family appears in projective mechanics (Baranyai, 2019). There, planar forces, velocities, and infinitesimal displacements are encoded as 3-vectors such as

$(x_{s0},y_{s0},z_{s0})$0

The projective plane $(x_{s0},y_{s0},z_{s0})$1 is constructed from $(x_{s0},y_{s0},z_{s0})$2 via

$(x_{s0},y_{s0},z_{s0})$3

and incidence is given by $(x_{s0},y_{s0},z_{s0})$4. Invertible $(x_{s0},y_{s0},z_{s0})$5 matrices, modulo scale, correspond to projective transformations and dualities of $(x_{s0},y_{s0},z_{s0})$6. The paper therefore shows that many planar mechanical transformations factorize into a linear map class plus an equilibrium- or compatibility-preserving congruence (Baranyai, 2019). This does not use the modern machine-learning vocabulary, but it makes explicit the deeper algebraic idea that a 3D vector system can represent a lower-dimensional mechanical geometry.

Across the contemporary literature, empirical behavior follows the geometry of the chosen projection space. In cube-mapped 360-degree video, HM-16.6 experiments under HEVC common test conditions report Y-component BD-rate savings of average $(x_{s0},y_{s0},z_{s0})$7 in RA, $(x_{s0},y_{s0},z_{s0})$8 in LD, and $(x_{s0},y_{s0},z_{s0})$9 in LP, with the strongest gain on Glacier at $(x_{s1},y_{s1},z_{s1})$0 in RA; the paper attributes the larger RA gain to larger temporal distance between reference frames and therefore more pronounced irregular motion (Li et al., 2017). In Half-Physics, the method is learning-free, real-time, and reported at about 953 fps in the dance-tracking test while preserving exact tracking fidelity when no collision occurs (Siyao et al., 31 Jul 2025). In Zo3T, an ablation replacing 3D-aware projection with raw 2D trajectory guidance changes FID from 74.83 to 76.12, FVD from 197.63 to 201.55, and ObjMC from 12.74 to 12.98, while the paper states that qualitative impact is stronger in scenes with noticeable depth changes, where unscaled 2D boxes cause “severe object distortions” (Zhang et al., 8 Sep 2025). In unsupervised human pose estimation, the kinematic-structure-preserved model reports 89.4 mm MPJPE on Human3.6M under protocol-II and 76.5 PCK / 39.8 AUC / 115.3 MPJPE on MPI-INF-3DHP in the fully unseen unsupervised setting (Kundu et al., 2020).

Several recurrent limitations also appear. In Zo3T, the 3D-aware module depends on monocular depth estimation from the first frame and remains a pseudo-3D approximation rather than a full scene reconstruction (Zhang et al., 8 Sep 2025). In Half-Physics, the human remains kinematically controlled while only the environment and objects are governed by physics, which is why the method is termed “half physics” rather than a full physically actuated control framework (Siyao et al., 31 Jul 2025). In kinematic-aware prompting, performance depends on accurate joint/contact estimation and on the LLM’s spatial/math reasoning (Xia et al., 2023). In PhysGraph, articulation inference is restricted to revolute and prismatic primitives, and runtime is about 25 minutes per 500-frame scene (Li et al., 7 Jun 2026). In KCS-based monocular motion capture, the method assumes a weak-perspective camera model and a kinematic-chain structure without translational joints (Wandt et al., 2017).

A common misconception is that any 2D warp, latent edit, or trajectory tokenization already constitutes 3D-aware kinematic projection. The literature described here does not support that view. The defining requirement is that the intermediate representation preserve or reconstruct the geometric, kinematic, or mechanical constraints that make the target motion physically or structurally coherent. Where that reconstruction is approximate, as in monocular depth-scaled masks or weak-perspective pose lifting, the methods remain useful but should be understood as partial rather than exhaustive 3D models (Zhang et al., 8 Sep 2025, Wandt et al., 2017).

Taken together, these works show that 3D-aware kinematic projection is not limited to a single discipline. It is a recurring strategy for dealing with distorted observation domains, discrete pose sequences, articulated object structure, and physically grounded scene reasoning by projecting them into a space where kinematics is easier to preserve, constrain, or optimize (Li et al., 2017, Siyao et al., 31 Jul 2025, Xia et al., 2023, Xu et al., 20 Oct 2025, Li et al., 7 Jun 2026).