Joint Feed-Forward Geometry Models

• Joint feed-forward geometry models are deep neural architectures that simultaneously predict multiple geometric quantities in a single pass using transformer backbones and multi-branch decoding.
• They integrate permutation-equivariant data flow and explicit metric scale supervision to deliver dense scene reconstructions, human body meshes, and globally consistent camera parameters.
• The training strategy leverages staged supervision with synthetic priors and unlabeled in-the-wild data, achieving state-of-the-art performance in 3D reconstruction and human-scene alignment.

Joint feed-forward geometry models are deep neural architectures that, in a single pass over one or more images, simultaneously predict multiple geometric quantities—such as scene structure, camera pose, surface normals, depth maps, and articulated body meshes—unifying tasks traditionally handled by separate, sequential modules. These models leverage expressive transformer-based backbones, permutation-equivariant data flow, and multi-branch decoding to couple disparate geometric reasoning tasks. Modern joint feed-forward frameworks enforce global metric consistency, robustly align human and nonhuman entities, and increasingly integrate both strong synthetic priors and unlabeled in-the-wild data, achieving significant advances in both geometric fidelity and downstream application performance (Li et al., 3 Jan 2026).

1. Architectural Principles and Model Structure

Joint feed-forward geometry models, exemplified by systems such as UniSH, utilize unified transformer backbones with specialized decoding branches for distinct geometric primitives. A typical input is a monocular or multi-view video sequence $\{I_i\}_{i=1}^N$, processed through a stack of transformer layers (e.g., ViT-Large, 36 layers, $D=1024$) equipped with permutation equivariance and alternating spatial/global self-attention. The output suite in a single pass can include:

• Per-frame camera extrinsics $E_i = [R_i | T_i]$ and intrinsics $K_i$.
• Dense scene pointmaps $P_i \in \mathbb{R}^{H \times W \times 3}$ and confidence maps $C_i$.
• Parametric human body pose $\theta_i \in \mathbb{R}^{72}$, shape $\beta \in \mathbb{R}^{10}$, and translations $t_i$ via SMPL.
• A scalar global metric scale $s$, enforcing coherent units.

A hierarchical architecture typically decomposes into:

• Scene branch: multi-frame ViT encoding, cross-view fusion, and parallel decoders for geometry, pose, and confidence.
• Human-centric branch: person-detection-driven cropping, pose regression (e.g., CameraHMR with ViTPose-Base), with shape temporal consistency enforced by averaging over time.
• Alignment branch (e.g., AlignNet): transformer decoding fusing scene and human geometrics to yield metric scale and global translations (Li et al., 3 Jan 2026).

The core design ensures all outputs are delivered in a single forward pass without any iterative optimization at inference time.

2. Geometry Representations and Output Parameterization

Scene geometry is captured by per-pixel pointmaps $P_i(x, y) = [X, Y, Z]$ in each camera's frame, with confidence scores $C_i(x, y)$. Fused across frames, this gives a global point cloud $V_\text{scene} = \bigcup_{i, x, y} P_i(x, y)$. Human pose and shape are represented parametrically via the SMPL mesh $M(\theta_i, \beta)$, yielding a dense set of body vertices $V_\text{smpl},i \in \mathbb{R}^{6890 \times 3}$, with the visible human body pointcloud further extractable nonparametrically via predicted masks.

A crucial aspect is the explicit learning of a metric scale $s$, ensuring that all translations and pointmaps can be consistently expressed in real-world units (meters), a property essential for robotics, motion capture, and cross-dataset generalization (Li et al., 3 Jan 2026).

3. Training Paradigms: Multi-Stage Supervision and Losses

To bridge the synthetic–real data gap and enable high-fidelity reconstruction, training consists of multiple supervised and self-supervised stages:

Stage 1 (Human surface refinement, unlabeled real videos):

• Pseudo-depth $D_p$ from a monocular expert (e.g., MoGe-2) is distilled via confidence-aware local human losses, aligning local surface geometry patches by solving for rigid scale–shift and minimizing

$$\mathcal{L}_{h,i} = \frac{1}{K} \sum_{k=1}^K \frac{1}{|\mathcal{N}_k|} \sum_{j \in \mathcal{N}_k} C_k^j |s_k \hat{D}_k^j + t_k - D_k^j|$$

with KL-style patch-based regularization.

Stage 2 (Coarse alignment, synthetic data):

• Ground-truth supervision of SMPL mesh parameters and translations via weighted per-vertex, joint 3D/2D, pose, shape, and translation losses under the optimal synthetic scale $s_{\text{opt}}$.

Stage 3 (Fine-grained alignment, real videos):

• Chamfer alignment between rendered and predicted point clouds, depth-order regularization, and 2D reprojection loss:

$$\mathcal{L}_{\text{align},i} = \sum_{v \in V_{\text{src},i}} \min_{u \in V_{\text{tgt},i}} \| v - u \|_2^2$$

Regularization by

$$\mathcal{L}_{\text{dreg},i} = \mathrm{ReLU}(\bar{d}_{\text{tgt},i} - \bar{d}_{\text{src},i})$$

combines with joint-2D projection terms for final alignment (Li et al., 3 Jan 2026).

This curriculum achieves both fine human surface detail and robust global human-scene alignment.

4. Domain Adaptation and Utilization of Unlabeled Data

A pivotal challenge for joint geometry models is the sim-to-real gap arising from the scarcity of labeled real-world data. The paradigm outlined in UniSH uses unlabeled in-the-wild videos in two key capacities:

• Surface refinement (Stage 1), where high-frequency detail is distilled from a pseudo-depth expert trained on real images, refining human/body geometry in real backgrounds.
• Fine-scene alignment (Stage 3), where Chamfer and depth-order losses leverage the natural correspondences in real videos, allowing geometric adaptation without manual labeling (Li et al., 3 Jan 2026).

This two-stage approach obviates the need for expensive manual annotation, leveraging real-world diversity for model generalization and metric accuracy.

5. Inference, Outputs, and Downstream Application

At inference, a single forward pass produces per-frame camera parameters, point clouds, SMPL body meshes, and metric scales. Outputs include:

• Camera extrinsics/intrinsics $(E_i, K_i)$ and scene point cloud $s \cdot \bigcup_i P_i$.
• Human parameters $(\theta_i, \beta, t_i)$ and global body meshes $s \cdot (M(\theta_i, \beta) + t_i)$.
• Confidence maps for quantifying per-pixel geometric reliability.

These representations enable multiple downstream tasks, including global human motion estimation (evaluated on WA-MPJPE, W-MPJPE, RTE), human-centric scene reconstruction, AR/VR applications, and motion capture benchmarking. UniSH achieves AbsRel = 0.035 and $\delta < 1.25 = 0.980$ on the Bonn depth dataset, outperforming prior methods (e.g., $\pi^3$ and VGGT), and delivers competitive global body motion accuracy (EMDB-2: WA-MPJPE 118.5 mm, W-MPJPE 270.1 mm, RTE 5.8%) (Li et al., 3 Jan 2026).

6. Impact and Future Prospects

Joint feed-forward geometry models such as UniSH represent a key shift toward end-to-end differentiable systems capable of simultaneously reconstructing complex, interacting geometric entities (scenes and humans) at metric scale, exploiting unlabeled data and explicit multi-branch transformer designs. The compositionality of architecture (scene, body, alignment modules), explicit scale supervision, and reliance on strong synthetic and in-the-wild signals yield robust cross-domain performance, as confirmed in comprehensive benchmarking and ablations.

A critical insight from recent work is that direct metric scale supervision at the wrong pipeline stage degrades performance, necessitating careful multi-stage training. The effectiveness of self-supervised fine-tuning using unlabeled videos reflects a broader trend toward data-efficient, generalizable 3D geometry modeling.

This direction opens pathways for unified 3D perception systems in AR/VR, robotics, and human-computer interaction, driving research into further coupling of complex entities, handling of dynamic scenes, uncertainty quantification, and integration with generative frameworks. Continued advances in this area promise to narrow the gap between synthetic/simulated and real-world perception systems, making robust, multi-entity metric 3D reconstruction broadly accessible (Li et al., 3 Jan 2026).