UniSH: Feed-Forward 3D Scene and Human Reconstruction

• UniSH is a fully feed-forward framework for joint metric-scale 3D scene and human reconstruction, integrating scene, human, and alignment subnetworks in a single pass.
• It employs a dual training strategy with robust distillation and a multi-stage supervision scheme, effectively mitigating sim-to-real domain gaps using large-scale unlabeled videos.
• Benchmarking results show state-of-the-art performance in scene reconstruction and human motion estimation, achieving high accuracy and real-time processing capabilities.

UniSH, introduced by Murphy et al., denotes a fully feed-forward framework for joint metric-scale 3D scene and human reconstruction from monocular video. In a single network pass, UniSH recovers scene point clouds, camera parameters, and globally aligned SMPL bodies, overcoming longstanding challenges arising from the sim-to-real domain gap and the need for coherent metric alignment between humans and their environment. UniSH consolidates distinct scene and human priors and leverages large-scale, unlabeled real-world video to close the generalization gap commonly observed with synthetic data-only training (Li et al., 3 Jan 2026).

1. Architectural Composition

UniSH comprises three principal subnetworks: a Scene Reconstruction Branch (π³-inspired), a Human Body Branch (CameraHMR-based), and AlignNet, a lightweight transformer fusion module. In a sequence of $N$ video frames $\{I_i\}$:

• The Scene Branch encodes cross-frame geometric features ($F_{geo}$), and decodes, per frame: camera extrinsics $E_i = [R_i|T_i]$, a per-pixel metric point cloud $P_i \in \mathbb{R}^3$, and a confidence map $C_i$. Camera intrinsics $K_i$ are inferred from $P_i$ via depth gradient analysis.
• Detected human crops $b_i$ and contextual features $\{b_i, K_i, I_i\}$ feed into the Human Branch, regressing SMPL pose $\{I_i\}$0, a shared body shape $\{I_i\}$1, and high-level human feature tokens $\{I_i\}$2.
• AlignNet (two-layer transformer decoder) receives key/value $\{I_i\}$3 and query $\{I_i\}$4, where $\{I_i\}$5 is a learned scale token. AlignNet outputs a global metric scene/human scale $\{I_i\}$6 and per-frame SMPL translations $\{I_i\}$7.

The output comprises metric-scale-aligned scene point clouds and SMPL-body parameterizations, all computed in a single forward pass.

2. Training Paradigm and Domain-Gap Mitigation

UniSH is trained with a dual strategy engineered for robustness in real-world generalization:

• Robust Distillation: Utilizes a monocular depth "expert" (MoGe-2) for local, confidence-weighted distillation specifically over the human-foreground regions. For every anchor point $\{I_i\}$8 in the human mask, local neighborhoods $\{I_i\}$9 in 3D are established (pseudo-depth/surface), and prediction is forced to match the expert in both scale and offset ($F_{geo}$0). Pre-trained scene priors are regularized by an $F_{geo}$1 term $F_{geo}$2.
• Two-Stage Supervision Scheme:
  • Stage 2: Coarse alignment is learned via full-annotation synthetic datasets (BEDLAM). Loss terms enforce SMPL-vertex, 2D/3D joint, pose, shape, and translation consistency versus ground truth, with an additional optimal scale regression loss.
  • Stage 3: Fine-tuning exploits geometric correspondence in real data. One-way Chamfer distance aligns visible SMPL mesh points with point clouds reconstructed from the scene under human-mask filtering; regularization on average depth ordering ensures plausible embeddings within the scene context.

This curriculum structurally preserves scene priors while transferring high-frequency detail and robust alignment to in-the-wild data.

3. Outputs and Their Parameterizations

UniSH yields, for each input sequence:

• Scene Geometry: Per-frame point clouds $F_{geo}$3, with downstream integration into TSDF or mesh representations possible.
• Camera Parameters: Intrinsics $F_{geo}$4 and extrinsics $F_{geo}$5 for each frame.
• Human Representation: SMPL pose $F_{geo}$6 (72D), shape $F_{geo}$7 (10D), and the visible body surface as a point cloud. Each body is globally placed and metric-aligned using the shared scale $F_{geo}$8 and per-frame translations $F_{geo}$9.

This unified output format enables both human-centric scene reconstruction and global human motion estimation relative to the reconstructed scene.

4. Training Regime and Implementation

• Dataset Composition: Synthetic data (BEDLAM) provides full supervision for Stage 2, enabling reliable initial alignment; 1.2M frames of unlabeled dance video constitute the real training corpus for domain transfer.
• Optimization: Training uses AdamW ($E_i = [R_i|T_i]$0, $E_i = [R_i|T_i]$1, weight decay 0.05), batch size 64, and 100 epochs per training stage. Learning rates adopt a 3-epoch warmup and cosine decay schedule.
• Domain Adaptation: Human surface distillation and the two-stage alignment strategy are crucial for bridging the synthetic-to-real gap. Direct metric supervision of the scene branch destroys the structural prior, and synthetic-only fine-tuning leads to poor real-world alignment.

5. Benchmarking, Results, and Ablations

Quantitative evaluations demonstrate that UniSH achieves state-of-the-art results in both human-centric scene reconstruction and global human motion estimation:

• On the Bonn benchmark, Abs Rel is 0.035 and $E_i = [R_i|T_i]$2 reach 0.980, surpassing both π³ (0.049/0.975) and other baselines.
• On EMDB-2 and RICH (global motion), UniSH attains WA-MPJPE 118.5 mm, W-MPJPE 270.1 mm, RTE 5.8%, competitive with or surpassing HMR-only methods while outputting full scenes.
• Ablation studies confirm that local human surface refinement on real videos is essential (no refinement: 0.049/0.975; full pipeline: 0.035/0.980). Synthetic-only or direct metric alignment severely degrade performance and generalizability.

6. Discussion, Strengths, and Limitations

UniSH is fully feed-forward and delivers real-time, single-pass joint scene and SMPL-based human reconstruction, with metric alignment and high-fidelity surface details. The methodology leverages large-scale unlabeled video to counteract the generalization limitations of synthetic data.

Notable limitations include the following: occluded or invisible human surfaces are not reconstructed (reconstruction is limited to the observed geometry), and non-parametric surfaces may introduce artifacts such as floating points ("floaters"). Future directions include using the globally aligned SMPL mesh as a prior for further regularization and human surface refinement (Li et al., 3 Jan 2026).

7. Impact, Applications, and Future Directions

UniSH establishes a new practical baseline for applications requiring coherent metric-scale scene and human reconstruction from monocular video, such as embodied AI, AR/VR environment synthesis, and activity understanding. The feed-forward design and robust domain transfer open avenues for integration with larger-scale perception systems; future extensions may enhance non-visible surface completion and further leverage geometric priors for refinement.