HAMSt3R: Human-Aware 3D Reconstruction

• HAMSt3R is a human-aware multi-view stereo 3D reconstruction framework that recovers both detailed scene geometry and articulated human bodies.
• The system employs a Siamese Vision Transformer with cross-attention and multiple prediction heads, including instance segmentation and DensePose, to generate enriched 3D point clouds.
• Empirical evaluations show competitive performance in human-centric benchmarks and robust generalization in static scene reconstruction and pose regression tasks.

HAMSt3R is a human-aware multi-view stereo 3D reconstruction framework designed for recovering detailed 3D geometry from sparse, uncalibrated image sets, including challenging human-centric scenarios. It extends the MASt3R architecture by integrating advanced image encoding, additional human-focused prediction heads, and a unified end-to-end feed-forward design. HAMSt3R simultaneously reconstructs both the static environment and articulated human bodies, producing dense 3D point clouds enriched with semantic and pose information. The system demonstrates competitive performance on human-centric datasets (e.g., EgoHumans, EgoExo4D) while preserving strong results in standard scene reconstruction and pose regression benchmarks (Rojas et al., 22 Aug 2025).

1. Model Architecture

HAMSt3R is architecturally based on the MASt3R network, with key modifications to enable joint reconstruction of humans and scenes from pairs of images. The core pipeline comprises:

• A Siamese Vision Transformer (ViT) encoder, processing each input image to generate feature maps.
• A cross-attention ViT decoder, fusing features from both images to facilitate stereo correspondence reasoning.
• Multiple prediction heads, including:
  • The original pointmap and match descriptor head from MASt3R, predicting pixel-aligned 3D points and descriptors.
  • An instance segmentation head for human silhouette extraction.
  • A DensePose head mapping image pixels to 3D locations on the SMPL body model.
  • A binary mask head signaling regions for SMPL applicability.

All components are trained jointly with a composite loss:

$$\mathcal{L} = \mathcal{L}_{\text{MASt3R}} + \lambda_1 \mathcal{L}_{\text{seg}} + \lambda_2 \mathcal{L}_{\text{dp}} + \lambda_3 \mathcal{L}_{\text{mask}}$$

Typical loss weights are $\lambda_1 = 0.01$, $\lambda_2 = 1$, and $\lambda_3 = 1$, balancing segmentation, DensePose, and mask contributions.

2. Image Encoding via DUNE

The DUNE encoder is central to HAMSt3R’s capability to process human-centric imagery. It is a ViT-B/14 backbone trained through a multi-teacher distillation process integrating:

• The MASt3R encoder (scene geometry specialization),
• Multi-HMR (human mesh recovery expertise),
• A generalist image encoder (e.g., DINOv2).

This distillation is performed using the UNIC projection mechanism, yielding image features that are simultaneously sensitive to scene structure and human body cues. This fusion eliminates the need for separate pipeline stages for human and scene understanding, enabling robust generalization to diverse environments.

3. Prediction Heads and Output Representation

Beyond MASt3R’s original point and feature prediction, HAMSt3R introduces:

• Instance Segmentation Head: Inspired by transformer-based segmentation (e.g., Mask2Former), this branch generates binary masks distinguishing human from non-human pixels. Training employs a combination of classification, binary cross-entropy, and dice losses to obtain full-coverage, accurate silhouettes.
• DensePose Head: Outputs $P_\text{dp} \in \mathbb{R}^{H \times W \times 3}$ (dense surface correspondences) for each detected human region, mapping image pixels to SMPL mesh coordinates. Supervision uses an L2 regression loss: $$\mathcal{L}_\text{dp} = \|P_\text{dp} - P_\text{gt}\|^2$$.
• Mask Head: Generates a binary map $M \in \mathbb{R}^{H \times W}$ for selecting regions where SMPL fitting is valid, trained with a cross-entropy loss.

By projecting segmentation and DensePose outputs onto the reconstructed 3D pointmap, every predicted point is dichotomized as human or non-human and can be further annotated with detailed pose information. This enables direct SMPL model fitting by minimizing spatial distances between predicted 3D points and SMPL mesh vertices.

4. Multi-View 3D Reconstruction Pipeline

HAMSt3R’s reconstruction pipeline operates as follows:

• Paired images are encoded and fused, outputting dense 3D pointmaps in the reference view’s coordinate system.
• Human-centric heads concurrently provide per-pixel segmentation and DensePose (SMPL coordinate) annotations.
• Each 3D point is assigned semantic labels (human instance ID, background, etc.) by back-projecting image-space segmentations.
• When reconstructing from more than two views, the system applies the global alignment and instance-matching procedure inherited from MASt3R—merging mask overlaps and confidence scores to reconcile consistent human identities across views.
• The result is a unified dense 3D point cloud with structured semantic-labeled regions, suitable for seamless downstream applications, including SMPL mesh fitting and combined scene-human rendering.

5. Empirical Evaluation and Performance

HAMSt3R has been rigorously evaluated on both human-centric and traditional 3D vision benchmarks:

Dataset	Task	Key Metric	HAMSt3R Performance	Notable Prior Comparison
EgoExo4D	Human 3D pose reconstruction	W-MPJPE	0.51 m	Comparable/Better than HSfM, UnCaliPose
EgoExo4D	Human 3D pose reconstruction	PA-MPJPE	0.09 m	Competitive with SOTA
KITTI/ScanNet/ETH3D	General depth estimation	Depth errors	Slightly increased vs MASt3R	Remains competitive
CO3Dv2, RealEstate10K	Pose regression	Rotation/Trans.	Competitive with SOTA methods

• On human-centric datasets, HAMSt3R demonstrates strong capability in reconstructing articulated human bodies, preserving sub-decimeter error.
• Camera pose estimation (translation and angle error) is robust in small/mid-scale scenes.
• For general, non-human-centric tasks (depth estimation, pose regression), HAMSt3R exhibits only marginal degradation compared to specialist models, retaining practical usability across scenarios.
• Performance reduction in large-scale scenes can be attributed to non-linear loss scaling rather than fundamental model limits.

6. Generalization and Scope of Application

While tailored to handle humans explicitly, HAMSt3R inherits the general 3D scene reconstruction capabilities of MASt3R. Outside of human-centric tasks, it can:

• Produce dense depth maps and point clouds for static environments.
• Regulate performance via task-specific loss weighting, adapting priorities flexibly.
• Apply to mixed scenarios (e.g., robotics, augmented/virtual reality, human-computer interaction) where both articulated body and scene reconstruction are requisite without separate or sequential systems.

The architecture’s fully feed-forward nature, end-to-end trainability, and reliance only on image pairs (with or without calibration) promote efficiency, suitability for real-time applications, and practicality in unconstrained settings.

7. Significance and Broader Implications

HAMSt3R establishes a unified approach that dissolves the traditional boundaries between human and scene modeling in 3D vision. Key advances include:

• Multi-teacher distillation for cross-domain image feature extraction (scene and body),
• Synchronous prediction of dense geometry, instance semantic masks, and articulated pose,
• Plug-and-play adaptability to standard and human-enriched benchmarks.

By enabling feed-forward, efficient, and semantically structured 3D reconstructions from sparse, uncalibrated images, HAMSt3R sets a precedent for integrated scene understanding. Its demonstrated robustness across domains suggests wide applicability in computational systems requiring nuanced human-environment modeling (Rojas et al., 22 Aug 2025).