SAM 3D: 3Dfy Anything in Images (2511.16624v1)

Abstract: We present SAM 3D, a generative model for visually grounded 3D object reconstruction, predicting geometry, texture, and layout from a single image. SAM 3D excels in natural images, where occlusion and scene clutter are common and visual recognition cues from context play a larger role. We achieve this with a human- and model-in-the-loop pipeline for annotating object shape, texture, and pose, providing visually grounded 3D reconstruction data at unprecedented scale. We learn from this data in a modern, multi-stage training framework that combines synthetic pretraining with real-world alignment, breaking the 3D "data barrier". We obtain significant gains over recent work, with at least a 5:1 win rate in human preference tests on real-world objects and scenes. We will release our code and model weights, an online demo, and a new challenging benchmark for in-the-wild 3D object reconstruction.

• The paper introduces a novel end-to-end model that generates detailed 3D object geometry, textures, and layouts directly from single images using a two-stage architecture.
• It employs a multi-stage generative training pipeline with synthetic pretraining, render-paste augmentation, and MITL alignment to overcome challenges like occlusion and clutter.
• The model significantly outperforms baselines in metrics such as F1, vIoU, Chamfer distance, and joint shape+pose accuracy, advancing general-purpose 3D perception.

SAM 3D: End-to-End 3D Object Reconstruction from Images

---

Overview

"SAM 3D: 3Dfy Anything in Images" (2511.16624) introduces SAM 3D, a foundation model for comprehensive 3D object reconstruction directly from single 2D images. The model predicts full 3D geometry, texture, and scene layout, handling extreme occlusion, clutter, and complex context within natural scenes. SAM 3D is enabled by a novel multi-stage data engine and human/model-in-the-loop (MITL) annotation pipeline, facilitating scalable acquisition of 3D supervision at unprecedented volume and diversity. The model leverages curriculum-inspired, multi-stage generative training that mirrors recent advances in LLM alignment and scaling.

---

Model Architecture and Pipeline

SAM 3D employs a two-stage architecture utilizing contemporary advancements in latent flow matching and mixture-of-transformers (MoT):

• Input Encoding: DINOv2 extracts multi-scale features from object crops, corresponding segmentation masks, and full-scene context; optional pointmap conditioning (e.g., LiDAR, monocular depth estimation) is also supported.
• Geometry Model: A 1.2B-parameter flow transformer in MoT configuration jointly predicts pose and coarse shape (voxel-based, $64^3$ resolution), capturing global and local spatial correlations.
• Texture Refinement: A 600M-parameter sparse latent flow transformer processes active voxels for high-fidelity geometric detail and texture synthesis, with outputs decoded via mesh or 3D Gaussian splat VAEs.
• Multi-Modal Prediction: The system learns to invert the 3D-to-2D mapping, modeling $p(S, T, R, t, s | I, M)$ generatively for shape $S$, texture $T$, pose $R$, translation $t$, and scale $s$, given image $I$ and mask $M$.

---

Data Engine and Annotation Strategy

The MITL data engine solves the core bottleneck of 3D data sparsity by synergizing synthetic pretraining, semi-synthetic render-paste augmentation, and iterative human/model preference alignment:

• Synthetic Pretraining (Iso-3DO): 2.7M Objaverse-XL object meshes rendered from 24 viewpoints, covering 2.5T training tokens, yield object-centric crop data to bootstrap shape/texture priors.
• Render-Paste Mid-Training (RP-3DO): 61M samples (occluders, occludees, object replacements) blend synthetic assets into real scenes with precise mask/pose mapping, augmenting the model’s robustness to layout and occlusion.
• Human/Model-in-the-Loop Post-Training: Preference-based selection for 3.14M shapes (and 100K textures), routed to expert 3D artists when all models fail (Art-3DO). The resulting SA-3DAO benchmark contains 1,000 high-fidelity artist-meshes paired to complex natural images.
• Expert Iteration and Data Flywheel: Annotators operate on best-of-$N$ candidate sets, continuously improving model proficiency through staged SFT and DPO on quality-controlled data, analogous to RLHF/RAFT/ExIT for policy amplification.

---

Training Paradigm

The multi-stage training mirrors efficient LLM pre/post-training protocols:

1. Pretraining: Isolated synthetic objects ($S$, $T$, $R$) on object-centric crops.
2. Mid-Training: Render-paste variants ($S$, $R$, $t$, $s$) on whole images and pointmaps, addressing occlusion, mask-following, and layout estimation.
3. Post-Training: SFT on MITL/Art-3DO data; DPO aligns the generative model to human preference, leveraging negative candidate sets for reward modeling; additional distillation reduces inference steps (shortcut models via consistency objectives).

Conditional rectified flow matching is used for multi-modality training. All stages were large-scale: pretraining/mid-training consumed up to 2.7T tokens. Post-training iteratively refines alignment using preference datasets, expert annotations, and high-threshold quality selection.

---

Evaluation and Empirical Results

Quantitative Performance

SAM 3D drastically outperforms contemporary baselines (Trellis, Hunyuan3D, Direct3D-S2, Hi3DGen, TripoSG) on real-world shape and layout reconstruction:

• SA-3DAO [email protected]: 0.2344 (SAM 3D) vs. 0.1475–0.1629 (others)
• vIoU: 0.2311 (SAM 3D) vs. 0.1266–0.1531 (others)
• Chamfer: 0.0400 (SAM 3D) vs. 0.0844–0.1126 (others)
• EMD: 0.1211 (SAM 3D) vs. 0.2049–0.2432 (others)
• Human Preference Win-Rate: At least 5:1 (object), 6:1 (scene) in pairwise preference tests across several benchmarks (SA-3DAO, LVIS, MetaCLIP).

For 3D scene layout on SA-3DAO and ADT, SAM 3D enjoys clear margin improvements:

• Joint Shape+Pose Prediction: ADD-S @ 0.1 jumps from <2% (pipeline baselines) to >77% (SAM 3D).
• ICP-Rotation Error and IoU: Markedly lower error after joint optimization and post-hoc render-and-compare refinement.

Ablations

Cascade training ablation demonstrates monotonic improvement with each additional data and alignment stage; removing MITL, Art-3DO, or DPO significantly degrades final accuracy. Texture ablations highlight the necessity of RP-3DO data, high-aesthetics selection, and lighting augmentation. Shortcut-model distillation enables sub-second generation with minimal fidelity loss.

---

Practical and Theoretical Implications

SAM 3D's robust single-image-to-3D reconstruction enables several actionable developments:

• General-Purpose 3D Perception: The foundation model paradigm unlocks direct, layout-aware, textured 3D asset creation for robotics, AR/VR, gaming, digital twin, and embodied AI applications—without reliance on multi-view, SLAM, or hand-crafted priors.
• Data Engine as Scalable Supervision: The MITL alignment and amplification loop establishes a template for bootstrapping supervision in sparse domains (e.g., 3D physical understanding, scene graph construction, physics modeling), leveraging synthetic data, human preference, and automated reward modeling.
• Model Scaling Laws: Methodological adoption of LLM training recipes corroborates scaling laws for transfer in generative perception, suggesting further gains should be attainable with continued data expansion and iterative alignment.
• Emergent Properties: Multi-modal, mixture-of-transformers architectures flexibly capture compositional scene structure, permitting modular extension to part-based, hierarchical, or implicit 3D representations.

---

Limitations and Future Directions

SAM 3D’s fixed output resolution ($64^3$ voxel grid; 32 splats/voxel) constrains fidelity for highly detailed, small-scale objects; architectural superresolution, implicit field modeling, or parts-based approaches can remedy this. Object interactions, physical contact, and global scene consistency remain outstanding, as layout estimation is per-object and does not incorporate physical constraints across multiple objects. Texture prediction remains agnostic to pose, occasionally producing rotationally inconsistent output for symmetric objects. Expanding domain coverage, enabling physics-aware reasoning, and integrating cross-modality inference will further enhance generalization and applicability.

---

Conclusion

SAM 3D establishes a new state-of-the-art for visually grounded, single-image 3D object and scene reconstruction. By integrating a scalable, preference-aligned data engine and a modern generative modeling pipeline, it sets a new bar for 3D perception in naturalistic environments. The model, data, and benchmarks are positioned to accelerate research across robotics, embodied AI, and content creation, and to catalyze further theoretical advances in general-purpose 3D foundation models.