ShapeR: Robust Conditional 3D Shape Generation from Casual Captures

Abstract: Recent advances in 3D shape generation have achieved impressive results, but most existing methods rely on clean, unoccluded, and well-segmented inputs. Such conditions are rarely met in real-world scenarios. We present ShapeR, a novel approach for conditional 3D object shape generation from casually captured sequences. Given an image sequence, we leverage off-the-shelf visual-inertial SLAM, 3D detection algorithms, and vision-LLMs to extract, for each object, a set of sparse SLAM points, posed multi-view images, and machine-generated captions. A rectified flow transformer trained to effectively condition on these modalities then generates high-fidelity metric 3D shapes. To ensure robustness to the challenges of casually captured data, we employ a range of techniques including on-the-fly compositional augmentations, a curriculum training scheme spanning object- and scene-level datasets, and strategies to handle background clutter. Additionally, we introduce a new evaluation benchmark comprising 178 in-the-wild objects across 7 real-world scenes with geometry annotations. Experiments show that ShapeR significantly outperforms existing approaches in this challenging setting, achieving an improvement of 2.7x in Chamfer distance compared to state of the art.

• The paper introduces a large-scale rectified flow model for robust, metric, object-level 3D reconstruction from casual, multi-view captures.
• It fuses geometric cues from SLAM points, multi-view images, camera poses, and language captions to denoise latent representations within a VAE framework.
• Extensive data augmentation and two-stage training, supported by a novel evaluation dataset, demonstrate its superior performance in challenging scenarios.

ShapeR: Robust Conditional 3D Shape Generation from Casual Captures

Motivation and Context

Existing 3D generative models achieve high-fidelity shape reconstruction from clean, segmented inputs, but their reliability degrades significantly in real-world settings with occlusions, clutter, poor viewpoints, and sensor noise. These conditions are typical for human-captured sequences that are not conducted with careful scanning protocols. Scene-level models often deliver incomplete object geometry under occlusion, while object-level methods—especially those relying on off-the-shelf segmentation—struggle to maintain fidelity and consistency when presented with real, casual, in-the-wild image sequences.

Figure 1: Scene examples reveal the major challenges casual captures introduce—clutter, occlusion, low resolution, noise, and ambiguous object boundaries all degrade existing SoTA model performance, while ShapeR reconstructs robustly.

ShapeR is introduced as a large-scale rectified flow model explicitly designed for metrically accurate, robust, and automatic object-level 3D reconstruction from casually captured, posed multi-view sequences. The method targets the core limitations of current approaches by deeply integrating geometric signals (sparse metric SLAM points), multi-view imagery, and multimodal cues (captions), and by specifically augmenting and training to reject degradations in input quality.

Methodology

ShapeR’s backbone is a multimodally conditioned rectified flow matching network operating in the latent space of a variational autoencoder (VAE) on 3D geometry. Input sequences are preprocessed using visual-inertial SLAM and 3D object detection to extract sparse metric point clouds, object-centric local image crops, camera poses, and vision-language captions per instance.

The denoising architecture is built atop the FLUX DiT paradigm, handling latent “VecSets” representing each shape. It conditions explicitly and jointly on:

• SLAM point tokens (encoded via a 3D ResNet),
• multi-view images (processed by a frozen DINOv2 vision backbone),
• camera poses (Plücker encoding),
• projection masks (2D convolutional encoding of object point projections into the images),
• and captions (T5 and CLIP embeddings).

The transformer fuses all modalities, robustly denoises Gaussian-noise-perturbed latents toward the true instance’s latent code, and eventually decodes each latent sequence into a signed-distance field (SDF), producing a mesh via marching cubes.

Figure 2: The model architecture combines posed images, SLAM points, captions, and projected 2D masks to robustly condition SDF mesh denoising and decoding.

Incorporating SLAM point clouds provides SLAM-derived, aggregated geometric priors for the object—critical for accurate metric scale, disambiguation under noise, and mitigating the often-weak coplanarity and segmentation boundaries in images.

Figure 3: Explicit conditioning on SLAM point clouds enhances reconstruction robustness and captures persistent shape cues unavailable in images alone.

Data Augmentation and Training

Training proceeds in two major stages. First, pretraining uses 600k+ synthetic object meshes with heavy compositional and modality-specific augmentations to simulate clutter, occlusion, masking errors, and degradations across all modalities—adversarially targeting the corruptions observed in natural captures. The second stage fine-tunes on synthetic scene crops (from Aria Synthetic Environment) that increase realism via occlusion, point cloud sparsity/noise, and inter-object interactions.

Figure 4: The training regimen: aggressive augmentation for pretraining (left) and scene-level, occlusion-rich crops for fine-tuning (right).

The inference pipeline is fully automatic. Per-object conditioning sets are constructed by associating SLAM points to views and generating point projections, then sequences are processed with no need for explicit image segmentation or manual intervention. Final outputs are guaranteed metric by direct rescaling with the detected point set’s real-space bounds.

Empirical Evaluation and Benchmarks

A key contribution is the ShapeR Evaluation Dataset: 178 annotated 3D object shapes across 7 scenes, captured casually with SLAM point clouds and ground-truth rigidly aligned meshes, filling a gap between studio control and wild-scene incompleteness.

Quantitative Metrics: Core geometry metrics (normalized Chamfer-$\ell_2$, normal consistency, F1 at 1% threshold) are compared across nine baselines, including EFM3D, FoundationStereo, LIRM, and DP-Recon. Foundation image-to-3D models—TripoSG, Direct3DS2, Hunyuan3D-2.0, Amodal3R—are also evaluated but always with high-quality mask and view selection for a fair upper-bound comparison.

Figure 5: Qualitative comparisons show ShapeR reconstructs fine-detailed, metrically accurate object shapes in the presence of occlusion and background clutter where other methods either fail or hallucinate interpolations.

User Study: Human evaluators consistently prefer ShapeR’s reconstructions ($>$80\% win rate over leading image-to-3D methods) given its consistency, metric accuracy, and completeness.

Figure 6: Foundation models can produce reasonable instance geometry when manually curated, but ShapeR operates end-to-end, robustly separating and reconstructing objects without any masking or view manual selection.

ShapeR also excels at reconstructing complex scene arrangements, automatically producing consistent metric layouts without interactive segmentation, surpassing MIDI3D and SceneGen in both utility and physical plausibility.

Figure 7: Scene-level assembly: ShapeR’s metric outputs remain consistent across objects and layouts, whereas image-to-scene approaches frequently fail in clutter.

Ablation Studies

Ablations reveal the criticality of each pipeline component. SLAM points significantly improve metric accuracy and robustness compared to image-only. On-the-fly modality augmentations (especially point cloud and image corruptions) are found vital for downstream generalization. Excluding 2-stage curriculum training, background compositing, or 2D point mask prompting all degrade performance, confirming the necessity of each for challenging, casual scenes.

Figure 8: Augmentation and conditioning studies demonstrate improvements in robustness, avoidance of overfitting to incomplete point observations, and reductions in object confusion.

Dataset Construction and Limitations

The ShapeR Evaluation Dataset is carefully constructed via both casual, in situ capture and controlled meshing to produce complete, metrically valid ground truth geometry, enabling fair quantitative and qualitative evaluation against prior work. Comparison with prior datasets explicitly demonstrates the added complexity—and the need for robust reconstruction—in wild-capture scenarios.

Figure 9: Comparison demonstrates that existing datasets either lack realism or object completeness, whereas ShapeR’s dataset fills this critical evaluation gap.

Figure 10: Distribution and examples of object annotations in the ShapeR benchmark; coverage spans categories, sizes, and occlusion scenarios.

Ground-truth meshes are created by displacing objects for high-quality imaging, performing image-to-3D modeling, then manually inserting, verifying, and refining meshes back in the original sequence, supporting reliable geometric and pose alignment.

Figure 11: Pseudo-ground truth mesh creation: isolate the object (left), reconstruct in ideal conditions (mid), then validate and realign in the original context (right).

Further experiments indicate the metric stability of ShapeR on other datasets, resilience to increasingly casual capture configurations, and flexibility to operate with monocular reconstructions using external metric estimators such as MapAnything.

Failure Cases and Limitations

ShapeR’s limitations are primarily: failures under severe image degradation or highly limited viewpoints (yielding rough/incomplete shapes), difficulty cleanly separating objects that are physically stacked/attached, and propagation of upstream 3D detection errors—if an object is not detected, it cannot be reconstructed.

Figure 12: Failure cases in limited-fidelity input, adjoining object contamination, and dependence on accurate detection.

Implications and Future Directions

ShapeR advances large-scale, multimodal conditional 3D generative modeling by demonstrating that robust, metric, and per-object geometry is achievable from highly imperfect, casual inputs. The architecture serves as a unified solution bridging generative per-object 3D modeling and metric scene reconstruction, with the robustness to act as the core of scalable scene understanding pipelines. Reliance on SLAM points and the absence of segmentation masks greatly reduce human interaction and annotation requirements.

Practically, ShapeR enables AR/VR and robotics pipelines to automatically produce trustworthy geometry in unconstrained environments, driving progress toward real-time, reliable 3D digital twins. Theoretically, ShapeR’s results suggest the value in tightly integrating geometric signals and vision-LLMs in 3D generation, as well as the power of adversarial augmentation and curriculum-based training regimens. Future work may address upstream detection bottlenecks, stronger instance separation, and extending to fully scene-wide interaction and relighting-ready assets.

Conclusion

ShapeR establishes a new state-of-the-art for automatic, metric, object-level 3D reconstruction in casual, real-world scenes, leveraging multimodal conditioning and aggressive, curriculum-driven training to overcome segmentation and input degradations. Through both the algorithm and the benchmark dataset, ShapeR provides a foundation for robust, large-scale, multimodal 3D generative modeling and sets an empirical standard for practical deployment in unconstrained environments (2601.11514).