Fast-SAM3D: 3Dfy Anything in Images but Faster

Abstract: SAM3D enables scalable, open-world 3D reconstruction from complex scenes, yet its deployment is hindered by prohibitive inference latency. In this work, we conduct the \textbf{first systematic investigation} into its inference dynamics, revealing that generic acceleration strategies are brittle in this context. We demonstrate that these failures stem from neglecting the pipeline's inherent multi-level \textbf{heterogeneity}: the kinematic distinctiveness between shape and layout, the intrinsic sparsity of texture refinement, and the spectral variance across geometries. To address this, we present \textbf{Fast-SAM3D}, a training-free framework that dynamically aligns computation with instantaneous generation complexity. Our approach integrates three heterogeneity-aware mechanisms: (1) \textit{Modality-Aware Step Caching} to decouple structural evolution from sensitive layout updates; (2) \textit{Joint Spatiotemporal Token Carving} to concentrate refinement on high-entropy regions; and (3) \textit{Spectral-Aware Token Aggregation} to adapt decoding resolution. Extensive experiments demonstrate that Fast-SAM3D delivers up to \textbf{2.67$\times$} end-to-end speedup with negligible fidelity loss, establishing a new Pareto frontier for efficient single-view 3D generation. Our code is released in https://github.com/wlfeng0509/Fast-SAM3D.

• The paper introduces a training-free acceleration framework that adaptively allocates computation per stage to overcome heterogeneity in 3D reconstruction.
• It employs three specialized modules—modality-aware caching, spatiotemporal token carving, and spectral-aware aggregation—to optimize performance and maintain geometric fidelity.
• Quantitative results demonstrate up to 2.67× speedup with improved F-Score and vIoU, validating the effectiveness of heterogeneity-aware design over generic acceleration.

Fast-SAM3D: Adaptive Inference Acceleration for Open-World Single-View 3D Reconstruction

Overview and Motivation

Fast-SAM3D (2602.05293) addresses a key scaling bottleneck in single-view 3D reconstruction with diffusion models, focusing explicitly on the state-of-the-art open-world pipeline SAM3D. While SAM3D demonstrates strong open-domain generalization and high-fidelity geometry via mask-conditioned diffusion and mesh decoding, its computational costs—dominated by iterative denoising and dense volumetric decoding—render it impractical for latency-sensitive applications. Prior generic acceleration strategies—uniform step skipping, token pruning, and instance-agnostic grid downsampling—result in catastrophic failures due to fundamental heterogeneity within the pipeline, such as contrasting evolutionary dynamics between shape and layout, spatial sparsity in texture refinement, and spectral variability across objects.

Fast-SAM3D introduces a holistic, training-free acceleration framework that identifies and adapts to these sources of heterogeneity at each pipeline stage, establishing a new Pareto frontier in quality-efficiency tradeoff for 3D generation from images. The key insight is stage- and instance-specific computation allocation, systematically matched to local complexity and redundancy.

Figure 1: Overview of the Fast-SAM3D pipeline with stage-wise heterogeneity-aware modules: Modality-Aware Step Caching (1), Joint Spatiotemporal Token Carving (2), and Spectral-Aware Token Aggregation (3).

Profiling SAM3D: Heterogeneity and Bottlenecks

Comprehensive profiling reveals that SAM3D's inference latency arises from three tightly coupled sources:

1. Dual-Stage Diffusion: Both structure and texture generators are dominated by the need for full denoising trajectories per token.
2. Sparse Mesh Decoding: The transition from sparse latent to explicit meshes involves combinatorial overhead as token counts scale with object complexity.
3. Token Modality Divergence: Structural and layout information traverse fundamentally different denoising trajectories.

Figure 2: SAM3D pipeline analysis shows that cost is driven by iterative denoising and combinatorial mesh decoding.

Straightforward application of uniform step skipping or generic token pruning is brittle: adverse effects include pose drift (from unstable layout inference), geometric collapse (due to over-pruning critical regions), or severe loss of surface detail (through careless downsampling).

Methodology: Three Heterogeneity-Aware Modules

Fast-SAM3D introduces three lightweight, inference-time modules—each tightly coupled to a specific pipeline heterogeneity—without any retraining or additional supervision.

Modality-Aware Step Caching

Within the sparse structure generator, two disjoint token trajectories are handled:

• Shape Tokens evolve smoothly and linearly, enabling first-order finite-difference extrapolation for aggressive step caching without incurring drift.
• Layout Tokens exhibit high-frequency volatility and global impact (pose), demanding an 'anchor-corrected' update: linear extrapolation with stabilization via a momentum-weighted anchor from the last full model evaluation. This mechanism mitigates the compounding error observed in naive extrapolation.

  Figure 3: Modality heterogeneity—shape tokens evolve smoothly, layout tokens are erratic; motivating bi-modal caching.

Joint Spatiotemporal Token Carving

Refinement in the SLaT generator is non-uniform—most tokens observe negligible change, while only high-entropy regions (edges, seams, thin structures) require persistent refinement. Fast-SAM3D quantifies temporal update magnitude and abruptness, fusing this with frequency-based spatial complexity to generate a unified per-token saliency score.

• At each step, only top-$K$ active tokens (by saliency) are updated; the remainder are cached.
• Dynamic, curvature-aware step skipping further reduces redundant denoising iterations during quasi-linear evolution. An error-bounded switching mechanism maintains quality by triggering refreshes when the approximation error accumulates beyond a threshold.

Figure 4: Intrinsic refinement sparsity—major updates are spatially sparse and temporally non-uniform; saliency map precisely targets these regions.

Figure 5: The predicted unified saliency map closely aligns with the ground-truth change map, ensuring critical region identification.

Spectral-Aware Token Aggregation

Mesh decoding is accelerated via instance-adaptive grid aggregation. The system computes a composite high-frequency energy ratio (HFER) from both 2D masks and 3D coarse voxels, capturing geometric complexity. Token aggregation strength (quantization factor and pooling) is adaptively selected: simple objects undergo aggressive reduction, while complex geometries are preserved at full resolution, preventing aliasing of critical detail.

Figure 6: Instance-level spectral heterogeneity—simple objects permit high aggregation, complex objects require denser grids.

Figure 7: Fourier analysis—simple shapes have fast spectral decay, complex objects retain substantial high-frequency content.

Quantitative and Qualitative Results

Fast-SAM3D achieves 2.67× object-level and 2.01× scene-level speedup (over original SAM3D) with negligible or positive impact on geometric fidelity (F-Score: 92.59 vs. 92.34; vIoU: 0.552 vs. 0.543). Notably, random or uniform token-level acceleration protocols result in catastrophic breakdowns (e.g., 3D-IoU dropping to 0.094), underscoring the necessity of heterogeneity-aware design. In most benchmarks—including perceptual and geometric metrics—saliency-aware carving even denoises spurious high-frequency noise found in the full baseline, improving surface regularity and semantic consistency.

Figure 8: Qualitative comparison—Fast-SAM3D closely matches the original SAM3D, while generic baselines lose critical structure or semantic attributes.

Ablations and Component Analysis

Ablation studies confirm that each acceleration module offers orthogonal benefits, with mesh aggregation providing the highest single-stage latency reduction. Modality-aware caching is critical for pose stability (preventing layout drift), while spatiotemporal carving is essential for both efficiency and fidelity (removing redundant low-saliency regions).

Parameter sensitivity analyses (e.g., carving thresholds, cache stride, layout momentum) reveal that moderate settings allow for substantial acceleration without sacrificing high-level metrics; over-aggressive reductions quickly degrade quality.

Figure 9: Visualization of shape vs. layout token trajectories during denoising, motivating differential treatment.

Implications and Future Directions

Fast-SAM3D demonstrates that system-level, pipeline-aware acceleration—informed by a detailed understanding of token and data domain heterogeneity—outperforms isolated, instance-agnostic speedup techniques, especially for high-dimensional diffusion-based 3D generation. The framework is fully training-free, universally applicable to existing SAM3D checkpoints.

Practically, this work unlocks the deployment of high-quality, real-time 3D asset creation in applications ranging from AR/VR to robotics and content creation. Theoretically, it establishes a foundation for broader exploitation of structural and spectral sparsity in diffusion models and generative pipelines, especially in modalities beyond images (e.g., video, audio, multi-modal world models).

Potential future work includes:

• Extending this heterogeneity-aware paradigm to multi-view and scene-level modeling.
• Compiler- or hardware-aligned implementations leveraging the identified sparsity at inference.
• Integrating learning-based modules to further optimize instance complexity prediction and resource allocation dynamically.

Conclusion

Fast-SAM3D systematically resolves the prohibitive inference cost in open-world 3D generation by modeling and exploiting pipeline- and data-level heterogeneity. It demonstrates that fine-grained adaptive allocation of computation, specifically aligned to the evolutionary dynamics and spectral properties of each pipeline stage, achieves strong empirical speedups (up to 2.67×) while preserving or even enhancing geometric fidelity relative to the unoptimized baseline. This approach provides a robust blueprint for the acceleration of complex, large-scale generative models in structured domains (2602.05293).