MV-SAM3D: Adaptive Multi-View Fusion

• The paper introduces a novel multi-diffusion paradigm that fuses per-view latent velocities for enhanced 3D asset generation.
• It combines attention entropy and visibility cues for adaptive weighting, ensuring robust integration of multi-view observations.
• The framework employs generation-time and post-generation physics-based optimization to achieve physically plausible scene layouts.

MV-SAM3D is a training-free, layout-aware 3D generation framework designed for adaptive multi-view fusion, integrating confidence-aware latent-space diffusion and physics-based optimization for scene-level 3D asset reconstruction and composition. Extending single-view SAM3D, MV-SAM3D fuses multi-view observations via a novel Multi-Diffusion paradigm and enforces physical plausibility through both generation-time and post-generation constraints. It achieves substantial improvements in reconstruction fidelity and physical correctness across synthetic and real-world scene benchmarks (Li et al., 12 Mar 2026).

1. Multi-Diffusion Fusion in Shared 3D Latent Space

MV-SAM3D generalizes SAM3D’s flow-matching diffusion from single- to multi-view conditioning. Let $x_t\in\mathbb{R}^D$ be the latent state at time $t$, with per-view conditioning $\{c_i\}_{i=1}^N$. Each view predicts a velocity $v_\theta(x_t, t, c_i)$, and the fused velocity is computed as:

$$\hat v(x_t,t)=\sum_{i=1}^N w_i\,v_\theta(x_t, t, c_i), \quad \sum_{i=1}^N w_i = 1.$$

This fused velocity $\hat v$ updates the latent state via the discretized ODE:

$x_{t+\Delta t} = x_t + \hat v(x_t,t)\,\Delta t$

The weights $w_i$ are derived adaptively for each latent point, enabling local adaptation based on per-view confidence.

Such confidence-weighted Multi-Diffusion enables MV-SAM3D to exploit complementary information across all input views, contrasting with feature-pooling, concatenation, or independently estimated per-view fusions found in other paradigms such as VPFusion (Mahmud et al., 2022) and MVAF-Net (Wang et al., 2020).

2. Adaptive Weighting: Attention-Entropy and Visibility Confidence

MV-SAM3D introduces two complementary mechanisms for adaptive per-view weighting.

Attention-Entropy Weighting (Implicit Confidence):

• For each view $i$ and latent point $l$, extract normalized cross-attention logits $\hat{\mathbf a}_{i,l}$ from a specified transformer layer.
• Compute normalized Shannon entropy:

$$H_i(l) = -\frac{1}{\log P} \sum_{p=1}^P \hat a_{i,l}^{(p)}\, \log(\hat a_{i,l}^{(p)}), \quad H_i(l)\in[0,1]$$

• Derive entropy-based confidence:

$$w_i^{\rm ent}(l) = \frac{\exp(-\alpha H_i(l))}{\sum_{j=1}^N \exp(-\alpha H_j(l))}$$

Visibility Weighting (Explicit Confidence):

• After Stage 1 structure generation, perform DDA-based 3D ray tracing for each view and latent point to estimate visibility $V_{i,l}\in\{0,1\}$.
• Compute:

$$w_i^{\rm vis}(l) = \frac{\exp(\beta V_{i,l})}{\sum_{j=1}^N \exp(\beta V_{j,l})}$$

Combined Confidence:

$$w_i(l) = (1-\gamma) w_i^{\rm ent}(l) + \gamma w_i^{\rm vis}(l)$$

This joint confidence scheme enables MV-SAM3D to set each view's contribution dynamically, robustly handling occlusions, view redundancy, and ambiguous observations. Ablation results confirm that combining entropy (geometry-stage) and visibility (texture-stage) outperforms naïve or single-mode fusions (Li et al., 12 Mar 2026).

3. Physics-Aware Scene Layout Optimization

To address physically implausible layouts such as object penetration and floating—frequent in independently estimated multi-object scenes—MV-SAM3D injects a two-stage physics-based optimization.

Layout Injection (Generation-Time Guidance):

• During diffusion, at scheduled steps, add a physics loss to the flow update:

$$\mathcal L_{\rm phys}^{(k)}(t) = \mathcal L_{\rm collision}(\tilde{\mathcal V}_t^{(k)}, \{\mathcal V^{(j)}\}_{j\neq k}) + \lambda_{\rm contact}\ \mathcal L_{\rm contact}(\tilde{\mathcal V}_t^{(k)}, \{\mathcal V^{(j)}\}_{j\neq k})$$

• The latent update becomes:

$$x_{t+\Delta t} = x_t + v_\theta(x_t,t,c)\,\Delta t - \eta\,\nabla_{x_t}(\sum_k\mathcal L_{\rm phys}^{(k)}(t))$$

• This steers trajectories away from collisions and toward plausible contacts during sampling.

Post-Generation Pose Refinement:

• After mesh extraction, optimize per-object similarity transforms $\theta_k$ with a total loss incorporating alignment (chamfer), collision, contact, and regularization:

$$\mathcal L_{\rm total} = \sum_k \left[\mathcal L_{\rm align}^{(k)} + \lambda_{\rm col}\sum_{j\neq k} \mathcal L_{\rm col}^{(j,k)} + \lambda_{\rm con}\sum_{j\neq k} \mathcal L_{\rm con}^{(j,k)} + \lambda_{\rm reg} \mathcal L_{\rm reg}\right]$$

This dual-stage process guarantees high-fidelity alignment and enforces non-penetration, with ablations demonstrating marked reductions in collision rates and increases in physical coverage and accuracy.

4. Implementation Details and Hyperparameters

• Stage 1: Sparse structure $\mathcal V\in\mathbb R^{K_1\times d_1}$
• Stage 2: Dense latent grid $\mathcal Z\in\mathbb R^{K_2\times d_2}$ (architecture matches SAM3D)
• Diffusion schedule: 25 steps, guidance activated at $T_s=15$, interval $\Delta_g=3$
• Entropy temp ($\alpha$) and occlusion penalty ($\beta$): 30 each. Weight blending $\gamma=0.5$
• Physics guidance: $\eta=0.1$, $\lambda_{\rm contact}=0.5$
• Refinement: 100 Adam iterations (lr=0.01), $\lambda_{\rm col}=200$, $\lambda_{\rm con}=50$, proximity threshold $\tau=0.05$

5. Experimental Results and Comparative Evaluation

Datasets:

• GSO-30: Rendered multi-view Google Scanned Objects, with ground-truth 3D, for single-object
• MV-SAM3D-Scenes: Real 2–8 object scenes, up to 15 views per scene, for multi-object layout

Metrics:

• Chamfer Distance (CD), PSNR, SSIM, LPIPS for single-object
• Per-object PSNR, Depth MAE/RMSE, Acc@5 cm, RelAcc@5%, Coverage, Collision rate for multi-object scenes

Key Results:

Method	Views	CD ($\downarrow$,$\times 10^{-3}$)	PSNR ($\uparrow$)	Acc@5 ($\uparrow$)
EscherNet	2	21.5	22.91
TRELLIS+M.D.	2	24.8	21.35
MV-SAM3D (Ours)	2	20.2	23.45
MV-SAM3D-Scenes (Full)	5		21.83	91.66

MV-SAM3D achieves the lowest CD and highest PSNR/SSIM/LPIPS on GSO-30 and dramatically improves Acc@5 cm, relative accuracy, and zero collision rate on multi-object scenes. Diminishing returns are observed beyond three input views.

Ablations show that entropy weighting in early stages recovers geometry, while augmenting with visibility in late stages resolves texture and symmetry ambiguity. Combined physics optimization (layout injection + post-refinement) eliminates all mesh interpenetration in evaluated cases.

6. Comparative Context and Methodological Innovations

MV-SAM3D’s confidence-adaptive fusion in 3D latent space markedly contrasts both pooling-based fusions (which lack local reliability) and transformer-based fusion architectures (Mahmud et al., 2022, Wang et al., 2020). Its explicit weighting based on both learned attention uncertainty and scene geometry provides fine-grained adaptability. The use of generation-time and post-generation physical layout optimization is distinctive among current multi-view 3D diffusion methods.

Compared to approaches such as MVAF-Net (Wang et al., 2020)—which employs pointwise attention for adaptive LiDAR/camera fusion but does not target explicit multi-object layout or physical constraints—or VPFusion (Mahmud et al., 2022)—which leverages transformer-based pairwise view association for 3D implicit volume fusion—MV-SAM3D uniquely integrates adaptive multi-view weighting with latent diffusion and explicit, bi-stage physics-aware pose guidance.

7. Significance, Limitations, and Future Directions

MV-SAM3D enables multi-view, multi-object, and layout-aware 3D asset generation without any retraining or model modification. It demonstrates robust handling of occlusions, view ambiguity, and object interactions, as supported by benchmarks. The framework’s reliance on inference-time weighting and optimization eliminates the need for further training data or calibration.

One limitation is the two-stage optimization’s computational overhead. Additionally, physics-based refinement relies on accurate initial mesh and pose estimation accuracy from the generative diffusion process.

Potential future directions include extending adaptive weighting mechanisms to integrate additional physical priors or semantic cues, and improving efficiency for larger-scale or dynamic scenes. Potential cross-disciplinary applications may emerge in SLAM, layout estimation, and physically plausible virtual world construction.

---

References:

MV-SAM3D (Li et al., 12 Mar 2026), VPFusion (Mahmud et al., 2022), MVAF-Net (Wang et al., 2020), Joint Multi-View SAM (Shvets et al., 2023)