SAM3D-Phys: Recovering Simulation-Ready Objects

• SAM3D-Phys is a framework that recovers complete object geometry from partially reconstructed real-world scenes for accurate physics-based simulation.
• It integrates multi-view scene reconstruction with a fixed generative 3D prior and enforces scene consistency through coupled pose and physics constraints.
• The pipeline refines object appearance using mask-guided distillation and converts outputs into simulation-ready assets via mesh reconstruction and convex decomposition.

Searching arXiv for the cited paper and closely related work for context. SAM3D-Phys is a framework for recovering complete, simulatable object geometry from reconstructed real-world scenes so that objects embedded in a scene can participate in physics-based interaction. It addresses a specific failure mode of modern multi-view reconstruction pipelines: visually accurate environments often contain objects that are incomplete because of occlusions and limited observations, and such incompleteness makes them unsuitable for downstream simulation. The framework integrates multi-view scene reconstruction with the generative 3D priors of SAM3D, then restores scene-consistent object state through physics-constrained spatial optimization and mask-guided appearance distillation, producing clean object representations suitable for simultaneous and physically consistent interactive simulation of multiple objects within a reconstructed scene (Dong et al., 28 May 2026).

1. Problem formulation and scope

SAM3D-Phys is designed for the setting in which a reconstructed scene already exists, but the objects inside that scene are not directly usable for simulation because their geometry is only partially observed. The central problem is therefore not generic novel-view synthesis or generic 3D asset generation; it is the recovery of complete object geometry and scene-consistent state from a reconstructed real-world scene, with the explicit goal of enabling interaction in a physics engine (Dong et al., 28 May 2026).

The framework assumes a calibrated set of RGB images $\{I_j\}_{j=1}^N$ with known camera intrinsics $K_j$ and extrinsics $(R_j,t_j)$. From these images it reconstructs a scene representation, extracts partial observations of selected objects, completes those objects with a frozen SAM3D prior, refines their pose relative to the reconstructed scene under physical constraints, and distills appearance back into a 3D Gaussian representation (Dong et al., 28 May 2026).

A key conceptual distinction is that SAM3D-Phys treats geometric completion, spatial consistency, and simulability as coupled requirements. Partial reconstruction alone is insufficient because physics engines require watertight or otherwise collision-ready geometry; generative completion alone is insufficient because the generated object must still be restored to its original location and appearance in the scene; and appearance recovery alone is insufficient if the object remains floating or interpenetrating. This suggests that the framework’s defining contribution is the integration of these requirements into a single sequential pipeline rather than any isolated module.

2. Four-stage pipeline

The pipeline consists of four sequential modules: Multi-View Scene Reconstruction, Generative 3D Prior Inference (SAM3D), Object–Scene Alignment, and Mask-Guided Appearance Distillation (Dong et al., 28 May 2026). The workflow can be summarized as follows.

Stage	Input	Output
Multi-View Scene Reconstruction	Calibrated RGB images	3D Gaussian scene model, rendered views, depth maps
Generative 3D Prior Inference	Partial object geometry, images, masks, point-maps	Complete object surface or occupancy field
Object–Scene Alignment	Generated object, scene masks, initial pose	Scene-consistent pose under rendering and physics constraints
Mask-Guided Appearance Distillation	Fixed geometry and pose, original images	Refined object texture in 3DGS form

In the first stage, the method adopts PGSR (Planar-based Gaussian Splatting) to build a 3D Gaussian scene model

$$\mathcal{S} = \bigl\{G_i=(x_i,\Sigma_i,C_i,\alpha_i,s_i)\bigr\}_{i=1}^M,$$

where $x_i\in\mathbb R^3$ is the Gaussian center, $\Sigma_i$ its covariance, $C_i$ spherical-harmonic color coefficients, $\alpha_i$ opacity, and $s_i$ a segmentation affinity feature (Dong et al., 28 May 2026). Novel views and depth maps $D_j$ are rendered by ray-Gaussian intersection. User-selected objects are then removed from the Gaussian scene; the resulting image holes are inpainted with LaMa; and the background model is fine-tuned to yield a clean static scene (Dong et al., 28 May 2026).

In the second stage, the method collects the partial geometry of each object of interest from $K_j$0 by selecting Gaussians whose segmentation-affinity $K_j$1 indicates membership. These are converted into a partial point-cloud or surfel set

$K_j$2

Per-view point-maps $K_j$3 are computed by back-projection of depth $K_j$4:

$K_j$5

The tuple $K_j$6, together with $K_j$7, is passed into the frozen SAM3D feed-forward network $K_j$8 to predict a complete object surface or occupancy field,

$K_j$9

No training is performed; SAM3D is kept frozen (Dong et al., 28 May 2026).

The third stage restores the generated object to its original scene location through coupled pose refinements. The fourth stage refines texture fidelity by optimizing local 3DGS appearance parameters against the observed images (Dong et al., 28 May 2026).

3. Partial-to-complete object inference

The generative component of SAM3D-Phys uses SAM3D as a fixed prior for completion from partial observations. The partial object can be represented either as a point-cloud

$(R_j,t_j)$0

or as a voxel occupancy field

$(R_j,t_j)$1

The frozen network $(R_j,t_j)$2 implements a conditional diffusion or occupancy model and formally defines a distribution

$(R_j,t_j)$3

from which it samples or directly outputs a complete mesh or occupancy grid (Dong et al., 28 May 2026).

The technical role of the image-side inputs is explicit. Each view contributes the RGB image $(R_j,t_j)$4, the object mask $(R_j,t_j)$5, and the point-map $(R_j,t_j)$6. This pairing allows SAM3D-Phys to use scene-derived metric geometry from depth back-projection together with 2D appearance and segmentation cues, while still relying on SAM3D’s internal priors to infer unobserved regions (Dong et al., 28 May 2026).

The paper states that SAM3D’s internal losses and diffusion-based priors guarantee plausible completion, and it summarizes the internal objective schematically as

$(R_j,t_j)$7

Because SAM3D remains frozen, the framework’s completion stage is not a re-training procedure over the scene-specific data. A common misunderstanding would be to equate “training-free” with the absence of optimization anywhere in the system. In SAM3D-Phys, “training-free” applies to the generative prior itself: the network weights of SAM3D are not fine-tuned, even though later stages optimize pose and local appearance parameters (Dong et al., 28 May 2026).

4. Object–scene alignment and physics-constrained refinement

After completion, SAM3D-Phys restores each generated object to its original pose in the scene by two coupled refinements: render-and-compare pose refinement and physics-constrained spatial refinement (Dong et al., 28 May 2026).

In render-and-compare pose refinement, the object pose is parameterized by translation $(R_j,t_j)$8 and rotation $(R_j,t_j)$9. At iteration $$\mathcal{S} = \bigl\{G_i=(x_i,\Sigma_i,C_i,\alpha_i,s_i)\bigr\}_{i=1}^M,$$0, the current estimate is $$\mathcal{S} = \bigl\{G_i=(x_i,\Sigma_i,C_i,\alpha_i,s_i)\bigr\}_{i=1}^M,$$1. The object is rendered into each view, and mask-based similarity to the ground-truth mask $$\mathcal{S} = \bigl\{G_i=(x_i,\Sigma_i,C_i,\alpha_i,s_i)\bigr\}_{i=1}^M,$$2 is measured with MS-SSIM or SSIM:

$$\mathcal{S} = \bigl\{G_i=(x_i,\Sigma_i,C_i,\alpha_i,s_i)\bigr\}_{i=1}^M,$$3

Translation and rotation are updated by alternating minimizations, similar to iG-6DoF: \begin{align*} t^{k+1}&=\arg\min_{t}\;\mathcal{L}_\text{render}(R^k,\,t),\ R^{k+1}&=\arg\min_{R}\;\mathcal{L}_\text{render}(R,\,t^{k+1}). \end{align*} This stage aligns the completed object with the image evidence but does not by itself prevent physically implausible placement (Dong et al., 28 May 2026).

To prevent floating and inter-penetration, the framework then constructs a small relation graph of object–scene and object–object edges and adds two differentiable penalty terms. For an object’s bottom-face sample set $$\mathcal{S} = \bigl\{G_i=(x_i,\Sigma_i,C_i,\alpha_i,s_i)\bigr\}_{i=1}^M,$$4 and a ground plane $$\mathcal{S} = \bigl\{G_i=(x_i,\Sigma_i,C_i,\alpha_i,s_i)\bigr\}_{i=1}^M,$$5, the object–scene ground-contact loss is

$$\mathcal{S} = \bigl\{G_i=(x_i,\Sigma_i,C_i,\alpha_i,s_i)\bigr\}_{i=1}^M,$$6

For each interacting pair $$\mathcal{S} = \bigl\{G_i=(x_i,\Sigma_i,C_i,\alpha_i,s_i)\bigr\}_{i=1}^M,$$7, the object–object separation loss is \begin{align*} d_{\min}(\mathcal O_a,\mathcal O_b) &=\min_{\mathbf x\in\mathcal O_a,\;\mathbf y\in\mathcal O_b}|\mathbf x-\mathbf y|2,\ \mathcal{L}{oo} &=\sum_{(a,b)}\Bigl[\max\bigl(0,m - d_{\min}(\mathcal O_a,\mathcal O_b)\bigr)\Bigr]^2. \end{align*} The full pose-refinement objective becomes

$$\mathcal{S} = \bigl\{G_i=(x_i,\Sigma_i,C_i,\alpha_i,s_i)\bigr\}_{i=1}^M,$$8

and it is solved by a few steps of gradient descent over $$\mathcal{S} = \bigl\{G_i=(x_i,\Sigma_i,C_i,\alpha_i,s_i)\bigr\}_{i=1}^M,$$9 (Dong et al., 28 May 2026).

The paper also gives the compact gradient update form

$x_i\in\mathbb R^3$0

The significance of this stage is methodological: scene consistency is not enforced only through image-space alignment, but through a joint objective that includes ground-contact and non-penetration terms. This suggests that pose restoration in SAM3D-Phys is explicitly physics-aware rather than merely photometrically or mask consistent.

5. Mask-guided appearance distillation

Once object geometry and pose are fixed, SAM3D-Phys refines texture within the 3DGS representation rather than retraining the generative model (Dong et al., 28 May 2026). For each view $x_i\in\mathbb R^3$1, let $x_i\in\mathbb R^3$2 be the rendered color image of the object and $x_i\in\mathbb R^3$3 the original image. The overlapping mask is defined as $x_i\in\mathbb R^3$4 (rendered mask). On this overlap region, the framework applies a patch-level VGG feature-distance loss:

$x_i\in\mathbb R^3$5

where $x_i\in\mathbb R^3$6 is the $x_i\in\mathbb R^3$7-th layer of a pretrained VGG network (Dong et al., 28 May 2026).

An optional L2 color-difference term can also be added:

$x_i\in\mathbb R^3$8

The mathematical summary further includes

$x_i\in\mathbb R^3$9

$\Sigma_i$0

Only the 3DGS color coefficients $\Sigma_i$1 and/or opacities $\Sigma_i$2 in the object’s local Gaussians are optimized, preserving the training-free nature of the method (Dong et al., 28 May 2026).

This module is important because completion and alignment can restore geometry and pose while still leaving a mismatch between the generated object’s appearance and the original scene observations. The mask-guided distillation step addresses that mismatch locally, restricted to regions where rendered and observed object support overlap. A plausible implication is that the appearance refinement is intentionally conservative: it updates scene-embedded object Gaussians without altering the frozen SAM3D prior itself.

6. Conversion to simulation-ready assets

SAM3D-Phys does not stop at geometric completion or rendering consistency; it specifies how recovered objects are prepared for simulation (Dong et al., 28 May 2026). The completed surfel or occupancy output $\Sigma_i$3 is converted into a watertight triangular mesh via Poisson surface reconstruction. The mesh is then decomposed into convex sub-hulls $\Sigma_i$4, for example using VHACD, and each hull becomes a collision primitive in the physics engine (Dong et al., 28 May 2026).

For each object $\Sigma_i$5, a material density $\Sigma_i$6 is assigned, with the example value 1000 kg/m³ for a generic solid. Mass is computed as

$\Sigma_i$7

The inertia tensor is approximated by integrating over the convex hull(s), or via standard analytic formulas for convex polyhedra:

$\Sigma_i$8

These steps convert the recovered assets into the quantities typically required by rigid-body or continuum simulators (Dong et al., 28 May 2026).

The framework supports both rigid-body and continuum simulation. For continuum dynamics it provides a Material Point Method formulation. Given per-particle state $\Sigma_i$9 on a regular grid, the per-time-step simulation consists of Particle-to-Grid (P2G), Grid Update, and Grid-to-Particle (G2P) phases. In the P2G phase, particle mass and momentum are transferred to neighboring grid nodes using weights $C_i$0. In the grid update,

$C_i$1

In the G2P phase,

$C_i$2

$C_i$3

$C_i$4

$C_i$5

Here $C_i$6 is the stress computed from $C_i$7, $C_i$8 is the velocity-gradient, and $C_i$9 gravity (Dong et al., 28 May 2026).

The paper explicitly notes that an alternative rigid-body loop would perform collision detection, impulse resolution, velocity integration, and transform update. This places SAM3D-Phys at the interface between reconstruction/generation and simulation systems rather than solely within one of those domains.

7. Position within SAM3D-derived research

SAM3D-Phys belongs to a line of work built around SAM3D as a generative 3D prior, but its target problem differs from work that focuses on layout-aware scene generation. In SAM3D-Phys, the objective is to recover complete, scene-consistent objects from reconstructed real-world scenes so that they can be simulated (Dong et al., 28 May 2026). By contrast, MV-SAM3D extends layout-aware 3D generation with multi-view consistency and physical plausibility by formulating multi-view fusion as a Multi-Diffusion process in 3D latent space and adding physics-aware optimization for multi-object composition (Li et al., 12 Mar 2026).

This comparison clarifies a potential misconception. SAM3D-Phys is not described as a multi-view latent fusion method; its multi-view component enters through scene reconstruction, point-map computation, render-and-compare pose refinement, and appearance distillation. Likewise, its physics module is not the same as the collision/contact guidance used during generation in MV-SAM3D. Instead, SAM3D-Phys applies physics-constrained spatial optimization after object completion to restore scene-consistent object states (Dong et al., 28 May 2026, Li et al., 12 Mar 2026).

The full algorithmic summary in SAM3D-Phys makes this system boundary explicit. The input is multi-view images $\alpha_i$0, masks $\alpha_i$1, and camera poses; the output is a simulation-ready background plus a set of dynamic objects. The pipeline trains 3DGS until novel-view and depth errors converge, removes user-selected Gaussians and fine-tunes the background, runs SAM3D on each object’s partial geometry and point-maps, optimizes $\alpha_i$2 using render-and-compare with added $\alpha_i$3 and $\alpha_i$4, optimizes color and opacity using $\alpha_i$5 and $\alpha_i$6, reconstructs meshes, computes mass and inertia, and then steps simulation by MPM or a rigid-body loop (Dong et al., 28 May 2026).

Taken together, these design choices define SAM3D-Phys as a reconstruction-to-simulation framework centered on frozen generative completion, differentiable spatial restoration, and physics-engine integration. A plausible implication is that its main significance lies in bridging visually reconstructed real-world scenes and interactive physical simulation without retraining the underlying generative prior.