Picasso: Holistic Scene Reconstruction with Physics-Constrained Sampling

Abstract: In the presence of occlusions and measurement noise, geometrically accurate scene reconstructions -- which fit the sensor data -- can still be physically incorrect. For instance, when estimating the poses and shapes of objects in the scene and importing the resulting estimates into a simulator, small errors might translate to implausible configurations including object interpenetration or unstable equilibrium. This makes it difficult to predict the dynamic behavior of the scene using a digital twin, an important step in simulation-based planning and control of contact-rich behaviors. In this paper, we posit that object pose and shape estimation requires reasoning holistically over the scene (instead of reasoning about each object in isolation), accounting for object interactions and physical plausibility. Towards this goal, our first contribution is Picasso, a physics-constrained reconstruction pipeline that builds multi-object scene reconstructions by considering geometry, non-penetration, and physics. Picasso relies on a fast rejection sampling method that reasons over multi-object interactions, leveraging an inferred object contact graph to guide samples. Second, we propose the Picasso dataset, a collection of 10 contact-rich real-world scenes with ground truth annotations, as well as a metric to quantify physical plausibility, which we open-source as part of our benchmark. Finally, we provide an extensive evaluation of Picasso on our newly introduced dataset and on the YCB-V dataset, and show it largely outperforms the state of the art while providing reconstructions that are both physically plausible and more aligned with human intuition.

• The paper introduces Picasso, a model-based post-processor that uses contact graphs, robust Chamfer matching, and hard non-penetration and contact constraints to refine independently estimated object poses.
• Picasso improves SAM3D mean ADD-S from 21.68 mm to 10.56 mm on YCB-V and reduces the Scene Plausibility Score from 5.84 to 1.85 while operating in about 0.35 seconds.
• The method’s geometric gains primarily come from sampling-based refinement, while physics constraints reduce floating and interpenetrating objects, though performance remains limited by DAG contact approximations and inaccurate shape estimates.

Motivation and problem statement

The paper addresses a gap between geometrically accurate and physically plausible scene reconstruction. Object pose and shape estimators typically process each object in isolation; when the resulting estimates are imported into a simulator as a digital twin, small errors produce interpenetration, floating objects, or unstable equilibria, undermining simulation-based planning for contact-rich manipulation. The authors argue that pose and shape estimation should reason holistically over the scene, accounting for object interactions and static equilibrium, and they position Picasso as a model-based post-processor that retrofits existing estimators such as SAM3D (2602.08058).

Method: physics-constrained rejection sampling over a contact scene graph

Picasso formulates scene reconstruction as a maximum likelihood estimation problem over per-object similarity transforms $T_i = (s_i, R_i, t_i)$, with a Chamfer-based measurement likelihood and four analytical constraints: inter-object non-penetration, object-environment non-penetration, object-free-space consistency, and a contact constraint requiring every object to touch at least one other body (no floating objects). Rather than optimizing a physics loss, the authors treat physics as hard constraints and enforce them via rejection sampling. This design choice is deliberate: penalty-based formulations can converge to physically plausible but geometrically incorrect local minima, and differentiable-simulator supervision is brittle due to modeling inaccuracies and numerical noise. Rejection sampling avoids reshaping the objective and encourages global exploration of the $\SIM(3)$ state space.

The key tractability mechanism is the contact scene graph (CSG). The CSG topology—which objects are in contact—is inferred from a vision-LLM rather than from raw depth adjacency, because depth-based contact inference is sensitive to noise and occlusion. The undirected, possibly cyclic CSG is then approximated as a DAG via breadth-first search rooted at the environment node, enabling a greedy rollout in which each object's subproblem is solved conditioned on its parents. The paper concedes this is an approximation: greedy rollout does not guarantee recovery of the global MLE solution, though it performs well empirically, and inference calls are exactly $N$. Each decoupled subproblem is solved with a GPU-parallelized coarse-to-fine sampler over scale and $(3)$ pose, with a Geman-McClure robustified Chamfer objective; among the top $B=16$ candidates by Chamfer loss, the first constraint-satisfying pose is accepted, with fallback to the best-scoring pose if none satisfy the constraints. Per-object solve times are under one second, making the approach practical as an inference-time corrector.

The Picasso dataset and scene plausibility score

The paper contributes the Picasso dataset: 10 contact-rich real-world static scenes (2–8 objects each, 10 annotated images per scene) captured with an iPad LiDAR at $1920\times1440$ resolution, with SAM2-assisted, human-verified masks, Blender-refined object scans, and 6D poses from PnP on annotated 2D–3D keypoint correspondences. Scenes cover point, line, and flush contact types (e.g., mugs on a rack, a spatula on a pan, stacked Jenga blocks). The companion Scene Plausibility Score (SPS) builds a digital twin in PyBullet, runs a short rollout ($T=20$ steps at $1/240$ s), and measures the kinetic energy of the system under canonical mass and inertia, thresholded at 10 $\text{kg}\cdot\text{m}^2/\text{s}^2$ to absorb simulator modeling error. SPS is simulator-agnostic by design, though its reliance on a rollout makes it dependent on the fidelity of the chosen physics engine.

Experimental results

On YCB-V with CRISP, the strongest claim in the paper is that Picasso closes the sim-to-real gap: CRISP-Syn+Picasso (mean ADD-S 0.008 m, ADD-S AUC@3cm 0.77) surpasses both CRISP-Syn alone (0.015, 0.55) and CRISP-Real (0.009, 0.75), and also outperforms gradient-descent physics-loss refinement (which yields no improvement over CRISP-Syn) and Generalized ICP correction (0.012, 0.67). On physics metrics, CRISP-Syn+Picasso achieves SPS 8.71 and NPS 3.99 versus 9.36/6.38 for CRISP-Syn, with observable correctness improving from 15% to 55%, at a runtime of about 0.35 s.

With SAM3D as the base estimator, the gains are larger. Across 12 YCB-V trajectories, mean ADD-S drops from 21.68 mm to 10.56 mm with Picasso, SPS from 5.84 to 1.85, and NPS from 9.29 to 2.79. On the Picasso dataset, SAM3D+Picasso improves pose accuracy by 35.34% (mean ADD-S 11.71 mm to 7.57 mm), NPS by 28.56% (5.23 to 3.74), and SPS by 10.21% (8.50 to 7.63) relative to SAM3D. Notably, the ablation without physics constraints achieves comparable geometric accuracy (7.55 mm), indicating that most of the geometric gain comes from the sampling-based pose refinement itself, while the physics constraints primarily deliver the plausibility improvements; the paper acknowledges the physics benefit over the unconstrained sampler is modest because constraints are checked only on a small buffer of top candidates. The paper also reports failure cases—holder_bowl, pan_spatula_steak_broccoli, rack_cup_bowl_2—where SAM3D alone attains better physics scores, attributed to inaccurate shape estimates and severe depth occlusions altering the optimization landscape. A Mechanical Turk user study indicates human plausibility rankings align with SPS and favor Picasso reconstructions.

Setting	Metric	Baseline	+ Picasso
CRISP-Syn, YCB-V	ADD-S mean (m)	0.015	0.008
SAM3D, YCB-V	ADD-S mean (mm)	21.68	10.56
SAM3D, Picasso dataset	ADD-S mean (mm)	11.71	7.57
SAM3D, Picasso dataset	NPS	5.23	3.74
SAM3D, Picasso dataset	SPS	8.50	7.63

Limitations and open questions

The paper identifies several constraints on its results. The DAG approximation of the CSG discards cyclic contact structures (such as Jenga rings), so mutual interactions between upstream and downstream nodes are not modeled; whether this can be relaxed without sacrificing the linear-in-$N$ inference cost is left open. Physics constraints are applied only to the top $\SIM(3)$0 Chamfer-ranked candidates, which limits their corrective power and explains the modest physics gains over unconstrained sampling. Shape estimates are taken as given from the base network, so errors in $\SIM(3)$1 propagate into the constrained pose solve, as the reported failure cases show. The environment is assumed to be a planar surface fit by RANSAC, and scenes are static, restricting the constraint set to holonomic configuration-space conditions rather than full equilibrium analysis. The SPS metric depends on simulator fidelity and canonical mass/inertia assumptions, which the authors mitigate only by thresholding.

Conclusion

Picasso demonstrates that physics can be imposed as hard constraints within a sampling-based, scene-level pose corrector, yielding reconstructions that are simultaneously more geometrically accurate and more physically plausible than state-of-the-art estimators, at sub-second per-object cost. The combination of the open-sourced contact-rich dataset, the SPS metric, and the demonstrated sim-to-real gap closure provides a concrete evaluation framework for physics-aware reconstruction, while the DAG approximation, candidate-buffer constraint checking, and reliance on upstream shape estimates remain the principal open technical questions.