Physics-Aware Scene Composition

Updated 18 March 2026

Physics-aware scene composition is a computational paradigm that synthesizes 3D scenes obeying constraints like non-penetration and stable support.
These methods integrate geometric registration with physics constraints using optimization, sampling, and LLM-guided feedback for realistic layouts.
Applications span simulation, planning, and photorealistic rendering, with metrics such as non-penetration scores ensuring physical plausibility.

Physics-aware scene composition encompasses a class of computational frameworks and algorithms designed to generate, reconstruct, or synthesize 3D scenes in which object arrangements are jointly consistent with both geometry and physical laws—such as non-penetration, contact, support, gravity, and sometimes more advanced notions like balance, adhesion, or lighting consistency. This paradigm emerges at the intersection of computer vision, graphics, machine learning, and physics, aiming to produce digital environments that are not only visually coherent but are also simulation-ready and capable of supporting downstream tasks such as planning, manipulation, and photorealistic rendering.

1. Formulations and Key Principles

Physics-aware scene composition mandates that solutions must satisfy constraints beyond geometric or appearance matching. Core principles include:

Geometric registration: Each object's estimated pose and shape must explain sensory data (e.g., RGB images, depth, or LiDAR). Typically encoded by registration losses, e.g., Chamfer or L₂ distances between point clouds and mesh surfaces (Yu et al., 8 Feb 2026).
Non-penetration: No two solid bodies interpenetrate; enforced via signed distance field (SDF) evaluations or collision loss terms (Yu et al., 8 Feb 2026, Shan et al., 17 Mar 2026, Yao et al., 18 Feb 2025).
Stable contact and support: Every object must rest on, be attached to, or be supported by another object or the environment (e.g., table, floor). This precludes "floating" arrangements and enforces realistic physical support graphs (Yao et al., 18 Feb 2025, Yu et al., 8 Feb 2026, Wang et al., 16 Feb 2026).
Static equilibrium (optional): Certain frameworks further maintain force and torque balance at contacts, considering friction and contact complementarity to ensure simulation stability (Huang et al., 23 Feb 2026).
Physical plausibility metrics: Quantitative proxies such as the Scene Plausibility Score (SPS) or Non-penetration Score (NPS) directly measure kinetic energy under simulation or the average SDF penetration depth, respectively (Yu et al., 8 Feb 2026).

In all cases, physics-aware scene composition is distinct from naive compositional synthesis by its explicit enforcement of physical constraints either during generation, through post-hoc optimization, or via simulation-in-the-loop.

2. Methods for Physics-Aware Scene Generation and Reconstruction

A wide variety of frameworks implement physics-aware scene composition, ranging from optimization-based pipelines to agentic, LLM-driven systems.

Optimization-Based Approaches

Joint Shape and Pose Optimization: Methods such as the shape-differentiable contact model jointly optimize object geometries and poses under physical constraints. The global decision vector—including pose, shape variables, and (when friction is modeled) tangential forces—minimizes an energy subject to collision-free, equilibrium-contact, and friction constraints. The constrained problem is solved using Augmented Lagrangian and structure-aware solvers exploiting sparsity (see (Huang et al., 23 Feb 2026)).
Physics-constrained Sampling: Rejection-sampling systems (e.g., Picasso) sequentially sample each object’s SIM(3) transformation, governed by a learnt contact scene graph. Samples are retained only if they fit the sensory data, satisfy non-penetration with parent objects and environment, and do not "float" (contact constraint) (Yu et al., 8 Feb 2026).
SDF-based Correction: Multi-object solutions refine independent asset generations via gradient-based optimization over object poses, penalizing SDF-overlap and minimally enforcing contact (as in the "constraint graph" of CAST (Yao et al., 18 Feb 2025) or SDF losses in Interact3D (Shan et al., 17 Mar 2026)).

Agentic and Feedback-Loop Frameworks

LLM-guided Layout with Physics Simulation: Systems such as PhyScensis orchestrate an LLM agent proposing object assets and spatial/physical predicates, and a solver powered by a physics engine optimizes and validates the resulting arrangement, iterating with feedback on stability, collision, and other metrics (Wang et al., 16 Feb 2026).
Graph-Driven Generation: LayoutDreamer and PhiP-G parse input text into scene or relation graphs, then instantiate physical and semantic energy terms over the graph's nodes and edges (e.g., enforcing gravity, anchor, placement, and rotation constraints), optimizing for both physical plausibility and semantic fidelity (Zhou et al., 4 Feb 2025, Li et al., 2 Feb 2025).
Multimodal agent collaboration: PhiP-G iterates between LLM-based extraction, visual supervision agents (for violation detection), and physical “pool/magnet” systems that enforce pairwise contact or separation (Li et al., 2 Feb 2025).

Realistic Lighting, Rendering, and Fusion

Relightable, physically-aware composition: ComGS introduces Surface Octahedral Probes (SOPs) to accelerate object-scene composition under realistic, relightable conditions—storing indirect lighting and occlusion caches for efficient PBR rendering with proper shadows and appearance (Gao et al., 9 Oct 2025).
Object-centric radiance models: OSF-based frameworks train a neural scattering function per object and use a physically-based volumetric path tracer for inter-object light transport, enabling scene-independent rendering, shadowing, and fidelity to radiative physics (Guo et al., 2020).

3. Physical Constraints and Optimization Formalism

Table: Core Types of Physical Constraints and Their Mathematical Encodings

Constraint Type	Mathematical Representation	Key References
Non-penetration	SDF-based: $\Phi_i(T_ib) \ge 0\ \forall b\in S_j$	(Yu et al., 8 Feb 2026, Yao et al., 18 Feb 2025)
Contact / Support closure	$\min_{b\in S_i, j\neq i}\|\Phi_j(T_i^{-1}b)\|\leq \delta$	(Yu et al., 8 Feb 2026, Yao et al., 18 Feb 2025)
Gravity, centroid, anchor, rotation	Energy terms penalizing $z$ -displacement, principal axis, etc.	(Zhou et al., 4 Feb 2025)
Static equilibrium and friction	$\sum f_k = 0$ , $\sum (p_k-p_{COM})\times f_k=0$ , friction cones	(Huang et al., 23 Feb 2026)
Lighting and shadow consistency	Monte Carlo integration, occlusion via SOPs, attention models	(Gao et al., 9 Oct 2025, Guo et al., 2020)

Optimization typically combines these terms via weighted sums, sometimes using alternating phases (e.g., physical → layout schedule in LayoutDreamer (Zhou et al., 4 Feb 2025)). Auto-differentiation, penalty and barrier methods, Woodbury/Schur-based solvers, and gradient-based minimization (Adam, Levenberg–Marquardt) are common.

4. Scene Graphs, Contact Graphs, and Semantic Input

Many pipelines employ explicit scene or contact graphs to encode high-level inter-object relationships and propagate corresponding constraints:

Directed scene graphs: Compositional scene generators extract scene graphs from text, mapping nodes to objects and edges to semantic/physical relations such as "on", "under", "lean", which become terms in the objective energy (Zhou et al., 4 Feb 2025, Li et al., 2 Feb 2025, Yao et al., 18 Feb 2025).
Contact scene graphs: In reconstruction, VLMs analyze sensor data or images to extract which objects are in contact, forming a DAG that governs the order of placement and constraint application (Picasso, (Yu et al., 8 Feb 2026)).
Predicates and control: Agent-based frameworks use predicates for 2D and 3D spatial relations (front-of, support, place-in), encoding fine-grained controllability over the physics-aware placement process (Wang et al., 16 Feb 2026, Zhou et al., 4 Feb 2025).

This graph-aware approach enables physical priors to propagate along semantically meaningful links, yielding more plausible and controllable scene layouts.

5. Evaluation Metrics and Empirical Advances

Empirical validation combines geometric, physical, and semantic assessments:

Quantitative physics metrics: SPS (Scene Plausibility Score; kinetic energy after simulation) and NPS (Non-penetration Score; mean SDF violation) are used to measure, respectively, the stability and interpenetration of inferred scenes. Lower values indicate physically correct reconstructions (Yu et al., 8 Feb 2026).
Visual and physics-based ablations: Removing physics-aware losses or modules dramatically increases artifact rates (floating, penetration) and reduces scene plausibility (Li et al., 12 Mar 2026, Li et al., 2 Feb 2025).
Rendering realism: Relightable methods such as LANe and OSF-based approaches achieve scene-level photometric consistency and capture shadows, indirect illumination, and texture fidelity under novel arrangements (Krishnan et al., 2023, Guo et al., 2020, Gao et al., 9 Oct 2025).

Notable improvements in layout plausibility, coverage, and physical validity are consistently reported when integrating physical knowledge—e.g., zero collision rate after optimization in MV-SAM3D, or >4.5/5 physical rationality ratings for PhiP-G (Li et al., 12 Mar 2026, Li et al., 2 Feb 2025).

6. Limitations, Extensions, and Outlook

Physics-aware scene composition, while a major step beyond pure geometric or semantic layout, faces several open limitations:

Rigid body focus: Most frameworks currently address only rigid, static object configurations. Articulated, deformable, or dynamically interacting scenes remain challenging (Li et al., 2 Feb 2025).
Approximate or soft physics: Many methods enforce “soft” constraints via penalty or energy minimization, lacking physically simulated force/torque optimization except in specialized settings (e.g., static equilibrium in (Huang et al., 23 Feb 2026)) or by brute-force simulation (PhyScensis (Wang et al., 16 Feb 2026)).
Lighting/material limitations: Current pipelines are often restricted to simple or locally estimated lighting models, with limited support for complex inter-reflections or material-specific transport (Gao et al., 9 Oct 2025, Krishnan et al., 2023).
Computational efficiency: Sophisticated physical models can introduce substantial computational overhead, although recent work on surface probes (ComGS) and sparse solvers (Picasso, real-to-sim) yields significant speedups (Gao et al., 9 Oct 2025, Huang et al., 23 Feb 2026, Yu et al., 8 Feb 2026).

Promising future directions include learned, differentiable physics engines for dynamic scenes, translation to articulated or soft materials, improved high-frequency lighting and material models, and fully integrated symbolic-to-physical-agent chains for closed-loop, feedback-driven scene synthesis (Li et al., 2 Feb 2025, Wang et al., 16 Feb 2026, Huang et al., 23 Feb 2026).