UniPixie: Unified and Probabilistic 3D Physics Learning via Flow Matching

Abstract: Existing feed-forward networks excel at predicting a single set of physical properties from visual appearance, but this point-estimate paradigm fundamentally fails to capture the real world's inherent physical ambiguity. We address this by reframing physics prediction as a task of learning a controllable, continuous distribution of material properties. We introduce UNIPIXIE, a framework trained to predict a continuous and parameterized path of physically plausible material properties from a single visual input. By learning a direct mapping along an object's softest-to-stiffest spectrum on our PIXIEMULTIVERSE dataset, UNIPIXIE allows for controllable generation of diverse, physically valid material fields via a single intuitive parameter. Crucially, UNIPIXIE introduces a novel unified architecture to produce simulation-ready parameters for diverse physics solvers, including continuum-based Material Point Method (MPM), reduced-order deformation based on Linear Blend Skinning (LBS), and anchor-based Spring-Mass systems, addressing a key portability issue in prior work. Experiments show our approach not only generates a rich variety of plausible dynamics but also reduces Young's Modulus prediction error by over 50% against the strongest deterministic baseline, bridging the gap between static point estimates and the continuous nature of physical reality. Project page: https://unipixie.github.io/

• The paper introduces a unified encoder-decoder that uses flow matching to generate probabilistic, simulation-ready 3D physical properties.
• It employs specialized decoder heads and a simulation-agnostic latent space to ensure consistent performance across MPM, LBS, and Spring-Mass solvers.
• Empirical results show over 50% improvement in Young’s modulus estimation and robust control over material interpolation along the soft-to-stiff continuum.

UniPixie: Unified and Probabilistic 3D Physics Learning via Flow Matching

Introduction and Motivation

"UniPixie: Unified and Probabilistic 3D Physics Learning via Flow Matching" (2606.05399) targets a fundamental challenge in physics-based 3D vision: enabling the generation of plausible, controllable physical parameters from visual input. Conventional approaches in this domain—either test-time optimization or feed-forward prediction—either suffer from scalability/generalization bottlenecks or limit inference to static, deterministic properties. UniPixie advances the field by modeling the inherent physical ambiguity of visual scenes, reframing property inference as a generative task that outputs a continuous spectrum of simulation-ready material fields from a single observation. The explicit modeling of the soft-to-stiff continuum allows dynamic, parameterized simulation and offers portability across multiple physics solvers.

Figure 1: Overview of the UniPixie framework, detailing the unified encoder-decoder architecture that generates controllable physical properties for multiple physics solvers from visual features.

Framework Overview

UniPixie introduces a unified encoder-decoder architecture, operationalizing the notion of a simulation-agnostic latent space. Multi-view RGB observations are processed through a CLIP feature extractor and voxelized, then embedded by a Grid Encoder leveraging cross- and self-attention (Perceiver-IO-like). This yields a compact latent representation capturing geometry, appearance, and semantic priors. Controllable generative modeling is facilitated by a Flow Matching Transformer (FMT) decoder, modulated by a scalar $\alpha \in [0,1]$, representing the interpolation from soft to stiff material states. The FMT is trained with the conditional flow matching objective, learning to transform noise samples to property fields aligned with sampled points on the soft-stiff continuum.

A key architectural contribution is specialized decoder heads for MPM, LBS, and Spring-Mass simulation backends, enabling the model to map its unified latent space to distinct parameterizations required by different physics engines. The consistency of physical behavior across solvers is maintained through a standardized annotation and fitting protocol.

Figure 2: PixieMultiVerse pipeline and annotated parameter range statistics, demonstrating the scalability and diversity of the annotation process.

PixieMultiVerse Dataset

A novel aspect of this work is the introduction of PixieMultiVerse—a large-scale dataset that does not merely annotate objects with point estimates but provides plausible intervals for continuous physical properties (Young's modulus, density, Poisson's ratio) and corresponding discrete material classes. The annotation pipeline combines VLM-generated proposals (GPT-4o for active bounding and Gemini-2.5-Flash for critique) with dense semantic segmentation and is subject to expert verification with an interactive interface.

For solvers with complex parameterizations like LBS and Spring-Mass, ground-truth labels are generated by test-time optimization to MPM-simulated boundary dynamics (for $\alpha=0,1$), followed by interpolation. This ensures cross-solver consistency and bridges the gap between continuum-based and reduced-order/anchor-based simulation paradigms.

Figure 3: Annotation web interface for manual verification, showing simulation outputs and physical property maps at both ends of the annotated parameter range.

Experimental Evaluation

Comparison with Deterministic and Generative Baselines

Empirical results demonstrate that UniPixie achieves statistically significant improvements over state-of-the-art deterministic predictors (e.g., PIXIE, NeRF2Physics, PUGS), with Young's modulus MSE reduced by $>50\%$ relative to the strongest baseline and demonstrating robust generalization across the evaluated physical continuum. This is supported by both pointwise and averaged results over the generative distribution.

Figure 4: Qualitative comparison of predicted dynamics across methods at $\alpha=0.5$—showing improved diversity, plausibility, and consistency for UniPixie, especially when tested on challenging viscoelastic and compliant objects.

Multi-Solver Portability

A central contribution is the demonstration of solver-agnostic, unified generative modeling. UniPixie maintains or exceeds the performance of specialist models in all solver domains—MPM, LBS, and Spring-Mass—both quantitatively (PSNR, SSIM, and LPIPS for video simulation) and qualitatively, with orders-of-magnitude faster inference (∼21s for all solvers vs. 500–4000s for test-time optimization). The results show that a single framework can replace solver-specific pipelines without performance degradation, which is significant for practical deployment at scale.

(Figure 5)

Figure 5: Visualization of controllable generation across solvers and spectrum endpoints, illustrating the learned mapping from soft to stiff dynamics in comparison with solver-specific baselines.

Controllability and Physical Spectrum

UniPixie provides explicit control over the generated material properties via the $\alpha$ parameter, enabling continuous and intuitive modulation of simulation behavior. The predicted distributions at range endpoints align with annotated ground-truth, and the framework robustly interpolates between soft and stiff state behavior in all tested objects and solvers, without the instability or material-mode collapse seen in baselines. This is validated both in simulation metrics and visualizations of dynamic response.

Implications and Future Directions

UniPixie addresses two critical gaps: (1) the need for generative modeling of physical ambiguity from vision, and (2) solver-agnostic portability for simulation-ready property inference. The architecture is compatible with large-scale datasets and fast enough for interactive or real-time pipelines—essential for simulation-centric digital environments, robotics, and AR/VR.

Theoretically, UniPixie's latent space and the flow-matching generative process provide a foundation for extending to multidimensional material manifolds or incorporating uncertainty quantification. Practically, the unified multi-solver design greatly simplifies downstream application engineering, removing the need for per-solver re-identification or tailoring.

Future work may focus on incorporating physical property estimation for occluded or ambiguous regions, expanding the dimensionality of controllable axes beyond stiffness, and integrating task-driven or inverse-design objectives directly into the generative loop.

Conclusion

"UniPixie" (2606.05399) advances the field by providing a robust, unified, and probabilistic architecture for 3D physics learning from vision. Combining flow-matching generative modeling, dataset-level property interval annotations, and multi-solver compatibility, it achieves state-of-the-art accuracy, simulation diversity, and practical efficiency. The framework sets a strong foundation for scalable, controllable physical reasoning in 3D scenes, with substantial implications for simulation-driven applications and future interdisciplinary research.