Latent Particle World Model (LPWM)

• LPWM is a self-supervised object-centric world model that represents each video frame as a set of latent particles with interpretable spatial and appearance features.
• It introduces per-particle latent actions for stochastic dynamics, enabling conditioning on actions, language, and goal images for multi-object prediction.
• Empirical results show LPWM outperforms baselines on metrics like LPIPS and FVD and enhances decision-making in downstream control tasks.

Latent Particle World Model (LPWM) is a self-supervised object-centric world model that learns directly from raw videos and represents each frame as a set of learned latent particles with explicit spatial and visual attributes. It is designed to scale beyond toy object-centric benchmarks to complex real-world multi-object data, supports conditioning on actions, language, image goals, and multiple camera views, and models stochastic future evolution through per-particle latent actions rather than a single global latent transition (Daniel et al., 4 Mar 2026). In this formulation, “latent particle” does not denote a physical particle simulator; it denotes a structured latent object token carrying interpretable attributes such as keypoint location, scale, depth-like ordering, transparency, and appearance.

1. Conceptual position and scope

LPWM addresses a recurrent tension in world modeling. Large general-purpose video models are powerful but holistic and computationally expensive, whereas earlier object-centric models are often more interpretable and more naturally aligned with reasoning and control, but historically have struggled to scale to complex real-world videos. LPWM is introduced to combine compact structured representations, end-to-end learning from videos alone, stochastic prediction, and compatibility with decision-making (Daniel et al., 4 Mar 2026).

Its immediate precursor is Deep Latent Particles (DLP), which represents an image as a set of latent particles, each with a 2D spatial coordinate and a local appearance code, but remains a static image model with only a downstream particle-based video prediction experiment layered on top (Daniel et al., 2022). LPWM inherits the particle formulation and extends it into a temporal VAE with learned stochastic dynamics, end-to-end training on video, and explicit support for action, language, and goal conditioning (Daniel et al., 4 Mar 2026).

A central distinction from slot-based world models is that LPWM uses particle-like units with explicit coordinates, scale, depth, transparency, and appearance rather than unordered slot vectors. A central distinction from DDLP is that LPWM removes explicit tracking and preserves all particle identities by keeping patch-origin identity, enabling parallel frame encoding and better scalability (Daniel et al., 4 Mar 2026). This places LPWM between fixed-patch latent models and fully tracked object-particle systems: particles retain local patch-origin structure, but their attributes evolve through learned dynamics.

A common misconception is to read LPWM as a physical particle simulator in the style of point-cloud dynamics. That is not its intended ontology. In LPWM, particles are compact object-centric latent units. By contrast, visual particle simulators such as VPD and HD-VPD use 3D particles grounded in RGB-D geometry and evolve them as scene-centric point clouds for rendering and dynamics prediction (Whitney et al., 2023, Whitney et al., 2024). LPWM instead is object-centric, video-native, and designed around interpretable latent decomposition plus stochastic decision-relevant prediction.

2. Latent particle state and unsupervised scene decomposition

The core LPWM state is a foreground particle set together with a single background particle. Each foreground particle is

$$z^{m,\mathrm{fg}} = [z_p, z_s, z_d, z_t, z_f] \in \mathbb{R}^{6 + d_{\text{obj}}},$$

where $z_p \in \mathbb{R}^2$ is 2D position, $z_s \in \mathbb{R}^2$ is scale, $z_d \in \mathbb{R}$ is depth-like compositing order, $z_t \in [0,1]$ is transparency, and $z_f \in \mathbb{R}^{d_{\text{obj}}}$ is appearance (Daniel et al., 4 Mar 2026). The background is represented by a separate latent $z^{\mathrm{bg}} \in \mathbb{R}^{d_{\text{bg}}}$.

These attributes are not supervised directly. LPWM discovers keypoints, bounding boxes, and object masks from raw videos alone by combining patch-based keypoint proposals, spatial softmax, STN-based glimpse extraction, object-wise decoding, alpha compositing, and transparency regularization (Daniel et al., 4 Mar 2026). The encoder first divides the image into $M$ non-overlapping patches of size $D \times D$, predicts one keypoint proposal per patch, and refines each proposal via a learned offset. In the encoder factorization,

$$q_\phi(z|x) = q_\phi(z_a|x)\; q_\phi(z_o,z_s,z_d,z_t \mid x,z_a)\; q_\phi(z_f \mid x,z_p,z_s),$$

$z_p \in \mathbb{R}^2$0 denotes a proposal location, $z_p \in \mathbb{R}^2$1 an offset, and $z_p \in \mathbb{R}^2$2 the refined keypoint position (Daniel et al., 4 Mar 2026). Spatial-softmax moments provide proposal coordinates and uncertainty estimates through $z_p \in \mathbb{R}^2$3.

The particle attributes correspond directly to common object-centric outputs:

Attribute	Role
$z_p \in \mathbb{R}^2$4	keypoint location
$z_p \in \mathbb{R}^2$5	box size proxy
$z_p \in \mathbb{R}^2$6	compositing depth
$z_p \in \mathbb{R}^2$7	visibility / activeness
$z_p \in \mathbb{R}^2$8	appearance descriptor

Because the model uses a fixed particle budget $z_p \in \mathbb{R}^2$9, the maximum number of represented entities is architecture-bound, but the effective number of active entities is variable. Transparency suppresses unused particles, and the transparency regularizer

$z_s \in \mathbb{R}^2$0

encourages sparse active sets (Daniel et al., 4 Mar 2026). The background particle separately captures static scene content that remains after visible foreground regions are masked out.

Several inductive biases push the decomposition toward object-like structure. Position and scale are separated from appearance; local STN glimpses enforce spatially localized encoding and decoding; depth-aware alpha compositing makes visibility relational; and the attribute KL is masked by transparency so invisible particles do not dominate regularization (Daniel et al., 4 Mar 2026). Relative to earlier DLP variants, LPWM incorporates DLPv3-style refinements such as depth via particle attention, residual appearance encoding, and bounded Beta parameterization for transparency, improving stability in multi-object video settings (Daniel et al., 4 Mar 2026).

3. Stochastic dynamics via per-particle latent actions

LPWM’s principal dynamical innovation is to represent transition stochasticity through per-particle latent actions. Instead of assuming that one global latent transition suffices for the entire frame, LPWM introduces a context module that infers one latent action distribution per particle and timestep (Daniel et al., 4 Mar 2026).

The context module has two heads. The first is a latent inverse dynamics posterior,

$z_s \in \mathbb{R}^2$1

and the second is a latent policy prior,

$z_s \in \mathbb{R}^2$2

The latent action is Gaussian,

$z_s \in \mathbb{R}^2$3

with training-time samples drawn from the inverse head and inference-time samples drawn from the policy prior (Daniel et al., 4 Mar 2026). The dynamics model then predicts the next-step particle prior as

$z_s \in \mathbb{R}^2$4

This factorization makes the source of uncertainty explicit. Multiple future trajectories arise because different samples of $z_s \in \mathbb{R}^2$5 induce different particle transitions, while the particle state itself remains interpretable. The design is especially useful when different objects move independently or when several localized interactions occur simultaneously. The paper contrasts this with global latent-action models, arguing that a single global latent action is inadequate for settings such as independent enemy movements in Mario or localized contact-induced motion in robotics (Daniel et al., 4 Mar 2026).

LPWM also supports several conditioning modalities through the same context pathway. Global actions are projected to the transformer hidden dimension, repeated across particles, and injected via AdaLN. Language is encoded with pretrained T5-large, appended as $z_s \in \mathbb{R}^2$6 language tokens to the $z_s \in \mathbb{R}^2$7 particle tokens at every timestep, and fused through self-attention. Goal images are encoded into goal particles and used to condition current particles through AdaLN. For multi-view input, each view contributes $z_s \in \mathbb{R}^2$8 particles, the particles are concatenated into a $z_s \in \mathbb{R}^2$9-particle set, and a learned view embedding enables cross-view attention (Daniel et al., 4 Mar 2026).

Because LPWM removes explicit tracking, particles maintain patch-origin identity. The paper characterizes this as a “particle-grid regime”: particles can move locally around their patch origins, and object features can transfer to nearby particles as objects move across regions. This is more flexible than fixed-patch models but less unconstrained than a fully tracked object-particle system (Daniel et al., 4 Mar 2026).

4. Temporal VAE objective, training, and inference

LPWM is trained end-to-end as a temporal VAE. The total loss is written as

$z_d \in \mathbb{R}$0

For the first frame, the static term is

$z_d \in \mathbb{R}$1

where the fixed prior $z_d \in \mathbb{R}$2 applies to particle attributes and $z_d \in \mathbb{R}$3 penalizes transparency (Daniel et al., 4 Mar 2026). The attribute KL is masked by transparency so that invisible particles do not dominate regularization, except for the transparency term itself.

For later frames, the dynamic term is

$z_d \in \mathbb{R}$4

The first KL term trains the particle dynamics prior; the second aligns inverse-inferred latent actions with the latent policy prior (Daniel et al., 4 Mar 2026). The context KL is not masked, because latent actions must also explain particle activation and deactivation.

Reconstruction uses $z_d \in \mathbb{R}$5 on simulated datasets and $z_d \in \mathbb{R}$6 on real-world datasets, with $z_d \in \mathbb{R}$7 and $z_d \in \mathbb{R}$8 denoting VGG features as in LPIPS (Daniel et al., 4 Mar 2026). This combines pixel fidelity with perceptual similarity for real videos.

Training uses Adam with learning rate $z_d \in \mathbb{R}$9, latent action dimension $z_t \in [0,1]$0, teacher forcing, and a warmup period in which static ELBO is applied to the first $z_t \in [0,1]$1 frames and dynamic ELBO only to the final frame (Daniel et al., 4 Mar 2026). Unlike earlier deterministic video-prediction setups, LPWM does not use burn-in frames, because stochasticity is handled through the latent action mechanism.

At inference time, the system becomes a generative world model. It samples latent actions from the latent policy prior, predicts future particle distributions autoregressively, and decodes them into future frames. Because the latent action prior is sampled rather than collapsed to a deterministic mode, LPWM supports multimodal rollouts on stochastic datasets (Daniel et al., 4 Mar 2026).

5. Empirical behavior and decision-making use

LPWM is evaluated on synthetic and real-world video datasets spanning deterministic and stochastic regimes. The reported datasets include OBJ3D, PHYRE, Mario, Sketchy, BAIR, Bridge, and LanguageTable, with video metrics including PSNR, SSIM, LPIPS, and FVD (Daniel et al., 4 Mar 2026).

Representative stochastic-generation results show substantial gains over a non-object-centric DVAE baseline. On Sketchy-U, DVAE reports LPIPS $z_t \in [0,1]$2 and FVD $z_t \in [0,1]$3, whereas LPWM reports LPIPS $z_t \in [0,1]$4 and FVD $z_t \in [0,1]$5. On Mario-U, DVAE reports LPIPS $z_t \in [0,1]$6 and FVD $z_t \in [0,1]$7, whereas LPWM reports LPIPS $z_t \in [0,1]$8 and FVD $z_t \in [0,1]$9. On Bridge-L, DVAE reports LPIPS $z_f \in \mathbb{R}^{d_{\text{obj}}}$0 and FVD $z_f \in \mathbb{R}^{d_{\text{obj}}}$1, whereas LPWM reports LPIPS $z_f \in \mathbb{R}^{d_{\text{obj}}}$2 and FVD $z_f \in \mathbb{R}^{d_{\text{obj}}}$3 (Daniel et al., 4 Mar 2026).

For deterministic prediction, LPWM is strongest on OBJ3D and approximately tied with strong baselines on PHYRE. On OBJ3D, LPWM reaches LPIPS $z_f \in \mathbb{R}^{d_{\text{obj}}}$4, compared with DVAE $z_f \in \mathbb{R}^{d_{\text{obj}}}$5, G-SWM $z_f \in \mathbb{R}^{d_{\text{obj}}}$6, SlotFormer/OCVP $z_f \in \mathbb{R}^{d_{\text{obj}}}$7, and DDLP $z_f \in \mathbb{R}^{d_{\text{obj}}}$8. On PHYRE, LPWM reports $z_f \in \mathbb{R}^{d_{\text{obj}}}$9, compared with DVAE $z^{\mathrm{bg}} \in \mathbb{R}^{d_{\text{bg}}}$0 and DDLP $z^{\mathrm{bg}} \in \mathbb{R}^{d_{\text{bg}}}$1 (Daniel et al., 4 Mar 2026). The paper also reports that a compact LPWM reaches FVD $z^{\mathrm{bg}} \in \mathbb{R}^{d_{\text{bg}}}$2 on BAIR-64 under the 1-frame-conditioned, 15-frame-predicted setting, using this to argue that object-centric inductive bias can compensate for scale (Daniel et al., 4 Mar 2026).

Qualitatively, the model is presented as maintaining object permanence over long rollouts, producing less blur and deformation than holistic baselines, and supporting language-conditioned and goal-conditioned future generation. The particle representation is directly inspectable through discovered keypoints, boxes, masks, and foreground/background layers (Daniel et al., 4 Mar 2026).

LPWM is also used for downstream control, specifically goal-conditioned imitation learning. A pre-trained unconditional LPWM is frozen, image sequences are encoded into particles, per-particle latent actions are inferred with the inverse dynamics head, and a compact two-layer attention-pooling transformer maps the particle-level latent actions to a single global action:

$z^{\mathrm{bg}} \in \mathbb{R}^{d_{\text{bg}}}$3

with training loss $z^{\mathrm{bg}} \in \mathbb{R}^{d_{\text{bg}}}$4 (Daniel et al., 4 Mar 2026). The paper describes this on PandaPush and OGBench-Scene and emphasizes that LPWM can first be learned from actionless videos and only later paired with video-action trajectories for decision-making.

6. Relation to neighboring world-model families and principal limitations

LPWM occupies a specific position within the broader latent world-model literature. It is more structured than monolithic latent predictors such as LeWorldModel, which uses a single global embedding and a deterministic JEPA-style next-latent predictor rather than a set of object-centric particles (Maes et al., 13 Mar 2026). It is also distinct from hierarchical latent planners that improve long-horizon control by planning across temporal scales in a shared latent space without explicit uncertainty or entity structure (Zhang et al., 3 Apr 2026). In the language of world models, LPWM is explicitly object-centric and stochastic; those models are latent but not particle-based.

Relative to particle-based visual simulators such as VPD and HD-VPD, LPWM’s “particles” are not 3D point-cloud elements grounded by RGB-D unprojection. VPD and HD-VPD maintain scene-centric 3D particle clouds and evolve them with geometric interaction networks, whereas LPWM maintains object-centric 2D latent units intended for semantic decomposition, stochastic prediction, and conditioning on task variables such as language and goals (Whitney et al., 2023, Whitney et al., 2024). Relative to the Deep Latent Space Particle Filter, LPWM is a world model for stochastic video modeling and decision-making rather than a latent Bayesian filtering method for data assimilation (Mücke et al., 2024).

The main limitations arise from the same design choices that make LPWM scalable. The particle budget $z^{\mathrm{bg}} \in \mathbb{R}^{d_{\text{bg}}}$5 is fixed by architecture, although the active subset varies through transparency. Because explicit tracking is removed, LPWM relies on patch-origin identity and a particle-grid regime rather than unconstrained long-range identity persistence. This supports scalability but imposes locality constraints when objects traverse large spatial extents (Daniel et al., 4 Mar 2026). The model therefore lies between fixed-patch latent tokenization and fully tracked object-particle dynamics, rather than reducing cleanly to either.

Another limitation is conceptual rather than empirical: LPWM’s particles are structured latent object tokens, not physical state variables in the sense used by particle simulators or data-assimilation particle filters. This distinction matters when comparing LPWM to methods that model calibrated uncertainty over physical state trajectories or explicit 3D geometry (Mücke et al., 2024, Whitney et al., 2024). A plausible implication is that LPWM is particularly well matched to stochastic video modeling, language- and goal-conditioned generation, and imitation-style downstream control, while geometry-heavy forecasting and physically grounded state estimation remain stronger in 3D particle systems.

Within object-centric world modeling, however, LPWM’s main significance is clear. It shows that keypoints, boxes, masks, stochastic latent dynamics, and decision-relevant conditioning can be learned end-to-end from videos alone in a single object-centric framework, and that per-particle latent actions provide a concrete mechanism for scaling stochastic multi-object prediction beyond deterministic or globally conditioned alternatives (Daniel et al., 4 Mar 2026).