GLAM: Grounded Latent-Action World Model

• GLAM is a family of latent-space control architectures that jointly model world dynamics and action representation for grounded planning and control.
• It employs latent states, action-conditioned transitions, and tailored grounding mechanisms to align predictions with real-world outcomes.
• Empirical studies in robotics, autonomous driving, and imitation learning demonstrate significant improvements using grounded latent-action interfaces.

Searching arXiv for the cited GLAM-related papers and formulations. Grounded Latent-Action World Model (GLAM) refers to a family of latent-space control architectures in which world dynamics and action representations are coupled in a compact internal space and then tied back to executable behavior, spatial objectives, or semantically specified outcomes. Across recent work, the common pattern is not a single canonical model but a recurring design: latent world states summarize observations, latent actions parameterize controllable transitions, and a grounding mechanism ensures that these latent transitions correspond to real control effects, language goals, or physically realizable trajectories. This pattern appears in task-specific visual-language control, heterogeneous imitation learning, and autonomous driving, with materially different instantiations but a shared emphasis on latent rollout, action-conditioned prediction, and grounded decision making (Lee et al., 23 Nov 2025, Wang et al., 19 Jun 2026, Lu et al., 12 May 2026).

1. Conceptual definition and scope

A useful way to delimit GLAM is to distinguish it from three adjacent classes. First, a conventional latent world model compresses observations and predicts future latent states, but may consume raw actions without learning a meaningful latent action interface. Second, a latent-action model may infer compact action codes from transitions, but need not ground them to executable controls or semantically specified planning objectives. Third, a grounded world model may score futures against language or semantics without learning a fully explicit latent-action control space. The GLAM pattern emerges where these strands coincide: latent dynamics are action-conditioned in latent space, and those latent actions are grounded to planning or control outcomes (Zeng et al., 10 Mar 2026).

This distinction is explicit in "Grounded World Model for Semantically Generalizable Planning" (Li et al., 13 Apr 2026). That work is grounded because future outcomes are predicted in a vision-language aligned latent space and scored against language, but it is only partially latent-action in the stronger sense: its primary formulation uses rendering-based action tokenization rather than a learned latent action variable with a decoder to controls. The paper therefore positions itself closer to a grounded latent-outcome world model than to a fully explicit latent-action architecture. This boundary case is important because it clarifies that GLAM is not merely any world model with language conditioning; it requires a substantive latent-action interface, not only grounded outcome scoring.

2. Architectural motifs and latent representations

Architecturally, GLAM is best understood as a pattern rather than a fixed module inventory. In the task-specific centering model described in "Weakly-supervised Latent Models for Task-specific Visual-Language Control" (Lee et al., 23 Nov 2025), the latent state is $$z_s = \mathrm{concat}(E^{img}_\phi(x_{img}), E^{inst}_\phi(x_{instr}))$$, actions are embedded as $z_a = E^{act}_\phi(a)$, and a learned shift function predicts $\hat{z}_{s'} = z_s + \Delta_\theta(z_s, z_a)$. The design is deliberately narrow: a shared latent space for image and instruction, explicit global action embeddings, and a latent shift toward a goal prototype $z^\star$ computed from centered images. Here, the world model is not a general simulator but a compact latent dynamics model for a fixed spatial objective.

A different architectural regime appears in "Learning Latent Action World Models In The Wild" (Garrido et al., 8 Jan 2026). There, a frozen V-JEPA 2-L frame-causal encoder produces state representations $s_t$, an inverse dynamics model infers latent actions via $z_t = g_\phi(s_t, s_{t+1})$, and a forward model predicts $s_{t+1} = f(s_{0:t}, z_t)$. The latent actions are continuous 128-D vectors by default, and conditioning enters the forward model through AdaLN-zero. The key architectural point is that the action space is not supervised directly; it is inferred from unlabeled video and then later connected to known controls through a separate controller.

In driving, "The DAWN of World-Action Interactive Models" (Lu et al., 12 May 2026) instantiates a more explicitly reciprocal form. Visual observations are encoded into compact world tokens $z = R_{\mathrm{stu}}(E_{\mathrm{stu}}(o))$, a World Predictor rolls out short-horizon latent futures, and a diffusion-based World-Conditioned Action Denoiser samples latent actions conditioned on either current or predicted world latents. The resulting action latents are decoded by an action head into ego trajectories $\hat{\tau} = H_{\mathrm{act}}(a_{1:H})$. This is a GLAM in a strong sense: both world and action live in latent space, and grounding occurs through supervised trajectory decoding.

A further variation is "Chain of World: World Model Thinking in Latent Motion" (Yang et al., 3 Mar 2026), which separates structure and motion through a pretrained video VAE. Video segments are factorized into structure latents $z_s$ and motion latents $z_a = E^{act}_\phi(a)$0, with the latter formed from directional pooled motion branches and flattened to $z_a = E^{act}_\phi(a)$1. A learnable motion query $z_a = E^{act}_\phi(a)$2 then predicts a continuous latent motion chain $z_a = E^{act}_\phi(a)$3 from instruction and initial frame, and that segment-level latent guides both terminal-frame prediction and multi-step discrete action decoding. The architectural emphasis here is disentanglement: GLAM need not use a single monolithic latent, and in some formulations the action-relevant part is a motion chain rather than a conventional control vector.

3. Grounding mechanisms and training objectives

The defining feature of GLAM is not only latent action, but grounded latent action. Different papers implement grounding through different supervisory routes. In "Imitation from Heterogeneous Demonstrations using Grounded Latent-Action World Models" (Wang et al., 19 Jun 2026), grounding is achieved by a shared forward dynamics model $z_a = E^{act}_\phi(a)$4 across sources, a source-invariant inverse dynamics posterior $z_a = E^{act}_\phi(a)$5, and a target-only action encoder $z_a = E^{act}_\phi(a)$6 tied to the inverse model by an asymmetric KL with stop-gradient on the action-encoder branch. The joint objective combines a heterogeneous ELBO, a target ELBO, and an alignment term $z_a = E^{act}_\phi(a)$7. Grounding here means that actions producing the same environmental effect should share the same latent representation, regardless of source or embodiment.

A distinct grounding strategy appears in "VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model" (Sun et al., 10 Feb 2026). Its core claim is leakage-free state prediction: future frames are used only to produce stop-gradient targets through a frozen V-JEPA2 encoder, never as student inputs. The world-model loss is $z_a = E^{act}_\phi(a)$8, and the action head is trained with conditional flow matching. The grounding signal comes from language-conditioned latent-action tokens that must predict future latent states rather than pixel differences, thereby avoiding appearance bias, nuisance motion, and information leakage.

World2Act introduces yet another route: contrastive action-to-dynamics alignment (Vuong et al., 11 Mar 2026). A skill-compositional instruction-conditioned video diffusion model provides video latents $z_a = E^{act}_\phi(a)$9, a video adapter maps them to dynamics embeddings $\hat{z}_{s'} = z_s + \Delta_\theta(z_s, z_a)$0, and an action adapter maps action chunks to $\hat{z}_{s'} = z_s + \Delta_\theta(z_s, z_a)$1. The alignment objective is a bidirectional, chunkwise InfoNCE loss with temperature $\hat{z}_{s'} = z_s + \Delta_\theta(z_s, z_a)$2, paired with an action reconstruction term. In this formulation, grounding is not imposed by decoding the world model into pixels or by direct action supervision from generated futures; instead, actions are post-trained to inhabit the latent manifold of a strong world model.

"Making Foresight Actionable: Repurposing Representation Alignment in World Action Models" identifies a failure mode that is central to GLAM’s rationale: plausible future prediction does not imply action-readability (Qiu et al., 10 Jun 2026). AGRA adds an alignment loss

$\hat{z}_{s'} = z_s + \Delta_\theta(z_s, z_a)$3

between intermediate world-model features and DINOv2 spatial semantics. The empirical diagnosis is an action-grounding gap: action decoders in baseline world-action models attend to distractors and remain sensitive to task-irrelevant tokens. AGRA therefore treats grounding as an intermediate-representation problem rather than only a latent-dynamics problem.

At the narrowest end of the spectrum, the centering model in (Lee et al., 23 Nov 2025) grounds latent actions through a weakly supervised goal prototype and a composite loss $\hat{z}_{s'} = z_s + \Delta_\theta(z_s, z_a)$4. The directional term encourages goalward movement in latent space, the ranking term enforces correct action selection, and the consistency and regularization terms tether state-specific shifts to global action anchors. This is a particularly clear example of grounding without dense transition labels: only triples $\hat{z}_{s'} = z_s + \Delta_\theta(z_s, z_a)$5 are available, yet latent actions acquire stable semantics.

4. Planning and control interfaces

Once grounded latent actions are available, GLAM supports several planning interfaces. In the autonomous inspection setting, inference is a repeated latent one-step rollout: encode the current image and instruction, predict $\hat{z}_{s'} = z_s + \Delta_\theta(z_s, z_a)$6 for each discrete action, measure the distance $\hat{z}_{s'} = z_s + \Delta_\theta(z_s, z_a)$7 to the goal prototype, and choose $\hat{z}_{s'} = z_s + \Delta_\theta(z_s, z_a)$8 (Lee et al., 23 Nov 2025). Planning is therefore not a separate optimizer but an action-ranking operation over a compact latent transition model.

In the in-the-wild latent action model, control requires an additional mapping from known actions to latent actions because the learned latent space is camera-relative and unlabeled (Garrido et al., 8 Jan 2026). The controller $\hat{z}_{s'} = z_s + \Delta_\theta(z_s, z_a)$9 maps known actions and context into latent actions, after which planning minimizes

$z^\star$0

using CEM over rollouts $z^\star$1. This formulation makes explicit that a GLAM may require a separate actuation interface when its latent actions are discovered from passive video.

DAWN pushes the control interface further toward joint inference (Lu et al., 12 May 2026). It alternates between a World Predictor and an Action Denoiser: $z^\star$2, $z^\star$3, $z^\star$4. This reciprocal loop is the characteristic WAIM contribution: actions shape the future hypothesis, and the future hypothesis shapes action denoising.

By contrast, the Grounded World Model for semantically generalizable planning replaces the usual goal-image metric in visuomotor MPC with instruction alignment in a frozen vision-language embedding space (Li et al., 13 Apr 2026). Candidate action chunks are proposed by KNN in joint space, future latent outcomes are predicted, and proposals are scored by cosine similarity to instruction embeddings. This suggests a broader interpretation of GLAM planning: some systems plan directly in latent action space, while others use grounded latent outcome prediction to evaluate externally proposed actions.

5. Empirical evidence across domains

The empirical record shows that GLAM-like designs are not confined to one application domain. In task-specific autonomous inspection, the latent shift model for centering a detected object achieves approximately $z^\star$5 success on held-out images and instructions, whereas direct multimodal LLM control remains at or below approximately $z^\star$6 under the same task framing (Lee et al., 23 Nov 2025). The paper’s ablation is also unusually sharp: removing the ranking loss collapses accuracy to approximately $z^\star$7.

In robotic VLA pretraining, segment-level latent motion and leakage-free latent state prediction both deliver strong evidence that grounded latent dynamics are more useful than pixel-grounded alternatives. CoWVLA reports LIBERO scores of $z^\star$8 across Spatial/Object/Goal/Long, for an average of $z^\star$9, and on SimplerEnv-WidowX reports an average of $s_t$0 (Yang et al., 3 Mar 2026). VLA-JEPA reports LIBERO scores of Spatial $s_t$1, Object $s_t$2, Goal $s_t$3, Long $s_t$4, average $s_t$5, and on LIBERO-Plus reports an average of $s_t$6, with the strongest gains under camera, language, light, background, noise, and layout perturbations relative to its ablations (Sun et al., 10 Feb 2026).

World-model-based post-training and self-supervised latent action pretraining provide complementary evidence. World2Act improves GR00T-N1.6-ft on RoboCasa from $s_t$7 to $s_t$8 and reports a real-world average success-rate improvement of $s_t$9 across three Franka Research 3 tasks (Vuong et al., 11 Mar 2026). The model-agnostic LAWM framework trained through world modeling reports LIBERO suite scores of Spatial $z_t = g_\phi(s_t, s_{t+1})$0, Object $z_t = g_\phi(s_t, s_{t+1})$1, Goal $z_t = g_\phi(s_t, s_{t+1})$2, Long $z_t = g_\phi(s_t, s_{t+1})$3, average $z_t = g_\phi(s_t, s_{t+1})$4, and raises real-world average success from $z_t = g_\phi(s_t, s_{t+1})$5 to $z_t = g_\phi(s_t, s_{t+1})$6 after pretraining on human videos (Tharwat et al., 22 Sep 2025). For control with unlabeled trajectories, the LAWM formulation of "Latent Action World Models for Control with Unlabeled Trajectories" reports an average normalized return of $z_t = g_\phi(s_t, s_{t+1})$7 using only $z_t = g_\phi(s_t, s_{t+1})$8 labeled trajectories, versus $z_t = g_\phi(s_t, s_{t+1})$9 for C-LAP and $s_{t+1} = f(s_{0:t}, z_t)$0 for TD3+BC under the reported averages across tasks and datasets (Alles et al., 10 Dec 2025).

The explicitly named GLAM formulation for heterogeneous demonstrations shows the strongest cross-source imitation result in the data block. Across three real and two simulated manipulation tasks, GLAM-aligned policies yield an average of $s_{t+1} = f(s_{0:t}, z_t)$1 improvement in task success rate under the same data-scarce setting, and on bimanual stack-three achieve $s_{t+1} = f(s_{0:t}, z_t)$2 success while all listed baselines remain at or below $s_{t+1} = f(s_{0:t}, z_t)$3 (Wang et al., 19 Jun 2026).

Driving results show that the pattern scales beyond manipulation. DAWN reaches PDMS $s_{t+1} = f(s_{0:t}, z_t)$4 on NAVSIM v1 and on nuScenes end-to-end planning reports L2 error $s_{t+1} = f(s_{0:t}, z_t)$5 m at $s_{t+1} = f(s_{0:t}, z_t)$6 s, $s_{t+1} = f(s_{0:t}, z_t)$7 m at $s_{t+1} = f(s_{0:t}, z_t)$8 s, $s_{t+1} = f(s_{0:t}, z_t)$9 m at $z = R_{\mathrm{stu}}(E_{\mathrm{stu}}(o))$0 s, with average collision rate $z = R_{\mathrm{stu}}(E_{\mathrm{stu}}(o))$1 (Lu et al., 12 May 2026). The semantically grounded MPC system in (Li et al., 13 Apr 2026) reports an $z = R_{\mathrm{stu}}(E_{\mathrm{stu}}(o))$2 success rate on the WISER benchmark test set of $z = R_{\mathrm{stu}}(E_{\mathrm{stu}}(o))$3 tasks with unseen visual signals and referring expressions, while the cited VLA baselines average $z = R_{\mathrm{stu}}(E_{\mathrm{stu}}(o))$4 on the same test set. In the unsupervised in-the-wild latent-action setting, continuous constrained latents support manipulation planning with $z = R_{\mathrm{stu}}(E_{\mathrm{stu}}(o))$5–$z = R_{\mathrm{stu}}(E_{\mathrm{stu}}(o))$6 m and navigation with RPE $z = R_{\mathrm{stu}}(E_{\mathrm{stu}}(o))$7, outperforming NoMaD’s $z = R_{\mathrm{stu}}(E_{\mathrm{stu}}(o))$8 while trailing the action-trained NWM at $z = R_{\mathrm{stu}}(E_{\mathrm{stu}}(o))$9 (Garrido et al., 8 Jan 2026).

Paper	Domain	Reported result
(Lee et al., 23 Nov 2025)	Drone centering	Approximately $\hat{\tau} = H_{\mathrm{act}}(a_{1:H})$0 success; direct LLM control $\hat{\tau} = H_{\mathrm{act}}(a_{1:H})$1
(Yang et al., 3 Mar 2026)	Robotic VLA	LIBERO average $\hat{\tau} = H_{\mathrm{act}}(a_{1:H})$2
(Sun et al., 10 Feb 2026)	Robotic VLA	LIBERO average $\hat{\tau} = H_{\mathrm{act}}(a_{1:H})$3; LIBERO-Plus average $\hat{\tau} = H_{\mathrm{act}}(a_{1:H})$4
(Vuong et al., 11 Mar 2026)	VLA post-training	RoboCasa $\hat{\tau} = H_{\mathrm{act}}(a_{1:H})$5; real-world $\hat{\tau} = H_{\mathrm{act}}(a_{1:H})$6
(Tharwat et al., 22 Sep 2025)	Self-supervised robot pretraining	LIBERO suites average $\hat{\tau} = H_{\mathrm{act}}(a_{1:H})$7; real-world $\hat{\tau} = H_{\mathrm{act}}(a_{1:H})$8
(Alles et al., 10 Dec 2025)	Offline RL with unlabeled data	Average normalized return $\hat{\tau} = H_{\mathrm{act}}(a_{1:H})$9 with $z_s$0 labeled trajectories
(Wang et al., 19 Jun 2026)	Heterogeneous imitation	Average $z_s$1 success-rate improvement
(Lu et al., 12 May 2026)	Autonomous driving	PDMS $z_s$2; nuScenes Avg L2 $z_s$3 m
(Li et al., 13 Apr 2026)	Language-grounded MPC	WISER test success $z_s$4

Taken together, these results suggest that GLAM is less a single benchmark-defined paradigm than a recurrent solution to a common bottleneck: planning and control benefit when the model’s predictive substrate and its action interface share a grounded latent semantics.

6. Limitations, misconceptions, and open directions

A recurring misconception is that accurate world prediction is sufficient for control. The AGRA analysis directly contradicts this: plausible future generation can coexist with an action-grounding gap in which the action decoder attends to distractors, is causally sensitive to background tokens, and fails to focus on hand-object interaction regions (Qiu et al., 10 Jun 2026). GLAM therefore cannot be reduced to generative fidelity; the latent interface between prediction and control is itself a design object.

A second misconception is that every latent-action model is inherently grounded. The in-the-wild latent action world model learns actions that are mainly localized in space relative to the camera because there is no common embodiment across videos (Garrido et al., 8 Jan 2026). This is still useful for transfer and planning, but it also means that mapping known controls into the latent space requires contextual information; without $z_s$5, the controller converges to “no movement.” Grounding is thus relative to what supervision and invariances are available.

Several current GLAM instantiations are intentionally narrow. The centering model is fixed-goal, single-object, and not a full world model; it does not model temporal stochasticity or long-horizon dynamics, and its dominant failure mode is hesitation near the image centerline between a small corrective action and none (Lee et al., 23 Nov 2025). DAWN, for its part, provides no formal convergence or safety guarantees for the interactive loop, and its performance saturates after four interactive rounds (Lu et al., 12 May 2026).

The broadest methodological open questions concern scale, stability, and evaluation. The driving taxonomy paper argues that latent world models should be assessed not only by open-loop fidelity but also by closed-loop safety and resource-aware deliberation, proposing the Closed-loop Safety Gap $z_s$6, the Temporal Coherence Score $z_s$7, and the Deliberation Cost $z_s$8 (Zeng et al., 10 Mar 2026). This suggests that future GLAM work will likely be judged less by latent elegance alone than by whether latent rollouts actually improve safe, closed-loop behavior per unit compute.

The directions proposed across the papers are consistent. They include extending task-specific GLAMs from $z_s$9D to $z_a = E^{act}_\phi(a)$00D and to multiple objects and longer horizons, improving simulation-to-real transfer, exploring adaptive regularization and better samplers for continuous latent actions, integrating uncertainty-aware planning in latent motion space, and adding tactile or force signals to vision-grounded latent action spaces (Lee et al., 23 Nov 2025, Garrido et al., 8 Jan 2026, Yang et al., 3 Mar 2026, Wang et al., 19 Jun 2026). The cumulative implication is that GLAM is converging toward a broader research program: not merely compressing world and action into latent variables, but constructing latent interfaces whose semantics are stable enough to plan with, rich enough to generalize, and grounded enough to execute.