---
title: 'World-Model Interfaces: Concepts & Design'
url: https://www.emergentmind.com/topics/world-model-interfaces
type: topic
---

# World-Model Interfaces: Concepts & Design

Searching arXiv for recent papers on world-model interfaces and related interface abstractions.
World-model interfaces are the formal, algorithmic, and software contracts through which a world model links perception, internal state, action, prediction, memory, and intervention. In recent work, these contracts appear at multiple levels: as probabilistic operators such as $p(s_{t+1}\mid s_t,a_t)$ and $p(o_t\mid s_t)$, as callable APIs such as `encode`, `transition`, `decode`, and `get_cost`, and as higher-level protocols for simulation, querying, rendering, planning, and memory updates. Contemporary research increasingly treats the interface itself as a first-class design object, because closed-loop utility depends not only on predictive fidelity but also on controllability, modularity, and the ability to expose internal structure to downstream agents and planners [2606.00133][2602.08968][2510.18135].

## 1. Formal definitions and core interface primitives

A general formalization treats the world model as a transition model together with an observation or emission model. The survey literature writes this in the compact form $p_\theta(s_{t+1}\mid s_t,a_t)$ and $q_\theta(o_t\mid s_t)$, or, in a factored control setting, $p_\theta(s_{t+1},o_{t+1},r_t\mid s_t,a_t)$ [2606.00133]. This is the minimal interface: a current internal state, an action-conditioned transition, and a map back to observable consequences.

OpenWorldLib makes this abstraction explicit at the software level by defining an advanced world model as a model or framework centered on building internal representations from perception, equipped with action-conditioned simulation and long-term memory [2604.04707]. Its interface decomposition is correspondingly modular: Perception is exposed through `BaseOperator`, implicit interaction through `BaseSynthesis`, explicit 3D state through `BaseRepresentation`, long-term storage through `BaseMemory`, and orchestration through `BasePipeline`. The pipeline follows a uniform `from_pretrained → process → infer → update_memory` lifecycle, which turns architectural capabilities into stable callable contracts rather than ad hoc task code [2604.04707].

The same logic appears in stable-worldmodel-v1, which assumes a world model implements
$$
z_{t+1}=f(z_t,a_t;\theta_f),\qquad \hat o_t=g(z_t;\theta_g),
$$
with an optional encoder $h(o_t)\to z_t$ [2602.08968]. Its required methods—`encode`, `transition`, `decode`, and `get_cost`—show a particularly clear separation between representation, dynamics, rendering, and planning cost. Because planners repeatedly call `transition` and accumulate `get_cost`, the world-model interface is directly optimized for MPC-style use rather than merely for offline prediction [2602.08968].

A distinct but related decomposition is introduced by Web World Models, which split state into a deterministic physics component and a model-generated imagination component:
$$
S_t=\bigl(S_t^\phi,S_t^\psi\bigr).
$$
The update rule is correspondingly bifurcated: `stepPhysics` deterministically computes $S_{t+1}^\phi=f_{\mathrm{code}}(S_t^\phi,a_t)$, while `stepImagination` samples $S_{t+1}^\psi\sim \pi_\theta(\cdot\mid S_{t+1}^\phi)$ under typed web interfaces and runtime validation [2512.23676]. This separation makes the interface itself a correctness mechanism: logical invariants live in code, while open-ended content generation is constrained by schemas.

## 2. Representation as interface: from explicit scene structure to latent control

One major axis of world-model interface design is the representation exposed between perception and dynamics. VDAWorld instantiates an explicitly structured interface in which an image-caption pair $(I,C)$ is mapped to an abstract scene representation
$$
S_0=\{\,o_i:1\le i\le N\},
$$
with each object parameterized by shape type, pose, orientation, mass, friction, restitution, and inferred initial velocity [2512.11061]. The representation is stored as native Python lists, dicts, or a small class, and is then passed to a simulator selector that chooses among `RigidBodyEngine`, `FluidEngine`, `SoftBodyEngine`, or `LogicEngine` based on $S_0$ and caption keywords such as “water,” “lava,” “smoke,” or “Game of Life” [2512.11061]. The paper explicitly notes that no end-to-end abstraction loss $L_{\rm abst}$ is reported; perception modules are pre-trained and fixed [2512.11061]. The interface is therefore deliberately transparent and simulator-oriented rather than end-to-end latent.

ViMo introduces a different abstraction layer for GUI dynamics. Its Symbolic Text Representation replaces dynamic text regions with placeholder rectangles while preserving graphics:
$$
\mathrm{STR}(x)(u,v)=
\begin{cases}
\mathrm{placeholder}(u,v) & \text{if }(u,v)\in\cup_i B_i\text{ and }B_i\text{ dynamic text}\\
x(u,v) & \text{otherwise.}
\end{cases}
$$
This interface decouples graphic prediction from text generation: a diffusion-based STR Predictor models layout, color, and style, while a GUI-text Predictor restores content token by token [2504.13936]. The representation is neither purely pixel-level nor purely textual; it is a structured intermediate contract tuned to GUI semantics.

WorldAct moves the interface into 3D scene decomposition. Starting from a monolithic 3D Gaussian Splatting world model, it produces a background Gaussian set $\mathcal G_{\rm bg}^{\rm comp}$, a collision mesh $\mathcal M_{\rm bg}$, and per-object tuples $(\mathcal G_m^{\rm gen},\mathcal M_m^{\rm gen},T_m)$ [2605.15843]. The interface is dual-form: 3DGS primitives for differentiable rendering and textured meshes for collision, physics, and editing. This duality is central to the framework’s purpose, because editability and embodied manipulation require more than a visually coherent renderer [2605.15843].

A more latent formulation appears in BRICKS-WM, which factorizes state into an actuated Agent slot, a Background slot, and a scalar latent interface:
$$
s_t\equiv (s_t^a,s_t^b),\qquad z_t\in\mathbb R^{d_I}.
$$
The transition factorization is unidirectional:
$$
p(s_t^a,z_{t-1},s_t^b\mid s_{t-1}^a,s_{t-1}^b,a_{t-1})
=
p_{\rm ag}(s_t^a\mid s_{t-1}^a,a_{t-1})
\,p_I(z_{t-1}\mid s_t^a,a_{t-1})
\,p_{\rm bg}(s_t^b\mid s_{t-1}^b,z_{t-1}),
$$
so background dynamics never directly observe the agent action [2606.16489]. Here the interface is explicitly functional rather than visual: it is a bottleneck through which one subsystem can influence another.

World modeling through Lie Action offers another latent-control interface. It encodes each frame into object slots $z_1,\dots,z_N$ and models action in a continuous Lie-group-structured latent space, with composition enforced by matrix multiplication,
$$
M(a_1\odot a_2)=M(a_1)\cdot M(a_2),
$$
and dynamics governed by $\frac{dz}{dt}=A(t)z(t)$ [2503.09911]. Olaf-World addresses a related problem at the level of latent action identifiability: Seq$\Delta$-REPA aligns the sequence-integrated latent action with a frozen video encoder’s effect direction,
$$
\mathcal L^{\rm Seq\Delta\text{-}REPA}_\psi
=
1-\langle \mathrm{norm}(u),\mathrm{norm}(\tau_*)\rangle,
$$
so that latent actions acquire a shared coordinate system across contexts [2602.10104]. Taken together, these systems suggest a spectrum in which interfaces range from explicit geometry and typed objects to low-dimensional, semantically aligned latent control variables.

## 3. Action channels and control formats

A world-model interface is also defined by how actions enter the system. World-in-World formalizes this point by requiring a standardized action API for heterogeneous models. Its controller supports three principal control formats: text prompts for text-to-video models, camera trajectories for trajectory-conditioned models, and low-level action vectors for action-conditioned models [2510.18135]. Observations are image tensors such as $[C,H,W]$ or panoramas of shape $[3,576,1024]$, while actions may be discrete integers or continuous vectors in $\mathbb R^7$ for a 7-DoF gripper [2510.18135]. The interface therefore abstracts over model family without collapsing action semantics.

ActWorld extends the action channel beyond navigation-centric control. It conditions chunk-autoregressive video generation on a starting frame, per-chunk semantic captions, low-level keyboard/mouse controls, high-level action commands such as pick up or pour, and hierarchical memory [2606.17730]. Its low-level camera control vocabulary comprises $9$ keyboard $\times$ $9$ mouse combinations, yielding $81$ discrete combos, while the high-level action vocabulary has $|\mathcal A|=40$ classes [2606.17730]. The same user action enters two conditioning streams: a symbolic stream encoded by a frozen UMT5 and, for camera motion, a geometric stream encoded as a Plücker-ray FiLM signal [2606.17730]. The interface is therefore multimodal even before prediction begins.

GUI-oriented work exposes still other control regimes. ViMo models a discrete app action space that includes taps, swipes, and typing, and learns an approximate transition $f:S\times A\to S$ over GUI images [2504.13936]. “How Mobile World Model Guides GUI Agents?” compares four downstream-facing interfaces for the same transition prediction problem: delta text, full text, diffusion-based images, and renderable code [2605.10347]. In that formulation, the interface is not only the action representation but also the representation of the predicted future state. A short natural-language delta can serve as a robust semantic feedback channel, whereas code can be rendered back into an image and also inspected as structured output [2605.10347].

Robotic interfaces introduce another layer of heterogeneity. The World-Language-Action model takes instruction $\ell$, recent visual observations $o_t$ and optionally $o_{t-h}$, proprioceptive state $q_t$, and a memory buffer of past subtasks, then autoregressively predicts a window of textual subtasks, a compact physical-dynamics vector $h_t$, and an $n$-step action chunk [2606.05979]. The World Expert uses $h_t$ to imagine a future static frame, and the Action Expert uses $(h_t,q_t)$ to synthesize the action chunk [2606.05979]. A plausible implication is that world-model interfaces are becoming increasingly factorized: one channel carries semantic intent, another physical dynamics, and another executable control.

## 4. Planning, querying, and intervention

The practical importance of a world-model interface becomes most visible in closed-loop planning. World-in-World organizes planning into a three-stage cycle at each time step: Proposal, Simulation, and Revision and Execution [2510.18135]. A proposal policy samples $M$ candidate action-sequence plans, the world model rolls out predicted futures, and a revision policy or score-and-select operator chooses the best plan:
$$
m^\star=\arg\max_m S(\hat A_t^{(m)},\hat o_{t+1:t+H}^{(m)}\mid o_{1:t},g),\qquad a_t=\hat a_{t+1}^{(m^\star)}.
$$
This interface is intentionally model-agnostic: any model that can accept the controller’s control input and return simulated observations can participate in closed-loop planning [2510.18135].

ViMo embeds the same logic into GUI agents. Given horizon $H$, the agent rolls out candidate action sequences,
$$
\hat s_{t+1}^k=f(s_t,a_t^k),\quad \hat s_{t+2}^k=f(\hat s_{t+1}^k,a_{t+1}^k),\ \dots,
$$
and scores them against the goal via an LLM or learned reward model, with beam search used as a concrete planning procedure [2504.13936]. Here the interface must preserve sufficient visual detail for action readiness, not merely coarse semantic plausibility.

Stable-worldmodel-v1 exposes planning even more explicitly. Its `CEMSolver`, `MPPI`, `SGD`, and `Adam` solvers all consume a world-model instance plus a `PlanConfig`, and return `solve(initial_z, goal, **kwargs) → action_sequence` [2602.08968]. Because `world.evaluate` and `world.evaluate_from_dataset` are also standardized, the same interface supports online MPC, offline zero-shot evaluation, and controlled factor-of-variation studies [2602.08968].

Some systems make intervention itself part of the interface. VDAWorld routes an external query such as “Apply force $F=(5,0,0)$ to object #3 at $t=0.5$s.” into generated Python, for example by injecting `self.engine.apply_force(body_id=3, force=(5,0,0))` inside `update_simulation` [2512.11061]. Because the world model is exposed as transparent Python code, arbitrary interventions—changing gravity, editing object shapes, redefining cellular automaton rules—can be encoded as small edits to the simulator, and re-running the edited code immediately yields new simulation results [2512.11061]. WorldAct provides a comparable intervention surface in 3D through APIs such as `get_objects()`, `set_pose(obj_id, pose)`, `remove(obj_id)`, and `insert(obj_asset, pose)`, enabling pick-and-place, rearrangement, or navigation tasks in reconstructed scenes [2605.15843].

Web World Models provide a different intervention model: actions deterministically update the physics state via ordinary web code, after which imagination is regenerated from the updated typed state [2512.23676]. This suggests that the same high-level interface pattern—observe, update structured state, imagine consequences, select action—can be realized either by learned latent transitions or by deterministic code-backed world engines.

## 5. Memory, modularity, and reusable protocols

Long-horizon behavior requires interfaces for memory as well as transition. ActWorld argues that interactive world models suffer from an action-forgetting pathology when recency-biased compression discards event-transition frames [2606.17730]. Its response is a hierarchical action-aware memory interface with three components: Event-Aware Frame Re-assignment, an Action-Conditioned History Amplifier, and a Persistent Action-Aware Memory Bank [2606.17730]. The importance score for a past chunk is
$$
w_k=\lambda_\varphi\,\varphi(y_k^{\rm int},y_k^{\rm ph})+\lambda_r e^{-(t-k)/\tau},
$$
with $\lambda_\varphi=1.0$, $\lambda_r=0.1$, and $\tau=4$ chunks, and the persistent bank stores at most $K_{\rm tot}=16$ tokens with pinning of interaction frames [2606.17730]. Memory here is not a generic cache; it is an interface specialized to causal interaction state.

BRICKS-WM treats modularity itself as an interface problem. After source-task training, it freezes the background RSSM core and reuses it across agents, re-initializing only the Agent RSSM, the interface policy, the agent slot query, and task-specific heads [2606.16489]. To accommodate protocol shifts in the interface code $z$, it inserts a zero-initialized residual adapter
$$
\tilde z=z+\mathrm{MLP}(z),
$$
so the frozen background initially sees the same interface it was trained on [2606.16489]. Proposition 4.6 states that if the conditional interface distribution matches between source and target agents, the frozen background dynamics produce valid transitions in the new setting [2606.16489]. Reusability is thus reduced to protocol matching.

Web World Models approach modularity through typed interfaces and deterministic generation. Physics objects and imagination objects are defined in TypeScript or JSON Schema, runtime validation enforces structural validity, and hash-based seed pinning yields object permanence without storing every state explicitly [2512.23676]. This is a software-engineering answer to the same problem addressed by latent bottlenecks in BRICKS-WM: how to keep a world open-ended while preserving stable contracts.

OpenWorldLib and stable-worldmodel-v1 generalize this concern into reusable research infrastructure. OpenWorldLib standardizes `BaseOperator`, `BaseReasoning`, `BaseSynthesis`, `BaseRepresentation`, `BaseMemory`, and `BasePipeline`, with dynamic loading from manifests and uniform `Dict[str,Tensor]`-style message passing [2604.04707]. stable-worldmodel-v1 standardizes environment wrappers, policy attachment, data collection, factors of variation, and planner/model integration under a single Python-based API [2602.08968]. A plausible implication is that, as world models diversify, software-level interface stabilization becomes a prerequisite for meaningful comparison and composition.

## 6. Evaluation, benchmarks, and recurrent design tensions

Evaluation results across recent systems indicate that interface design affects downstream success as much as raw generative quality. World-in-World makes this point most directly: its benchmark prioritizes task success over open-loop visual quality and reports three central findings—visual quality alone does not guarantee task success, controllability matters more; scaling post-training with action-observation data is more effective than upgrading the pretrained video generators; and allocating more inference-time compute substantially improves closed-loop performance [2510.18135]. It also fits an embodied scaling law,
$$
L(N,D)=\alpha N^{-\beta}+\gamma D^{-\delta}+\epsilon,
$$
with $\beta\approx 0.2$ and $\delta\approx 0.15$ for Active Recognition success rate [2510.18135].

ViMo evaluates its GUI interface by GUI consistency via DINO-feature cosine similarity, instructional accuracy by an LLM judge, and action readiness; it reports harmonic mean $s_h=0.7605$, a $+29.14\%$ relative gain over baselines [2504.13936]. In multi-step trajectory synthesis it attains $57.9\%$ at $T+1$ versus $52.6\%$ for the best vision baseline, and augmenting T3A and M3A with ViMo improves single-step action accuracy from $43.13\%\to 49.20\%$ and $46.01\%\to 50.16\%$, respectively [2504.13936]. These numbers tie interface quality to agent utility rather than to screenshot realism alone.

WorldAct similarly evaluates whether decomposition interfaces support interaction without collapsing scene quality. On its six-scene benchmark it achieves $78.8\%$ Interactable Object Recall overall versus $23.8\%$ without agent, while world-level fidelity in user study drops only marginally from $4.41\to 4.13$ overall and object-level quality rises from $2.49\to 3.75$ [2605.15843]. VDAWorld uses $L_{\rm IoU}$ and PhysicsIQ metrics—Spatial IoU, Weighted-Spatial IoU, and Spatiotemporal IoU—and, for Game of Life, computes per-frame $F_1$ scores [2512.11061]. These evaluations reflect the fact that simulator-facing interfaces are judged by physical and logical consistency as much as by appearance.

ActWorld evaluates an interaction-heavy video interface on I-Bench and reports `Subject-Consistency` $0.871$, `Background-Consistency` $0.896$, `Motion-Smoothness` $0.991$, `IF` $2.557$, `Succ. (%)` $57.8$, `≥2 (%)` $84.5$, and `Acc_full (%)` $20.62$ [2606.17730]. In user study it ranks first on `Action-Following`, `Key/Mouse-Following`, and `Overall Quality`, with `Action-Follow` $4.05$ versus the next best $2.68$ [2606.17730]. The evaluated object is not merely a generated video but a control-sensitive interface with both navigation and object interaction.

GUI-agent studies expose a different tension: high-fidelity structured output is not always the most robust interface at execution time. “How Mobile World Model Guides GUI Agents?” reports that delta text reaches Overall $\approx 12.53$ on MobileWorldBench, whereas full text reaches Overall $\approx 8.07$, while renderable code performs strongly in-distribution on Code2WorldBench with $S_{ad}=96.5$, $S_{id}=94.6$, $S_{ele}=79.1$, $S_{lay}=81.4$, Overall $\approx 85.4$ [2605.10347]. On AndroidWorld, adding delta-text guidance improves end-to-end success rate from $40.1\%\to 44.6\%$ for Qwen3-VL-8B (M3A), from $50.6\%\to 66.6\%$ for Gemini-3-Flash, and from $56.0\%\to 63.8\%$ for GPT-5.4 [2605.10347]. The same study also reports very low mean action entropy, $0.08$–$0.15$, which limits the gains of posterior self-reflection [2605.10347]. This suggests that a world-model interface may function more effectively as prior perception or training supervision than as a universal post-hoc verifier.

The survey literature characterizes these findings as part of a broader evaluation problem: perceptual metrics, task-performance metrics, and physics or consistency benchmarks often diverge, and fragmented evaluation remains a persistent challenge [2606.00133]. Across current systems, the recurring lesson is that world-model interfaces are not neutral wrappers around a learned dynamics model. They determine which variables are controllable, which interventions are expressible, which histories remain accessible, and which downstream tasks can treat the model as a usable world rather than as a visually impressive but operationally opaque predictor.

Source: https://www.emergentmind.com/topics/world-model-interfaces