---
title: 'EchoWM: Generative World Model'
url: https://www.emergentmind.com/topics/echowm
type: topic
---

# EchoWM: Generative World Model

EchoWM is an omnimodal world model for enterable generative media that responds to continuous navigation while jointly generating 720p video, environmental sound, music, and speech. It represents user interaction through camera intent, expressed as a calibrated metric-scale relative 6-DoF trajectory, and supports both first-person observer motion and learned third-person camera–character dynamics. Its design combines a complementary data engine, progressive audiovisual and control training, relative geometric conditioning, and autoregressive post-training for long-horizon causal generation [2608.23189].

## 1. Problem formulation and interaction model

EchoWM models an interactive generated environment that can be continuously traversed while producing synchronized audiovisual feedback. Its intended interaction scope is navigation and viewpoint evolution rather than arbitrary agent behavior. It supports camera translation, rotation, orbiting, and related viewpoint changes, but does not explicitly represent actions such as attacking, jumping, grasping, or manipulating objects.

Given optional media context $\mathcal{M}$, structured text condition $\mathcal{C}$, and a user control sequence $U_{1:T}$, the system converts the controls into a calibrated relative 6-DoF trajectory $P_{1:T}$ and models

$$
p_{\theta,\phi}\left(V_{1:T},A_{1:T}\mid\mathcal{M},\mathcal{C},P_{1:T}\right),
$$

where $V_{1:T}$ and $A_{1:T}$ denote generated video and audio, $\theta$ denotes the joint audiovisual generator, and $\phi$ denotes the trajectory-conditioning branch. Media context can be empty, a reference image, or a synchronized audiovisual prefix:

$$
\mathcal{M}\in\left\{\varnothing,I_{\mathrm{ref}},(V_{1:\tau},A_{1:\tau})\right\}.
$$

Text is divided into persistent and temporal information. Persistent fields describe the scene, style, viewpoint, and subject. Temporal fields describe narrative, speech, and sound. During trajectory-conditioned training, motion-bearing narrative text is removed so that navigation must be inferred from the trajectory rather than from textual descriptions of camera movement.

### Camera intent

In first-person scenes, camera intent represents observer ego-motion. The supplied trajectory specifies translations and rotations of the observer, including forward, lateral, vertical, and rotational motion.

In third-person scenes, camera intent does not specify an explicit character trajectory or camera-rig controller. Instead, the model learns from data how camera position, camera orientation, visible character motion, camera–character distance, orbiting, tracking, framing, and subject visibility evolve together. The same geometric input can consequently produce different realizations according to scene context:

$$
\text{camera intent}\longrightarrow
\begin{cases}
\text{observer motion}, & \text{first person},\\
\text{camera--character--world evolution}, & \text{third person}.
\end{cases}
$$

The initial viewpoint field specifies whether the scene is first- or third-person and describes the initial observer–subject relationship. The trajectory specifies subsequent camera evolution. No explicit subject path, embodiment label, camera rig, or viewpoint-specific controller is passed to the generator.

## 2. Relative trajectory representation

### Camera poses and relative motion

Let $T_t\in\mathrm{SE}(3)$ denote the camera-to-world pose at frame $t$. Relative to the initial pose $T_0$, the system uses

$$
\Delta T_t=T_0^{-1}T_t.
$$

This removes dependence on an arbitrary global coordinate frame while preserving relative translation and rotation. A relative pose is written as

$$
\Delta T_{i,k}=
\left[
\Delta\mathbf{R}_{i,k}\mid\Delta\mathbf{t}_{i,k}
\right],
$$

where $\Delta\mathbf{R}_{i,k}\in\mathrm{SO}(3)$ and $\Delta\mathbf{t}_{i,k}\in\mathbb{R}^3$.

### Discrete and continuous controls

For a discrete keyboard or command state $\mathbf{u}_t$, a fixed mapping $M$ produces a local 6-DoF increment:

$$
\boldsymbol{\xi}_t=M\mathbf{u}_t,\qquad
\boldsymbol{\xi}_t=
\begin{bmatrix}
\mathbf{v}_t\\
\boldsymbol{\omega}_t
\end{bmatrix},
$$

where $\mathbf{v}_t$ is translation and $\boldsymbol{\omega}_t$ is rotation in the current camera frame. The increments are integrated on $\mathrm{SE}(3)$:

$$
T_t=T_{t-1}\operatorname{Exp}\left(\widehat{\boldsymbol{\xi}_t}\right).
$$

The relative trajectory is then $\Delta T_t=T_0^{-1}T_t$. For continuous camera-pose sequences, command-to-pose mapping and integration are bypassed; poses are converted directly into the same coordinate convention and relative representation.

The model is therefore conditioned on geometric trajectories rather than raw keyboard labels. This permits trajectories recovered from ordinary videos to share a representation with trajectories generated from discrete commands or continuous poses.

### Dataset-level translation calibration

Relative coordinates remove global-frame ambiguity but do not resolve metric-scale differences between sources. For clip $i$, EchoWM defines

$$
m_i=\max_k\left\|\Delta\mathbf{t}_{i,k}\right\|_2
$$

and computes a global translation scale using the 90th percentile:

$$
s_{\mathrm{global}}
=
Q_{0.9}\left(\{m_i\}_{i\in\mathcal{D}_{\mathrm{train}}\right).
$$

Trajectories whose maximum displacement exceeds $s_{\mathrm{global}}$ are removed rather than clipped. For retained trajectories,

$$
\widehat{\mathbf{t}}_{i,k}
=
\frac{\Delta\mathbf{t}_{i,k}}{s_{\mathrm{global}}},
\qquad
P_{i,k}
=
\left[
\Delta\mathbf{R}_{i,k}\mid\widehat{\mathbf{t}}_{i,k}
\right].
$$

The same global scale is used during training and inference. This avoids per-clip normalization, which would erase differences in motion magnitude and potentially introduce velocity discontinuities at continuation boundaries.

The control-clean subset has a broader motion distribution than the speech-present audiovisual subset. Control-clean trajectories have mode $4.5$ and median $12.7$, whereas speech-present audiovisual trajectories have mode $0.5$ and median $3.9$. These statistics motivate a single robust global scale across heterogeneous sources.

## 3. Relative UCPE and model architecture

EchoWM is built on a pretrained joint audiovisual diffusion transformer, described in the post-training configuration as an LTX-2.3-based audiovisual generator. The full layer count, parameter count, latent downsampling factors, detailed audio and video encoders, and decoder implementations are not disclosed.

### Relative UCPE camera encoding

A Plücker representation describes a ray using direction and moment:

$$
\mathcal{P}_{t,s}
=
(\mathbf{d}_{t,s},\mathbf{m}_{t,s}),
\qquad
\mathbf{m}_{t,s}
=
\mathbf{o}_t\times\mathbf{d}_{t,s}.
$$

Because the moment depends on the camera origin, this representation is sensitive to absolute coordinates. EchoWM instead uses relative UCPE. For a calibrated camera-to-reference transform $P_t=[\mathbf{R}_t\mid\mathbf{o}_t]$ and intrinsic matrix $K_t$, the ray direction for homogeneous image coordinate $\mathbf{p}_s$ is

$$
\mathbf{d}_{t,s}
=
\mathbf{R}_t\operatorname{norm}
\left(K_t^{-1}\mathbf{p}_s\right).
$$

A local ray frame is formed from the ray direction, a down direction, and their cross product. The resulting ray-to-reference transform is inverted to obtain $\mathbf{D}_i$. For a head width $d$, half the channels use a tiled ray transform,

$$
\mathbf{G}_i
=
\mathbf{I}_{d/8}\otimes\mathbf{D}_i,
$$

while the remaining channels use spatiotemporal RoPE. Camera attention applies these transformations to queries, keys, and values, producing interactions that depend on relative geometry through

$$
\mathbf{G}_i\mathbf{G}_j^{-1}.
$$

A global rigid transformation does not alter pairwise geometry:

$$
(GT_i)^{-1}(GT_j)=T_i^{-1}T_j.
$$

The camera-attention branch operates in parallel with video self-attention and is merged through a zero-initialized projection. Branch-specific QKV and output projections constitute the principal trajectory-conditioning parameters $\phi$. Only video tokens receive the direct trajectory condition. Audio is influenced indirectly through the audiovisual backbone and video–audio cross-modal pathways; there is no explicit action-to-audio controller.

### Audiovisual pathways

The documented architecture contains video and audio latent streams, text conditioning, reference image or audiovisual context, and calibrated trajectory segments. Video self-attention, audio self-attention, audio-to-video attention, video-to-audio attention, and UCPE camera attention are temporally indexed attention operations. During autoregressive training, the same causal visibility pattern is applied to these operations, while text cross-attention remains globally available.

The model jointly generates environmental sounds, interaction effects, background music, and character or narrator speech. Audio is generated within the audiovisual model rather than added afterward as Foley post-processing. Its trajectory dependence is indirect: the trajectory enters the video pathway, and audio receives information through learned audiovisual coupling.

### Synchronization

Audiovisual synchronization is structurally supported through temporally aligned video and audio macro-chunks, shared diffusion noise levels, joint prediction of audio and video flow velocities, consistent causal attention, synchronized continuation contexts, and synchronized tail windows or persistent key–value caches.

For modality $m\in\{v,a\}$ and chunk $i$, clean latent $x_i^m$ is noised using a shared $\sigma_i$:

$$
x_{\sigma_i,i}^{m}
=
(1-\sigma_i)x_i^m+\sigma_i\epsilon_i^m,
\qquad
\epsilon_i^m\sim\mathcal{N}(0,I).
$$

The shared noise level places paired audio and video chunks at the same diffusion stage.

## 4. Data engine and progressive training

EchoWM uses four complementary data sources: internally collected gameplay, human-played Internet gameplay, Unreal Engine renders, and general Internet video. Their purposes differ: controlled gameplay provides action logs and metadata; human-played recordings provide natural timing and speech; Unreal Engine supplies ground-truth metric poses; and general Internet video broadens real-world, cinematic, acoustic, and camera–subject diversity.

Internally collected gameplay includes first- and third-person play, open-world exploration, urban driving, and equestrian traversal. Recordings contain high-resolution RGB video, native stereo game audio, action logs, and environment metadata. Unreal Engine data provides synchronized frames, keyboard states, metric camera poses, intrinsics, and scene metadata. General Internet videos are used for broader appearance and sound, with only a smaller subset entering trajectory training because reliable pose reconstruction is required.

The data engine separates an audiovisual path from a geometry path. Shot-boundary detection isolates continuous audiovisual spans. For geometry, approximately one-minute continuous windows are reconstructed before shorter training clips are sliced from a common reconstruction. Rotations are interpolated with SLERP and translations linearly interpolated. Video, audio, poses, intrinsics, and available action streams are sliced using identical temporal boundaries.

Non-Unreal trajectories are recovered with ViPE using VGGT-$\Omega$ for temporally coherent long-sequence geometry, MoGe-2 for per-frame metric-depth constraints, and per-frame intrinsic optimization. Filtering considers reconstruction reliability, intrinsic stability, temporal smoothness, motion plausibility, perceptual quality, aesthetics, exposure, brightness, audio validity, and loudness statistics.

Structured annotations produced by Qwen3-Omni and Gemini-3-Pro use the canonical fields `scene`, `style`, `viewpoint`, `subject`, `narrative`, `speech`, and `sound`. These fields are sampled differently by training stage to prevent motion leakage.

The training mixtures are:

- **AV-rich mixture**: high-quality gameplay and Internet video, including speech-rich examples; reliable trajectories are not required.
- **Control-clean mixture**: gameplay, Unreal Engine, and pose-eligible Internet video with reliable smooth trajectories; narrative and motion-dependent text are removed.
- **Balanced high-quality mixture**: a smaller intersection containing both reliable motion and strong audiovisual quality; used for final joint fine-tuning.

The control-clean trajectory analysis contains 28,605 sampled trajectories, including straight, diagonal, return-loop, and orbiting paths.

### Progressive training

EchoWM uses three principal stages.

**Audiovisual continued pretraining** trains the full backbone $\theta$ on the AV-rich mixture. The model receives empty context, a reference frame, or a synchronized audiovisual prefix, while full text annotations are retained. Conditioned regions are excluded from the denoising loss. This stage establishes audiovisual appearance, motion, sound, music, and speech priors.

**Action-SFT** freezes the backbone and trains only camera-branch parameters $\phi$ on the control-clean mixture. The retained condition is

$$
\mathcal{C}_{\mathrm{static}}
=
\{c_{\mathrm{scene}},c_{\mathrm{style}},c_{\mathrm{view}},c_{\mathrm{subject}}\}.
$$

Narrative, speech, and sound descriptions are removed, and the audio loss is disabled. Reference-frame conditioning is sampled more frequently than unconditional generation, making the reference define the initial state and the trajectory define subsequent evolution.

**Joint fine-tuning** updates both $\theta$ and $\phi$ on the balanced high-quality subset at a reduced learning rate. Static text, speech, and sound are retained, while narrative remains excluded. This stage consolidates trajectory control and audiovisual generation.

## 5. Autoregressive post-training and inference

The pretrained generator is bidirectional and multi-step. EchoWM converts it into a causal streaming model through audiovisual teacher forcing, short-horizon Self-Gradient Forcing (SGF), long-horizon SGF, and Distribution Matching Distillation (DMD).

### Chunk-level causal training

Video and audio are divided into aligned macro-chunks. For chunk $i$, the noisy query sees clean history from earlier chunks and its own noisy tokens, but not the current clean target or future chunks. The model jointly predicts video and audio flow velocities under the trajectory prefix available to that chunk.

Teacher forcing alone trains on ground-truth histories, whereas inference uses generated histories. SGF addresses this discrepancy with two forwards. The first performs a no-gradient autoregressive rollout and records noisy states and generated clean latents. The second treats generated clean latents as stop-gradient inputs while recomputing causal context representations differentiably. This allows losses on later chunks to propagate through reconstructed attention and KV representations without backpropagating through the serial denoising process.

DMD uses frozen real scores and trainable fake scores. The fake-score model is updated repeatedly using detached generated samples and a flow-matching objective.

### Long-horizon SGF

Long-horizon training chains multiple rollout segments. The terminal frame of one segment initializes the next, retaining accumulated errors rather than resetting them. To prevent unbounded KV-cache growth, EchoWM uses a sink-plus-FIFO cache containing persistent sink chunks and recent chunks. Direct attention fan-in is bounded by the number of sink and recent chunks regardless of total rollout length.

The method argues that retained upper-layer representations can encode information from earlier layers and intermediate chunks, giving gradients an effective temporal reach longer than the direct cache window. DMD score evaluation remains restricted to the temporal range of the original bidirectional model, while causal inference uses the bounded cache.

### Continuation and streaming

For bidirectional multi-turn continuation, the previous synchronized audiovisual output is re-encoded through a tail window:

$$
\mathcal{M}^{(j-1)}
=
\operatorname{Tail}_{w}
\left(V^{(j-1)},A^{(j-1)}\right).
$$

A new segment is generated from this context together with a new condition and trajectory. In streaming mode, persistent video, audio, and camera-attention KV caches are maintained. When local tokens are discarded, temporal RoPE and camera encodings are rebased over the retained cache.

The interaction loop receives a reference image, audiovisual prefix, or empty context; accepts static world description and viewpoint semantics; receives discrete commands or continuous poses; converts commands to local twists; integrates and calibrates relative poses; serializes trajectory segments; injects trajectory information into video attention; generates synchronized audiovisual chunks; and appends them to the tail context or streaming cache.

## 6. Evaluation and reported results

EchoWM is evaluated on WBench Navigation, SANA-WM-Bench, and a self-built WMB user-study benchmark. WBench Navigation contains 158 cases. SANA-WM-Bench uses 80 scenes per split with 241-frame short-horizon and 961-frame long-horizon protocols. WMB contains 200 cases: 100 game scenes, 50 humanoid-robot scenes, 20 natural outdoor scenes, 15 household indoor scenes, and 15 driving scenes. It includes 132 third-person and 68 first-person cases.

### Navigation and trajectory following

On WBench Navigation, EchoWM obtains an overall Average of 81.7, with Quality 81.5, Setting 79.4, Interaction 87.2, Consistency 89.8, and Physical 70.6. It achieves the best overall Average and Consistency score among the listed systems, while EchoWM-Flash achieves the best Interaction score at 87.9.

On the 241-frame SANA-WM-Bench protocol, EchoWM obtains Simple VBench 83.91 and Hard VBench 83.96. It has the best visual quality, translation error, and CMC on simple trajectories, while SANA-WM has slightly better rotation error. On hard trajectories, SANA-WM has lower pose errors, but EchoWM retains the best visual quality.

On the 961-frame undistilled protocol, EchoWM’s simple-split rotation error grows from $0.523^\circ$ at 241 frames to $3.223^\circ$ at 961 frames. On the hard split it reaches $12.05^\circ$. These results show that accumulated pose drift remains a significant long-horizon weakness.

EchoWM-Flash is the four-step causal streaming variant. On the 961-frame protocol it obtains the highest VBench score and strongest trajectory metrics among the evaluated causal systems. The paper cautions that a small $\Delta IQ$ does not necessarily imply better long-horizon quality when absolute quality remains low.

### User studies and qualitative behavior

Against LingBot-World-v2, EchoWM receives 46.50% overall-preference votes versus 21.57%. Its preference rates are 34.00% for semantic following, 32.64% for spatial-temporal consistency, and 32.50% for visual aesthetics.

Against HappyOyster, EchoWM receives 63.13% overall-preference votes versus 27.06%, 57.69% for spatial-temporal consistency, and 52.88% for visual aesthetics. Semantic-following preference is weaker in this comparison: EchoWM is preferred in 13.63% of cases versus 29.44% for HappyOyster, while 51.50% are judged “both good.” In embodied-robot and natural-outdoor subsets, EchoWM’s semantic-following preference is 8.67% and 2.50%, respectively.

Qualitative findings report that EchoWM maintains first-person semantics instead of switching to an exocentric view, preserves third-person subjects during orbiting, maintains smoother camera–subject relationships, produces recognizable 3D scene layouts under downstream reconstruction, generalizes qualitatively to unseen robot and 2D/2.5D game domains, generates environmental sound, music, and speech jointly with video, and supports return-to-viewpoint tests over extended trajectories.

No numerical audio-quality, speech-synchronization, music-synchronization, or lip-sync scores are reported. The supplied results therefore provide stronger evidence for visual interaction and trajectory following than for explicit audio controllability.

## 7. Limitations and research significance

EchoWM’s interaction space is limited to navigation and viewpoint-related actions. It does not provide explicit semantics for jumping, attacking, object manipulation, hand or robot commands, or deterministic game-state transitions.

The system has no explicit persistent 3D scene graph or world-state database. It uses audiovisual context and bounded KV memory, so repeated continuation can exhibit geometry drift, subject-identity drift, changing world state, inconsistent audio, and imperfect revisit behavior. The long-horizon evaluation confirms that pose drift increases with rollout duration, particularly for hard trajectories.

The global trajectory filter restricts training to a retained displacement range. Performance outside that range—including very large translations, high speeds, unusual rotations, or atypical camera motions—is not established. Recovered Internet-video trajectories remain susceptible to motion blur, zoom, stabilization, cropping, incorrect intrinsics, reconstruction failure, and scene-dependent estimation bias.

Third-person camera–character behavior is learned from data rather than governed by a camera rig, collision system, or explicit character controller. This permits subject-agnostic behavior but can produce subject loss, implausible orbiting, inconsistent distance, camera–subject decoupling, and physically implausible motion.

Audio controllability is indirect because trajectory conditioning enters only the video branch. Consequently, strong visual navigation results do not demonstrate that every footstep, collision, or acoustic event is causally correct. Exact training and inference costs, total model size, throughput, memory consumption, and device-level efficiency are not reported. Four-step causal sampling reduces the sampling burden in EchoWM-Flash, but no formal throughput or memory comparison is provided.

The central contribution of EchoWM is the combination of camera intent, relative metric-scale 6-DoF trajectories, relative UCPE conditioning, a complementary data engine, progressive training, and causal SGF/DMD post-training. The reported evidence supports a world model capable of interactive first-person and third-person visual generation with jointly produced audio, while also demonstrating unresolved limitations in long-horizon geometric stability, explicit physical consistency, persistent world memory, audio controllability, and quantified computational efficiency.

Source: https://www.emergentmind.com/topics/echowm