TR-World: Temporal Residual World Model

• The paper introduces a temporal residual framework that isolates dynamic objects via BEV feature differences, avoiding redundant static modeling.
• It employs a Future-Guided Trajectory Refinement module that directly uses predicted future BEV features to improve planning and collision avoidance.
• A sparse spatial-temporal supervision strategy is adopted to preserve multi-timestep dynamic structure and prevent latent world-model collapse.

Temporal Residual World Model (TR-World) is a world-model formulation for end-to-end autonomous driving introduced in "ResWorld: Temporal Residual World Model for End-to-End Autonomous Driving" (Zhang et al., 11 Feb 2026). It shifts future-scene modeling away from the entire bird’s-eye-view (BEV) scene and toward only the dynamic parts of the scene. The central mechanism is the computation of temporal residuals between aligned BEV features across time, so that dynamic-object information can be extracted without relying on detection and tracking. TR-World takes only temporal residuals as input, predicts the future spatial distribution of dynamic objects, and combines that prediction with static object information contained in the current BEV features to obtain future BEV features for planning. In ResWorld, these future BEV features are then coupled to planning through a Future-Guided Trajectory Refinement (FGTR) module (Zhang et al., 11 Feb 2026).

1. Conceptual motivation

The motivation for TR-World is framed as a critique of prior world models for autonomous driving. The paper identifies two main limitations. First, prior methods waste capacity on redundant static-region modeling. In BEV scenes, large regions correspond to road surfaces, buildings, curbs, and other static structures that are essentially preserved across nearby future steps. Explicitly generating these regions again as a future world is therefore redundant. The paper argues that prior world models devote the same effort to static and dynamic content, which weakens their ability to focus on moving agents, even though those agents dominate planning difficulty and safety risk (Zhang et al., 11 Feb 2026).

Second, prior world models are described as having weak interaction with trajectories. In that framing, future scene prediction is often used only as an indirect proxy task: the model predicts future representations, and planning is expected to improve as a side effect. ResWorld instead treats the predicted future scene as an explicit planning cue. The argument is that prior methods do not sufficiently exploit direct interaction between the ego trajectory and the predicted future environment, so the world model cannot directly verify collision risk or drivable-area violations along the planned path (Zhang et al., 11 Feb 2026).

These two claims define the conceptual scope of TR-World. It is not a general-purpose effort to reconstruct a future scene from scratch. Rather, it is a targeted attempt to concentrate model capacity on what changes and to make those predicted changes directly useful to planning.

2. Temporal residual extraction in BEV space

TR-World operates on aligned BEV features obtained from a strong geometric BEV backbone. Let the current BEV feature be $B_t \in \mathbb{R}^{C \times H \times W}$, and let the historical features be $\{B_{t-1}, \dots, B_{t-k}\}$. These historical BEV maps are transformed into the coordinate system of the current frame $B_t$ and then fused as

$$B_{fuse} = \text{Conv}(\text{Concat}(B_t, B_{t-1}, \dots, B_{t-k})) \tag{1}$$

The fused BEV $B_{fuse}$ is then used to generate a spatial attention mask that highlights regions likely to contain changes. This attention is applied to each timestamp’s BEV feature to extract sparse scene queries:

$$S_i = \text{AvgPool}(SA(B_{fuse}) \odot B_i) \tag{5}$$

where $SA(\cdot)$ is the spatial attention map and $\odot$ is elementwise multiplication (Zhang et al., 11 Feb 2026).

The key operation is the temporal residual itself: scene queries from adjacent timestamps are subtracted after spatial alignment. These residuals encode what changed at the same spatial location over time, which the paper identifies as the signal of dynamic objects. In this formulation, dynamics are derived from BEV feature differences rather than from explicit object detection or multi-object tracking. That design choice is central to the definition of TR-World: it isolates dynamic content from scene evolution directly in feature space.

A common misunderstanding is to treat TR-World as a detection-conditioned predictor. The paper states the opposite. Dynamic-object information is extracted without relying on detection and tracking. This makes the residual operation the primary carrier of motion information, rather than an auxiliary refinement on top of an object-centric pipeline (Zhang et al., 11 Feb 2026).

3. Future BEV construction through asymmetric static-dynamic modeling

A defining design choice is that TR-World takes only temporal residuals as input. Each temporal residual $R_i$ is processed with self-attention,

$\hat{R}_i = \text{SelfAttention}(R_i) \tag{6}$

and residuals across timestamps are accumulated into a latent dynamic representation. That representation is then fused back onto the current BEV feature base to predict the future spatial distribution of dynamic objects:

$\{B_{t-1}, \dots, B_{t-k}\}$0

where $\{B_{t-1}, \dots, B_{t-k}\}$1 is the aggregated dynamic representation, TokenFuser expands the sparse dynamic tokens over the BEV grid, and $\{B_{t-1}, \dots, B_{t-k}\}$2 denotes the matrix combination described in the paper (Zhang et al., 11 Feb 2026).

The output $\{B_{t-1}, \dots, B_{t-k}\}$3 is a future BEV feature map that preserves static content from the current BEV coordinate system while injecting predicted dynamic-object motion. This asymmetry is deliberate. Static information is kept in the current BEV feature $\{B_{t-1}, \dots, B_{t-k}\}$4 because, in the current coordinate system, the future static scene can be treated as unchanged. Dynamic information is predicted from temporal residuals only. These two components are merged to form $\{B_{t-1}, \dots, B_{t-k}\}$5.

This architectural choice clarifies what TR-World is and is not. It does not generate a fully new future scene from scratch. Instead, it uses the current BEV as the static scaffold and overlays predicted dynamic evolution on top of it. The paper presents this as a way to avoid redundant modeling of roads and buildings while preserving the information most relevant to planning: the future locations of moving agents and the resulting future drivable situation (Zhang et al., 11 Feb 2026).

4. Future-guided trajectory refinement and collapse prevention

ResWorld augments TR-World with a Future-Guided Trajectory Refinement (FGTR) module so that future BEV features participate directly in planning. The model first predicts a prior trajectory from the current fused BEV scene. It uses waypoint queries $\{B_{t-1}, \dots, B_{t-k}\}$6, where each query corresponds to one future timestep, and decodes a preliminary path as

$\{B_{t-1}, \dots, B_{t-k}\}$7

where $\{B_{t-1}, \dots, B_{t-k}\}$8 are sparse scene queries extracted from $\{B_{t-1}, \dots, B_{t-k}\}$9 (Zhang et al., 11 Feb 2026).

FGTR then makes the prior trajectory interact explicitly with the predicted future BEV feature $B_t$0. The prior trajectory $B_t$1 is used as reference points for deformable attention:

$B_t$2

and the final refined trajectory is decoded by

$B_t$3

The purpose is not described as a minor decoder adjustment. Rather, FGTR lets the prior trajectory query the future environment along the candidate path, so that the planner can inspect whether the ego vehicle would collide with moving objects or drift out of the drivable area, and then correct the plan accordingly (Zhang et al., 11 Feb 2026).

FGTR also provides sparse spatial-temporal supervision on future BEV features. The paper identifies collapse as a common failure mode of latent world models, in which different scenes map to nearly identical future representations. The supervision induced by FGTR is spatially sparse because only a few trajectory-linked reference points probe the future BEV, and temporally sparse because different waypoint queries correspond to different future timestamps. The authors explicitly avoid dense supervision of $B_t$4 with real future frames. They argue that supervising future targets at a single timestamp would destroy richer multi-step temporal structure, whereas sparse interaction encourages preservation of the cross-temporal spatial distribution of dynamic objects across multiple future times. The paper further states that full future supervision can hurt performance (Zhang et al., 11 Feb 2026).

5. Training objective and empirical performance

The training objective is intentionally simple. ResWorld uses only $B_t$5 losses on the prior and final trajectories:

$B_t$6

where $B_t$7 is the ground-truth ego trajectory. The world model itself is not supervised with ground-truth future BEV labels. The paper presents this as a deliberate choice and reports that supervising $B_t$8 with specific future frames reduces performance because it forces the latent representation to commit to one timestamp and loses richer temporal information (Zhang et al., 11 Feb 2026).

ResWorld is evaluated on nuScenes and NAVSIM. On nuScenes, the paper reports open-loop planning metrics consisting of L2 displacement error at 1s/2s/3s and average, and Collision Rate (CR) at 1s/2s/3s and average. The main reported results are the following.

Benchmark setting	Reported result	Evaluation mode
nuScenes, without ego status in planner	L2 Avg: 0.59; CR Avg: 0.17	Open-loop
nuScenes, with ego status in planner	L2 Avg: 0.30; CR Avg: 0.06	Open-loop
NAVSIM, ResWorld	88.3 PDMS	Closed-loop
NAVSIM, ResWorld* with historical frames	89.0 PDMS	Closed-loop

On nuScenes, these results are reported to outperform perception-heavy methods such as ST-P3, UniAD, OccNet, PARA-Drive, and GenAD, as well as perception-free or world-model-based methods such as SSR, LAW, and DiffusionDrive. On NAVSIM, PDMS is built from No At-Fault Collision (NC), Drivable Area Compliance (DAC), Time-to-Collision (TTC), Comfort (Comf.), and Ego Progress (EP), and the reported scores exceed prior baselines including LAW, World4Drive, Hydra-MDP, and DiffusionDrive (Zhang et al., 11 Feb 2026).

The paper also emphasizes that stronger planning performance with ego status indicates that the framework is robust enough not to overfit to ego-state cues alone. This suggests that the future BEV representation contributes information beyond a simple dependence on ego-state inputs.

6. Ablation findings, stated contributions, and practical implications

The ablation studies support the separation between residual world modeling and trajectory refinement. TR-World alone improves planning over a normal world model by focusing on dynamic objects. FGTR alone improves trajectory quality by refining prior paths with future BEV information. TR-World and FGTR together give the best results. The paper also shows that if future BEV features are supervised using real future data, performance drops relative to the sparse-supervision setup. That finding is used to support the claim that the latent future representation should preserve multi-timestamp dynamic structure rather than be forced into a single annotated future frame (Zhang et al., 11 Feb 2026).

The main contributions are stated as five points. They are: a temporal residual formulation for world modeling that isolates dynamic objects from BEV scene representations without detection or tracking; a dynamic-only TR-World that predicts future object motion while avoiding redundant static-scene modeling; a future-guided trajectory refinement module that explicitly couples trajectory planning with future scene predictions; a sparse spatial-temporal supervision strategy that helps prevent world-model collapse while preserving richer future structure; and state-of-the-art planning performance on nuScenes and NAVSIM (Zhang et al., 11 Feb 2026).

The practical implication drawn in the paper is a new paradigm for end-to-end autonomous driving: instead of asking the world model to reconstruct everything, let it concentrate on what changes, then use those predicted changes to refine the ego plan. This suggests a reformulation of world modeling in driving in which latent dynamics, rather than full-scene future synthesis, become the primary planning interface.

7. Relation to implicit residual world models

A closely related residual world-model direction appears in "Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models" (Mei et al., 19 Oct 2025). That paper introduces IR-WM (Implicit Residual World Model), which is described as very close to what is meant by a Temporal Residual World Model: a world model that predicts changes over time rather than reconstructing every future scene from scratch. IR-WM treats the previous BEV state as a strong temporal prior, predicts only the residual evolution, and emphasizes efficiency, better dynamics understanding, and temporal coherence.

The overlap with TR-World is conceptual but not identical. Both approaches move away from full future-scene reconstruction and toward residual temporal updates. Both use BEV latent states and couple future-state prediction to planning. However, IR-WM is formulated as a vision-centric driving-specific world model with BEVFormer as the scene encoder, an autoregressive predictor with streaming memory, an occupancy head for 4D occupancy forecasting, a planning head for trajectory prediction, and an alignment module designed to reduce rollout drift (Mei et al., 19 Oct 2025). By contrast, ResWorld’s TR-World is defined by temporal residuals between aligned BEV features across multiple timestamps, a dynamic-only prediction pathway, and FGTR-based sparse spatial-temporal supervision.

The comparison clarifies the position of TR-World within residual world modeling. A plausible implication is that residual prediction has emerged as a shared response to the inefficiency of reconstructing static backgrounds in driving scenes. ResWorld’s distinctive contribution within that trend is the use of temporal residuals to isolate dynamics without detection or tracking, together with direct future-environment interaction for trajectory refinement.