WorldKV: Persistent-World Generation

• WorldKV is a training-free framework for persistent-world generation in autoregressive video diffusion models that achieves revisit consistency without incurring linear memory growth.
• The framework introduces World Retrieval to selectively reinsert compressed historical KV-cache chunks, thereby maintaining scene fidelity and improving throughput.
• World Compression prunes redundant tokens within each frame chunk, enabling broader historical coverage with nearly a 2× reduction in per-chunk storage.

WorldKV is a training-free framework for persistent-world generation in autoregressive video diffusion models. It addresses the tension between long-horizon scene consistency and real-time inference by augmenting native KV-cache attention with two mechanisms: World Retrieval, which stores evicted KV-cache chunks and selectively reinserts scene-relevant history, and World Compression, which prunes redundant tokens within each chunk via key-key similarity to an anchor frame. The method is designed for settings in which revisiting a previously seen viewpoint should reproduce consistent static appearance and scene layout—termed revisit consistency—without incurring the linear memory growth and quadratic attention cost of full-KV inference (Yi et al., 21 May 2026).

1. Persistent-world generation and the scaling problem

In the formulation used by WorldKV, an autoregressive video diffusion world model generates a video $x^{1:N}=(x^1,\dots,x^N)$ conditioned on camera or action inputs $a^{1:N}=(a^1,\dots,a^N)$ as

$$p(x^{1:N}\mid a^{1:N}) =\prod_{t=1}^{N}p\bigl(x^t\mid x^{<t},\,a^{\le t}\bigr).$$

The conditioning on prior frames $x^{<t}$ is implemented through a causal Transformer with a key-value cache. At each denoising step, new keys and values are appended to the cache. Within this setting, a persistent world requires that if a camera returns at time $t_2>t_1$ to essentially the same pose used at $t_1$, then the generated frame should match the earlier frame in static appearance and scene layout. WorldKV defines this property as revisit consistency (Yi et al., 21 May 2026).

The core systems issue is the scaling behavior of full-KV attention. If each frame or chunk contributes $T$ keys and values, then after $M$ frames the cache contains $M\cdot T$ tokens, with memory footprint $O(M\,T\,d)$, where $a^{1:N}=(a^1,\dots,a^N)$0 is the hidden dimension. Causal attention over the entire cache has cost

$a^{1:N}=(a^1,\dots,a^N)$1

per layer per head. As rollout length increases, VRAM usage grows linearly and attention cost grows quadratically. The result is eventual GPU memory exhaustion and degraded frame rate.

A standard mitigation is sliding-window inference, which keeps only the most recent $a^{1:N}=(a^1,\dots,a^N)$2 frames. This bounds memory by $a^{1:N}=(a^1,\dots,a^N)$3 and attention cost by $a^{1:N}=(a^1,\dots,a^N)$4, restoring throughput. However, once keys are evicted from the active window, the model loses direct access to earlier viewpoints, and revisits are prone to hallucination or drift. WorldKV is explicitly positioned between these two extremes: it preserves bounded active attention while retaining access to a larger historical memory.

2. World Retrieval

World Retrieval modifies the eviction path of sliding-window inference. Instead of discarding the oldest chunk after the active window exceeds capacity, the method compresses that chunk and moves it into a secondary store in GPU or CPU memory. Each stored chunk is indexed by its generation state $a^{1:N}=(a^1,\dots,a^N)$5, defined as an absolute camera pose or an accumulated action embedding (Yi et al., 21 May 2026).

At inference time, the current state $a^{1:N}=(a^1,\dots,a^N)$6 is compared with the stored history $a^{1:N}=(a^1,\dots,a^N)$7 using a relevance score

$a^{1:N}=(a^1,\dots,a^N)$8

The system retrieves the top-$a^{1:N}=(a^1,\dots,a^N)$9 chunks ranked by this score:

$$p(x^{1:N}\mid a^{1:N}) =\prod_{t=1}^{N}p\bigl(x^t\mid x^{<t},\,a^{\le t}\bigr).$$0

In the default camera-action instantiation, similarity is defined as

$$p(x^{1:N}\mid a^{1:N}) =\prod_{t=1}^{N}p\bigl(x^t\mid x^{<t},\,a^{\le t}\bigr).$$1

where $$p(x^{1:N}\mid a^{1:N}) =\prod_{t=1}^{N}p\bigl(x^t\mid x^{<t},\,a^{\le t}\bigr).$$2 and $$p(x^{1:N}\mid a^{1:N}) =\prod_{t=1}^{N}p\bigl(x^t\mid x^{<t},\,a^{\le t}\bigr).$$3 are translation and rotation differences, and $$p(x^{1:N}\mid a^{1:N}) =\prod_{t=1}^{N}p\bigl(x^t\mid x^{<t},\,a^{\le t}\bigr).$$4 normalize them. Appendix C also reports that a query-based similarity function, ranking stored chunks by cross-attention score between the current query and stored keys, works nearly as well.

The retrieved chunks are not re-encoded. Instead, they are inserted directly into the native attention window. The active window is formed as

$$p(x^{1:N}\mid a^{1:N}) =\prod_{t=1}^{N}p\bigl(x^t\mid x^{<t},\,a^{\le t}\bigr).$$5

where $$p(x^{1:N}\mid a^{1:N}) =\prod_{t=1}^{N}p\bigl(x^t\mid x^{<t},\,a^{\le t}\bigr).$$6 is a small fixed sink cache anchoring the initial 3 frames, $$p(x^{1:N}\mid a^{1:N}) =\prod_{t=1}^{N}p\bigl(x^t\mid x^{<t},\,a^{\le t}\bigr).$$7 contains the most recent 3 frames, $$p(x^{1:N}\mid a^{1:N}) =\prod_{t=1}^{N}p\bigl(x^t\mid x^{<t},\,a^{\le t}\bigr).$$8 is the set of retrieved chunks, and $$p(x^{1:N}\mid a^{1:N}) =\prod_{t=1}^{N}p\bigl(x^t\mid x^{<t},\,a^{\le t}\bigr).$$9 is the chunk currently being denoised. Standard Transformer causal attention is then applied over this concatenation. The salient design choice is that retrieval operates on cached keys and values rather than on decoded visual features, so the method preserves compatibility with the backbone’s native attention mechanism.

3. World Compression

World Compression targets redundancy within a short chunk of frames. The method selects the first frame of a chunk as an anchor. If $x^{<t}$0 denotes the anchor-frame keys and $x^{<t}$1 the keys of a non-anchor frame $x^{<t}$2, then the redundancy score of token $x^{<t}$3 in frame $x^{<t}$4 is

$x^{<t}$5

A large score indicates that the token is well represented by the anchor. Pooling all $x^{<t}$6 non-anchor scores, WorldKV keeps only the bottom $x^{<t}$7-quantile of non-anchor tokens—those with the smallest redundancy scores—while retaining all anchor tokens. The resulting number of retained keys is

$x^{<t}$8

The implementation described for WorldKV uses $x^{<t}$9 frames per chunk and $t_2>t_1$0, which yields $t_2>t_1$1, described as a $t_2>t_1$2 reduction in per-chunk storage (Yi et al., 21 May 2026).

This compression rule has two direct consequences. First, uncompressed storage for a 3-frame chunk is $t_2>t_1$3 tokens, whereas compressed storage is $t_2>t_1$4. Second, under a fixed token budget, the system can store roughly $t_2>t_1$5 more chunks. The paper states that, because retrieval selects a fixed number $t_2>t_1$6 of chunks, broader historical coverage can improve revisit consistency. The associated ablations further indicate that moderate compression is preferable to either no compression or overly aggressive compression.

4. Integrated inference pipeline and computational properties

WorldKV operates as an inference-time pipeline layered on top of sliding-window denoising. At each generation step, the next chunk is denoised using causal attention over a window composed of sink, retrieved, recent, and current frames. The oldest recent chunk is then evicted, compressed, and appended to the stored history. Retrieval scores are computed against the current camera or action state, and the top-$t_2>t_1$7 chunks are inserted into the attention window for the next step (Yi et al., 21 May 2026).

The paper contrasts the asymptotic properties of full-KV inference and WorldKV. After $t_2>t_1$8 chunks, full-KV memory is

$t_2>t_1$9

and its per-layer attention cost is

$t_1$0

With compression fraction $t_1$1 and retrieval budget $t_1$2, WorldKV stores approximately

$t_1$3

tokens in history, while the active attention window is bounded by

$t_1$4

for sink, retrieval, recent, and current. The corresponding per-layer attention cost is

$t_1$5

which is independent of rollout length $t_1$6. This is the central systems claim of the method: the stored history can grow over time, but the active attention computation remains bounded.

The implementation details reported in the paper instantiate this design with a sliding-window size $t_1$7 latent frames, partitioned into Sink $t_1$8 frames, Retrieval $t_1$9 frames, Recent $T$0 frames, and Current denoising $T$1 frames. Compression is configured as anchor-only plus $T$2 of non-anchor tokens, giving $T$3 tokens per 3-frame chunk, and the retrieval budget is $T$4 chunks. Experiments are reported on LingBot-World-Fast (14B) using $T$5H200 GPUs, with additional results on $T$6B200, and on Matrix-Game-2.0 (1.3B) using $T$7H200.

5. Empirical evaluation

The evaluation uses 60 manually designed scene-trajectory pairs across indoor, outdoor, game, and AI-gen domains. Each trajectory includes loops, forward-back traversals, and at least one revisit. The reported metrics are PSNR, SSIM, LPIPS, and FID for revisit fidelity, together with FPS for throughput (Yi et al., 21 May 2026).

Two backbone models are used: LingBot-World-Fast (14B), described as long-video distilled with native full-KV support, and Matrix-Game-2.0 (1.3B), described as trained on short clips with a native sliding window. Memory-trained baselines are WorldPlay (8B) and Yume-1.5 (5B).

The main quantitative results show the following. On LingBot-World-Fast, sliding-window inference reports $T$8 FPS, LPIPS $T$9, PSNR $M$0, SSIM $M$1, and FID $M$2. Full-KV reports $M$3 FPS, LPIPS $M$4, PSNR $M$5, SSIM $M$6, and FID $M$7. LingBot-Fast + WorldKV reports $M$8 FPS, LPIPS $M$9, PSNR $M\cdot T$0, SSIM $M\cdot T$1, and FID $M\cdot T$2. On Matrix-Game-2.0, sliding-window inference reports $M\cdot T$3 FPS, LPIPS $M\cdot T$4, PSNR $M\cdot T$5, SSIM $M\cdot T$6, and FID $M\cdot T$7. Full-KV reports $M\cdot T$8 FPS, LPIPS $M\cdot T$9, PSNR $O(M\,T\,d)$0, SSIM $O(M\,T\,d)$1, and FID $O(M\,T\,d)$2. Matrix + WorldKV reports $O(M\,T\,d)$3 FPS, LPIPS $O(M\,T\,d)$4, PSNR $O(M\,T\,d)$5, SSIM $O(M\,T\,d)$6, and FID $O(M\,T\,d)$7.

These numbers support two distinct conclusions stated in the paper. First, WorldKV matches or exceeds full-KV memory fidelity at roughly $O(M\,T\,d)$8 throughput and outperforms sliding-window inference by large margins. Second, on the short-context Matrix model, full-KV actually hurts due to compounding errors, whereas selective retrieval avoids irrelevant KV caches. A common misconception is that simply attending to more historical tokens is always beneficial; the Matrix results provide an explicit counterexample.

Throughput measurements at the last chunk of a 1-minute rollout further emphasize the systems trade-off. For LingBot-Fast, sliding window is approximately $O(M\,T\,d)$9 FPS on H200 and $a^{1:N}=(a^1,\dots,a^N)$00 on B200, full-KV is approximately $a^{1:N}=(a^1,\dots,a^N)$01 and $a^{1:N}=(a^1,\dots,a^N)$02, and WorldKV is approximately $a^{1:N}=(a^1,\dots,a^N)$03 and $a^{1:N}=(a^1,\dots,a^N)$04. For Matrix-Game-2.0 on H200, sliding window is approximately $a^{1:N}=(a^1,\dots,a^N)$05 FPS, full-KV approximately $a^{1:N}=(a^1,\dots,a^N)$06, and WorldKV approximately $a^{1:N}=(a^1,\dots,a^N)$07. The VRAM behavior is also contrasted: full-KV memory grows linearly to out-of-memory by approximately 60 seconds, whereas WorldKV with compression and CPU offload remains nearly flat.

6. Ablations, limitations, and prospective extensions

The ablation studies isolate both compression and retrieval effects. For intra-chunk compression, the paper compares 3-frame chunks compressed to effective sizes of $a^{1:N}=(a^1,\dots,a^N)$08, $a^{1:N}=(a^1,\dots,a^N)$09, $a^{1:N}=(a^1,\dots,a^N)$10, $a^{1:N}=(a^1,\dots,a^N)$11, and $a^{1:N}=(a^1,\dots,a^N)$12 frames retained. On LingBot, the $a^{1:N}=(a^1,\dots,a^N)$13 setting gives LPIPS $a^{1:N}=(a^1,\dots,a^N)$14, PSNR $a^{1:N}=(a^1,\dots,a^N)$15, SSIM $a^{1:N}=(a^1,\dots,a^N)$16, and FID $a^{1:N}=(a^1,\dots,a^N)$17, while $a^{1:N}=(a^1,\dots,a^N)$18 gives LPIPS $a^{1:N}=(a^1,\dots,a^N)$19, PSNR $a^{1:N}=(a^1,\dots,a^N)$20, SSIM $a^{1:N}=(a^1,\dots,a^N)$21, and FID $a^{1:N}=(a^1,\dots,a^N)$22. On Matrix, $a^{1:N}=(a^1,\dots,a^N)$23 gives LPIPS $a^{1:N}=(a^1,\dots,a^N)$24, PSNR $a^{1:N}=(a^1,\dots,a^N)$25, SSIM $a^{1:N}=(a^1,\dots,a^N)$26, and FID $a^{1:N}=(a^1,\dots,a^N)$27, while $a^{1:N}=(a^1,\dots,a^N)$28 gives LPIPS $a^{1:N}=(a^1,\dots,a^N)$29, PSNR $a^{1:N}=(a^1,\dots,a^N)$30, SSIM $a^{1:N}=(a^1,\dots,a^N)$31, and FID $a^{1:N}=(a^1,\dots,a^N)$32. The paper summarizes this as evidence that moderate compression, specifically $a^{1:N}=(a^1,\dots,a^N)$33 or $a^{1:N}=(a^1,\dots,a^N)$34, retains fidelity nearly equal to no compression (Yi et al., 21 May 2026).

For inter-chunk coverage under a fixed 3-chunk window, storing 6 chunks and retrieving 3 with compression outperforms storing 3 chunks and retrieving 3 without compression. On LingBot, the $a^{1:N}=(a^1,\dots,a^N)$35 configuration gives LPIPS $a^{1:N}=(a^1,\dots,a^N)$36, PSNR $a^{1:N}=(a^1,\dots,a^N)$37, SSIM $a^{1:N}=(a^1,\dots,a^N)$38, and FID $a^{1:N}=(a^1,\dots,a^N)$39, compared with LPIPS $a^{1:N}=(a^1,\dots,a^N)$40, PSNR $a^{1:N}=(a^1,\dots,a^N)$41, SSIM $a^{1:N}=(a^1,\dots,a^N)$42, and FID $a^{1:N}=(a^1,\dots,a^N)$43 for $a^{1:N}=(a^1,\dots,a^N)$44. On Matrix, $a^{1:N}=(a^1,\dots,a^N)$45 yields LPIPS $a^{1:N}=(a^1,\dots,a^N)$46, PSNR $a^{1:N}=(a^1,\dots,a^N)$47, SSIM $a^{1:N}=(a^1,\dots,a^N)$48, and FID $a^{1:N}=(a^1,\dots,a^N)$49, compared with LPIPS $a^{1:N}=(a^1,\dots,a^N)$50, PSNR $a^{1:N}=(a^1,\dots,a^N)$51, SSIM $a^{1:N}=(a^1,\dots,a^N)$52, and FID $a^{1:N}=(a^1,\dots,a^N)$53 for $a^{1:N}=(a^1,\dots,a^N)$54. By contrast, $a^{1:N}=(a^1,\dots,a^N)$55 is reported as too aggressive, with degraded fidelity on both models. This supports the paper’s claim that broader historical coverage helps, but over-compression discards too much.

The retrieval-algorithm ablation compares sliding-window inference, query-based retrieval, and camera-based retrieval. On LingBot, query-based retrieval reaches LPIPS $a^{1:N}=(a^1,\dots,a^N)$56, PSNR $a^{1:N}=(a^1,\dots,a^N)$57, SSIM $a^{1:N}=(a^1,\dots,a^N)$58, and FID $a^{1:N}=(a^1,\dots,a^N)$59, whereas camera-based retrieval reaches LPIPS $a^{1:N}=(a^1,\dots,a^N)$60, PSNR $a^{1:N}=(a^1,\dots,a^N)$61, SSIM $a^{1:N}=(a^1,\dots,a^N)$62, and FID $a^{1:N}=(a^1,\dots,a^N)$63. On Matrix, query-based retrieval gives LPIPS $a^{1:N}=(a^1,\dots,a^N)$64, PSNR $a^{1:N}=(a^1,\dots,a^N)$65, SSIM $a^{1:N}=(a^1,\dots,a^N)$66, and FID $a^{1:N}=(a^1,\dots,a^N)$67, while camera-based retrieval gives LPIPS $a^{1:N}=(a^1,\dots,a^N)$68, PSNR $a^{1:N}=(a^1,\dots,a^N)$69, SSIM $a^{1:N}=(a^1,\dots,a^N)$70, and FID $a^{1:N}=(a^1,\dots,a^N)$71. Both retrieval signals outperform sliding-window inference, with camera-action retrieval best.

The paper’s limitations are explicit. Fidelity is ultimately bounded by the pretrained backbone’s generation quality. On multi-minute rollouts beyond training lengths, autoregressive drift still accumulates. CPU offload latency currently prevents pure CPU storage in a real-time loop, and the paper identifies future work on asynchronous DSP transfers. Proposed directions include combining WorldKV with training-based memory modules, improving retrieval signals through learnable key-query encodings or cross-attention memory networks, optimizing host-device offloading latency through NVMe-GPU direct DMA and sparsity-aware caching, and extending the framework to open-world or RL-driven embodied agents where memory can be indexed by semantic queries. A plausible implication is that WorldKV is best understood not as a replacement for learned memory architectures, but as an inference-time systems layer that exploits the latent structure already present in KV caches.