VideoSSM: Hybrid Memory for Long Videos

• VideoSSM is a long-video generation framework that unifies autoregressive diffusion with a hybrid state-space memory to mitigate error accumulation and motion drift over extended sequences.
• It employs a local sliding-window cache for precise short-term context and a learnable global SSM that compresses and evolves historical information for overall scene coherence.
• Empirical results on VBench show that VideoSSM maintains high subject and background consistency while preserving natural dynamism, outperforming comparable models.

VideoSSM is a long-video generation framework that unifies autoregressive diffusion with a hybrid state-space memory. It models video synthesis as a recurrent dynamical process with two coordinated memory tiers: a local sliding-window cache for precise short-term context and a learnable global state-space model (SSM) that serves as a compressed memory of the entire past sequence. In this formulation, the framework is designed to mitigate accumulated errors, motion drift, and content repetition in minute-scale autoregressive video generation while preserving streaming capability, prompt-adaptive interaction, and linear scaling in sequence length (Yu et al., 4 Dec 2025).

1. Conceptual scope and problem setting

VideoSSM is situated within autoregressive diffusion for long-form video synthesis. Its central premise is that causally generated video cannot rely only on local context if it is expected to remain coherent over hundreds or thousands of frames. The framework therefore treats long-video generation as a memory problem: small prediction errors compound over time, motion cues drift, and fixed global anchors can induce looping or frozen scenes. The model is explicitly constructed to address three named failure modes: error accumulation, motion drift, and content repetition (Yu et al., 4 Dec 2025).

The framework’s memory design is explicitly two-tiered. A local sliding-window cache preserves lossless short-term information such as fine appearance and motion cues, while a global SSM maintains a fixed-size latent state that is continuously updated from evicted context. This division separates short-range fidelity from long-range scene evolution. A plausible implication is that VideoSSM does not treat “global memory” as a static archive; rather, it treats it as an evolving compressed dynamical state, which differs from schemes that retain early frames as immutable anchors.

A common misconception is that long-horizon autoregressive generation can be stabilized merely by extending a causal attention window. VideoSSM instead assumes that purely local autoregressive models remain vulnerable to compounding errors, whereas static global-context schemes risk repetitive or frozen patterns. Its hybrid memory is intended to preserve global consistency without those frozen “attention-sink” artifacts (Yu et al., 4 Dec 2025).

2. Hybrid memory architecture

In VideoSSM, each diffusion block replaces standard masked causal self-attention with a two-stream hybrid module. The local stream is a causal sliding window containing the last $L$ frames’ key/value pairs together with a small “sink” anchor. The global stream is a fixed-size state $M_t \in \mathbb{R}^d$ that evolves through an SSM and absorbs information evicted from the local window. At frame $t$, the block performs four operations in sequence: it computes local attention over $$\{\text{sink}\} \cup \{\text{frames } t-L+1 \ldots t\}$$, evicts tokens older than $t-L$ and compresses them into $M_t$, reads out a global context vector from $M_t$, and fuses local and global contexts through a learned router gate $\gamma_t$ (Yu et al., 4 Dec 2025).

The local path follows the standard projection-and-cache pattern. For hidden input $H_{\text{in}}^t$, the model computes

$$Q_t = H_{\text{in}}^t W_Q,\qquad K_t = H_{\text{in}}^t W_K,\qquad V_t = H_{\text{in}}^t W_V,$$

appends $M_t \in \mathbb{R}^d$0 to a rolling KV cache of size $M_t \in \mathbb{R}^d$1, and evaluates causal self-attention:

$M_t \in \mathbb{R}^d$2

The global path is activated by eviction. Tokens leaving the local window are summarized and written into the SSM state. The architecture thus preserves full streaming capability while granting access to the entire history through a fixed-dimensional memory. In the paper’s description, this preserves $M_t \in \mathbb{R}^d$3 complexity, while the later computational analysis states $M_t \in \mathbb{R}^d$4 per token for local attention plus $M_t \in \mathbb{R}^d$5 for SSM gating and update; with $M_t \in \mathbb{R}^d$6, the total scales as $M_t \in \mathbb{R}^d$7 (Yu et al., 4 Dec 2025).

The architectural significance of this design lies in its asymmetry. The local cache is lossless but short, whereas the global SSM is lossy but persistent. This suggests that VideoSSM is not primarily an “attention replacement” architecture; it is a memory-allocation architecture in which attention and SSM dynamics are assigned different roles.

3. State-space dynamics and read–write mechanism

The global memory is implemented as a recurrent SSM with gating, summarized by the compressed state $M_t \in \mathbb{R}^d$8. The paper defines two gates from the token hidden state $M_t \in \mathbb{R}^d$9:

$t$0

Here $t$1, $t$2, $t$3, and $t$4 are learnable matrices of appropriate dimension. Over the tokens evicted at time $t$5, VideoSSM averages their keys, values, and gates to produce a single batch of SSM inputs. The update then proceeds as

$t$6

$t$7

The $t$8 term estimates the predictable part of $t$9 from the prior memory and the evicted keys, so that only novel information is written into the state (Yu et al., 4 Dec 2025).

This update is described as a gated delta-rule that replaces explicit $$\{\text{sink}\} \cup \{\text{frames } t-L+1 \ldots t\}$$0 recurrence in the standard linear SSM

$$\{\text{sink}\} \cup \{\text{frames } t-L+1 \ldots t\}$$1

The design mixes decay and injection: older content is attenuated through cumulative decay $$\{\text{sink}\} \cup \{\text{frames } t-L+1 \ldots t\}$$2, and new content enters via an outer-product update involving evicted keys and residualized values. The resulting state is therefore not a raw cache of past tokens, but a compressed second-order summary shaped by novelty filtering.

Reading from the global memory is likewise explicit. The model computes

$$\{\text{sink}\} \cup \{\text{frames } t-L+1 \ldots t\}$$3

A router gate based on the relative position ratio $$\{\text{sink}\} \cup \{\text{frames } t-L+1 \ldots t\}$$4 then controls the contribution of the global stream:

$$\{\text{sink}\} \cup \{\text{frames } t-L+1 \ldots t\}$$5

$$\{\text{sink}\} \cup \{\text{frames } t-L+1 \ldots t\}$$6

This makes the hybridization query-dependent at readout and schedule-dependent at fusion. A plausible implication is that the architecture can vary its reliance on local and global evidence over the rollout horizon rather than fixing that mixture a priori (Yu et al., 4 Dec 2025).

The same section of the paper situates this hybrid memory inside an autoregressive diffusion process. The forward diffusion is written as

$$\{\text{sink}\} \cup \{\text{frames } t-L+1 \ldots t\}$$7

and the reverse denoiser is conditioned on the fused context $$\{\text{sink}\} \cup \{\text{frames } t-L+1 \ldots t\}$$8 through

$$\{\text{sink}\} \cup \{\text{frames } t-L+1 \ldots t\}$$9

4. Training procedure, stabilization, and streaming inference

VideoSSM is trained in two stages. Stage 1 is ODE-based Self-Forcing Distillation. A student $t-L$0 is initialized from a pretrained bidirectional teacher $t-L$1, and on $t-L$2 teacher trajectories the model minimizes

$t-L$3

This stage aligns short-term denoising behavior and warms up the hybrid memory. Stage 2 is Long-Horizon DMD Distillation. The student is rolled out autoregressively over $t-L$4 while updating the local and global memories from its own outputs; random $t-L$5 windows are then sampled within the rollout and optimized with the Distribution-Matching Distillation objective

$t-L$6

The total loss is

$t-L$7

with $t-L$8 tuned to balance short versus long context (Yu et al., 4 Dec 2025).

The training logic is notable because it separates short-clip denoising imitation from long-horizon rollout correction. This suggests that VideoSSM is not trained only to match a teacher locally; it is trained to remain distributionally aligned under its own autoregressive dynamics.

For inference, the model is explicitly streaming. Memory is reset once, and frames are generated causally. At each frame, the model initializes latent noise, performs reverse diffusion steps, updates the local KV cache, performs a global update if eviction occurs, computes the global readout, fuses both streams, and uses the fused context for denoising. The paper states that only a small KV window and the fixed-size $t-L$9 are accessed per frame, enabling $M_t$0 or higher. It also states that interactive prompt switching can be performed by clearing or recaching the local KV cache while keeping $M_t$1, allowing rapid adaptation without full re-rollout (Yu et al., 4 Dec 2025).

The paper further states that no additional weight decay or special regularizers are required due to gating-based stability in $M_t$2. Within the paper’s framing, stability is therefore attributed primarily to the gated memory update rather than to auxiliary regularization.

5. Empirical results and computational profile

On the VBench suite, VideoSSM is reported on both short-video and minute-long generation. For short-video generation at $M_t$3, the paper reports a VideoSSM Total score of $M_t$4, compared with LongLive at $M_t$5 and Self-Forcing at $M_t$6. It also reports Quality $M_t$7, identified as best among AR models (Yu et al., 4 Dec 2025).

For minute-long generation at $M_t$8, the paper reports the following metrics:

Metric	Comparator values	VideoSSM
Temporal Flicker $M_t$9	Self-Forcing 97.86; LongLive 97.24	97.70
Subject Consistency $M_t$0	Self-Forcing 88.25; LongLive 91.09	92.51
Background Consistency $M_t$1	Self-Forcing 91.73; LongLive 93.23	93.95
Motion Smoothness $M_t$2	Self-Forcing 98.67; LongLive 98.38	98.60
Dynamic Degree $M_t$3	Self-Forcing 35.00; LongLive 37.50	50.50

The paper summarizes these results by stating that VideoSSM achieves the highest subject and background consistency while preserving natural dynamism, with Dynamic Degree $M_t$4 versus $M_t$5 for LongLive. In a user study with $M_t$6 participants and $M_t$7 prompts, VideoSSM received $M_t$8 Rank-1 votes and the lowest average rank, $M_t$9 (Yu et al., 4 Dec 2025).

Its computational profile is described in terms of both time and memory. The local KV cache requires $\gamma_t$0 memory, the global SSM state requires $\gamma_t$1 memory, and the gates cache requires $\gamma_t$2. Time complexity is reported as $\gamma_t$3 per token for local attention plus $\gamma_t$4 for SSM gating and update, with total scaling $\gamma_t$5 when $\gamma_t$6. The paper also links this profile to streaming and interactive operation.

An objective reading of the reported results is that VideoSSM emphasizes long-horizon coherence and motion stability without collapsing dynamic degree. This is important because long-video autoregressive systems can improve consistency by becoming overly static; the reported Dynamic Degree result is presented specifically as evidence against that failure mode.

6. Terminology, related work, and position within SSM-based video research

The term “VideoSSM” has more than one usage in the literature. In the paper titled "VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory," it denotes the long-video generation framework described above (Yu et al., 4 Dec 2025). In a separate line of work, the details for "VideoMamba: Spatio-Temporal Selective State Space Model" define VideoSSM as the core operator within VideoMamba, namely a Spatio-Temporal Selective State Space Model for video recognition that scans flattened tokens in forward and backward orders with linear complexity (Park et al., 2024). The two usages are related by their reliance on selective SSM dynamics, but they refer to different model families and tasks.

Within generative modeling, VideoSSM belongs to a broader shift from quadratic temporal attention toward structured state-space formulations. "SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spaces" replaces temporal attention in a U-Net-style video diffusion model with bidirectional SSM layers, reporting linear memory growth with sequence length and generation up to $\gamma_t$7 frames on MineRL where attention-based baselines ran out of memory (Oshima et al., 2024). VideoSSM differs in that it does not simply substitute SSMs for temporal attention everywhere; instead, it uses an SSM as compressed global memory coupled to a local attention window.

Outside video generation, the same design pattern—linear-complexity long-range modeling through SSM recurrence or scanning—appears across multiple video domains. VideoMamba adapts Mamba to video understanding with bidirectional scanning and reports strong performance on short- and long-term tasks (Li et al., 2024). TV3S uses Mamba state-space sharing for video semantic segmentation and maintains per-patch hidden states across frames (Hesham et al., 26 Mar 2025). TS-Mamba applies trajectory-aware shifted SSMs to online video super-resolution (Zhu et al., 14 Aug 2025). VFIMamba introduces a Mixed-SSM Block for video frame interpolation (Zhang et al., 2024). MambaVF reformulates video fusion as a sequential state update process with flow-free bidirectional SSM scanning (Zhao et al., 5 Feb 2026). RS-SSM addresses forgotten specifics in state-space video semantic segmentation through gate refinement (Zhu et al., 25 Mar 2026). UST-SSM extends spatio-temporal SSM modeling to point cloud videos through selection scanning and structure aggregation (Li et al., 20 Aug 2025).

Taken together, these works indicate that SSMs in video are not confined to a single role. They can function as temporal feature extractors, recurrent memories, scan-based spatio-temporal mixers, or compressed latent summaries. VideoSSM’s distinctive contribution within this landscape is the use of a hybrid sliding-window plus SSM memory specifically for autoregressive diffusion over minute-scale horizons, with prompt-adaptive streaming generation as an explicit design target (Yu et al., 4 Dec 2025).