Round-Decayed Memory Buffer Compression

• Memory buffer with round-decayed compression is a method that manages multi-turn contextual history, preserving detail in recent rounds while progressively compressing older ones.
• It uses a round-based strategy where older rounds undergo full or partial average-pooling to meet strict token budgets without sacrificing current context accuracy.
• Empirical results demonstrate that this approach maintains constant latency and higher benchmark performance compared to truncation and uniform compression methods.

A memory buffer with round-decayed compression is a data structure and algorithmic framework for managing long, multi-turn contextual histories in neural models, particularly for streaming or continual learning scenarios. It ensures that recent context is preserved with high fidelity while earlier context is incrementally compressed to meet strict resource constraints on memory or token budget. This is achieved by maintaining historical data in a sequence of "rounds" and applying compression preferentially to the oldest rounds, thus supporting multi-turn interactive systems such as streaming Video LLMs (Video-LLMs) and continual learning systems (Wang et al., 8 May 2025, Wang et al., 2022).

1. Memory Buffer Organization

The memory buffer, denoted $\mathcal{MB}$, is a logically ordered list of rounds. Each round $r$ contains two primary components:

• Visual Embeddings ($V_r \in \mathbb{R}^{n_r \times d}$): Representing features from a sequence of video frames, generally arranged as token-per-frame times number-of-frames for that round.
• Text Embeddings ($T_r \in \mathbb{R}^{m_r \times d}$): Encodings of textual user queries and/or model responses associated with that round.

In practice, $\mathcal{MB}$ is implemented as a list of tuples, e.g., [(V_1, T_1), (V_2, T_2), ...], with new frame features or textual events appended to the latest round as they arrive. This design supports efficient retrieval and update, and maintains strict temporal ordering for round-based compression strategies (Wang et al., 8 May 2025).

2. Producer–Consumer Flow and Buffer Management

The producer–consumer paradigm governs how data enters and leaves $\mathcal{MB}$:

1. Frame Arrival: Each new incoming frame $F_t$ is processed to obtain its visual embedding $v_t = \mathcal{I}(F_t)$, which is appended to the active visual chunk.
2. User Event: When a user query $\mathcal{Q}$ arrives, its embedding is appended to $\mathcal{MB}$ as a new text chunk $r$0.
3. Response Activation: For each subsequent frame, an activation model ($r$1) decides—based on context and user interaction—when to issue a model response.
4. Flatten/Compress/Respond: Upon activation, $r$2 is flattened into an embedding sequence. If length exceeds a fixed maximum ($r$3), the round-decayed compression subroutine $r$4 is triggered. The resulting sequence is provided to the model, and the model’s response is appended as a new text chunk, thus maintaining multi-turn interaction (Wang et al., 8 May 2025).

3. Mathematical Description of Round-Decayed Compression

Round-decayed compression is explicitly defined to prioritize compression of earlier rounds and preserve as much of the recent context as possible:

Given overall token length $r$5 (sum of embedding lengths over all rounds), token budget $r$6, and fixed token-per-frame $r$7, when $r$8,

$r$9

Rounds $V_r \in \mathbb{R}^{n_r \times d}$0 (earliest to latest) are visited in order:

• Full Compression: If $V_r \in \mathbb{R}^{n_r \times d}$1, average-pool all of $V_r \in \mathbb{R}^{n_r \times d}$2’s tokens into $V_r \in \mathbb{R}^{n_r \times d}$3 tokens, then decrement $V_r \in \mathbb{R}^{n_r \times d}$4 by $V_r \in \mathbb{R}^{n_r \times d}$5.
• Partial Compression: Otherwise, compress only the first $V_r \in \mathbb{R}^{n_r \times d}$6 tokens, average-pooling them to $V_r \in \mathbb{R}^{n_r \times d}$7, leaving the remainder untouched. The process terminates once the required compression target is achieved.

This algorithmic design ensures that only the oldest rounds are compressed, maintaining high-detail content in more recent rounds. The entire procedure is formalized and implemented efficiently in PyTorch, as provided in the originating publication (Wang et al., 8 May 2025).

4. Design Comparison and Trade-offs

Various memory management strategies were benchmarked:

Method	OVO-Bench Avg	Streaming-Bench Avg	DVCₛᵢₘ	SLCₛᵢₘ
Truncation	68.88	72.79	22.1	16.7
Round-Uniform	69.91	74.18	23.8	15.9
Round-Decayed	71.30	77.04	25.1	17.1

Truncation discards old and new tokens, severely impacting multi-turn reasoning. Round-uniform applies the same compression ratio to all rounds, diminishing the quality of current context. Round-decayed merges earliest rounds aggressively and leaves the most recent rounds uncompressed, optimizing for both real-time and historic context preservation (Wang et al., 8 May 2025).

Latency experiments confirm that, without compression, inference latency increases linearly and may exceed available GPU memory. In contrast, round-decayed compression maintains nearly constant latency as buffer length grows beyond $V_r \in \mathbb{R}^{n_r \times d}$8.

5. Connections to Continual Learning and Data Compression

In the context of continual learning, memory replay with data compression (MRDC) leverages similar principles for optimizing a fixed storage budget $V_r \in \mathbb{R}^{n_r \times d}$9. The system chooses an appropriate compression quality $T_r \in \mathbb{R}^{m_r \times d}$0 per round/task to pack as many samples as possible, using a determinantal point process (DPP) criterion to maintain diversity and fidelity in the compressed feature space (Wang et al., 2022).

Mathematically, MRDC solves:

$T_r \in \mathbb{R}^{m_r \times d}$1

where

$T_r \in \mathbb{R}^{m_r \times d}$2

with $T_r \in \mathbb{R}^{m_r \times d}$3 and $T_r \in \mathbb{R}^{m_r \times d}$4 representing the root-determinant volumes of compressed and uncompressed feature matrices, respectively. This ensures a "round-decay" of compression quality: compression is more aggressive early and can be relaxed as the buffer fills or as feature distortion constraints demand.

Empirical results demonstrate that MRDC, with JPEG quality optimization via $T_r \in \mathbb{R}^{m_r \times d}$5, consistently outperforms naive memory replay or grid-searched fixed-quality baselines, both in classification and semi-supervised object detection scenarios (Wang et al., 2022).

6. Broader Impacts and Practical Utility

Memory buffers with round-decayed compression enable scalable, proactive, multi-turn streaming systems by accommodating long-context reasoning at tractable computational cost. In practice, this enables offline Video-LLMs to be extended for streaming interaction, supporting advanced tasks such as real-time video understanding, multi-modality chat, and continual video-based learning.

Key benefits demonstrated empirically include:

• Retention of detailed context for the current turn while only sketching over older turns
• Stable inference latency and memory usage, even as the interaction history grows
• Superior benchmark performance relative to alternative compression or truncation methods

These findings position round-decayed compression as a core mechanism for long-context modeling in both streaming LLMs and continual learning memory systems (Wang et al., 8 May 2025, Wang et al., 2022).