---
title: 'StreamMem: Query-Agnostic KV Cache for Video Streaming'
url: https://www.emergentmind.com/topics/streammem
type: topic
---

# StreamMem: Query-Agnostic KV Cache for Video Streaming

StreamMem most commonly denotes a training-free, query-agnostic KV cache memory mechanism for streaming video understanding in multimodal large language models. It was introduced to address a setting in which storing and attending to the key-value cache for long visual contexts incurs substantial memory and computational overhead, while many prior visual compression methods either require encoding the entire visual context before compression or having access to the questions in advance [2508.15717]. In later literature, the term is also used more broadly for the streaming-memory capability itself: maintaining bounded memory under continuous observations and converting that memory into later question answering, reasoning, or future-oriented assistance [2606.14571]. Across these usages, StreamMem is defined by three recurring constraints: causal processing, query-agnostic memory formation, and bounded memory.

## 1. Definition, scope, and naming

In its specific 2025 sense, StreamMem is the method titled “StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding” [2508.15717]. Its central design claim is that new video frames can be encoded in a streaming manner while the KV cache is continuously compressed using attention scores between visual tokens and generic query tokens, thereby maintaining a fixed-size KV memory for long-video question answering under memory constraints [2508.15717].

Subsequent papers treat StreamMem as a reference point for a particular design family. “FluxMem” describes StreamMem as “Query-Agnostic KV Cache Memory” that operates after tokens have entered the LLM and focuses on cache-side retention and eviction rather than pre-LLM compression [2603.02096]. “What Should a Streaming Video Model Remember?” refers to “StreamMem (Yang et al., 2025)” as a query-agnostic KV-cache mechanism that keeps a cache of key-value states from prior decoding or encoding and retrieves them to augment current queries [2606.16353]. “Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding” similarly characterizes StreamMem as maintaining a bounded KV-cache through continuous re-compression [2605.07897].

The name is easily confused with other systems. “Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory” states explicitly that MemStream is a separate, contemporaneous work and that StreamMem is not an alias for MemStream [2602.18434]. A still earlier anomaly-detection paper is also named MemStream and states that the term StreamMem does not appear there [2106.03837]. In another domain, an earlier streaming Transformer acoustic model describes an augmented-memory self-attention mechanism as “StreamMem,” using short segments plus a bank of segment-level memories for speech recognition [2005.08042]. This naming history suggests that the stable meaning of StreamMem is not a single architecture across all fields, but a memory-organizing principle for streaming inference.

## 2. Query-agnostic KV cache memory for streaming video

The original StreamMem paper formulates streaming video with fixed KV memory as an online update problem. At time step $t$, a new video clip $v_t$ arrives, producing raw KVs $(K_t^i, V_t^i)$ at layer $i$, while the previous compressed memory is $(K_{t-1}^{i\prime}, V_{t-1}^{i\prime})$. StreamMem computes updated compressed KVs under a global budget shared across layers [2508.15717]:

$$
K_t^{i\prime}, V_t^{i\prime} = \mathrm{Compress}(K_{t-1}^{i\prime}, K_t^i, V_{t-1}^{i\prime}, V_t^i),
\qquad
\sum_{i=1}^{L} \|K_t^{i\prime}\|_0 \le M.
$$

In practice, the memory budget is evenly distributed as $M_i = \lfloor M/L \rfloor$ [2508.15717]. The streaming update has five named components: input frame filtering, query-agnostic saliency via generic query tokens, KV cache compression per step, fixed-size memory, and positional embeddings via YaRN visual context window extension [2508.15717].

The query-agnostic scoring mechanism is the defining feature. Because the user query is unknown during streaming, StreamMem uses chat template tokens as proxy queries, specifically ``<|im_end|><|im_start|>assistant\n`` appended after visual tokens [2508.15717]. Their cross-attention to visual tokens yields token-importance scores. For a layer with proxy-query representations $Q \in \mathbb{R}^{q \times d}$ and visual keys $K_t^i \in \mathbb{R}^{n \times d}$, the paper defines

$$
A_t^i = \mathrm{Softmax}\!\left(\frac{Q (K_t^i)^\top}{\sqrt{d}}\right),
\qquad
A_t^i \in \mathbb{R}^{q \times n},
$$

with visual-token importance aggregated as

$$
s_{t,j}^i = \frac{1}{q}\sum_{r=1}^{q} A_{t,rj}^i.
$$

The same scoring is applied to the previous compressed memory, so old and new tokens are jointly ranked [2508.15717]. Selection is then performed by TopK over the concatenated score vector, reserving slots for frame-level prototypes. The frame-wise merging step constructs a single prototype KV per retained frame using attention-weighted pooling:

$$
\bar{K}_t^i = \sum_{j \in F} \alpha_j^i \cdot K_{t,j}^i,
\qquad
\bar{V}_t^i = \sum_{j \in F} \alpha_j^i \cdot V_{t,j}^i.
$$

The final compressed memory is the concatenation of selected salient tokens and inserted frame prototypes, with position IDs preserved rather than reassigned [2508.15717].

The streaming loop is fully incremental. At each time step, StreamMem reads a new frame chunk, filters adjacent frames whose cosine similarity exceeds $\delta$, encodes the filtered chunk with proxy queries, scores both old memory and new tokens, performs joint pruning, constructs one prototype per retained frame, and updates each layer’s memory so that $|K_t^{i\prime}| \le M_i$ always holds [2508.15717]. The default chunk size is $8$ frames, the default filtering threshold is $\delta = 0.95$, and many experiments use a per-layer KV budget of $6\mathrm{K}$ tokens, with additional studies at $12\mathrm{K}$ and $24\mathrm{K}$ [2508.15717].

## 3. Empirical behavior and ablation results

The original evaluation covers three offline long-video benchmarks—MLVU, EgoSchema, and VideoMME—and two streaming QA benchmarks, RVS-Ego and RVS-Movie [2508.15717]. The reported results position StreamMem as state-of-the-art for query-agnostic KV cache compression and competitive with query-aware compression approaches [2508.15717].

| Setup | StreamMem result | Comparison stated in the paper |
|---|---:|---|
| LLaVA-OneVision-7B, KV $6\mathrm{K}$/layer | MLVU 66.9, EgoSchema 63.0, VideoMME All 59.4 | baseline 64.7, 60.1, 56.9 |
| Qwen2-VL-7B, KV $6\mathrm{K}$/layer | MLVU 65.9, EgoSchema 67.2, VideoMME All 62.1 | InfiniPot-V 65.8, 65.6, 62.8 |
| Qwen2.5-VL-3B, KV $6\mathrm{K}$/layer | MLVU 62.3, EgoSchema 62.2, VideoMME All 59.5 | InfiniPot-V 62.1, 61.8, 59.3 |
| Qwen2-VL-7B, KV $24\mathrm{K}$/layer | MLVU All 66.3 | exceeds full KV 65.9 |

The budget study is one of the strongest empirical points. On Qwen2-VL-7B for MLVU, StreamMem reports 65.9 at $6\mathrm{K}$, 66.0 at $12\mathrm{K}$, and 66.3 at $24\mathrm{K}$, while the full-KV setting is 65.9 [2508.15717]. This is a compression regime in which a bounded memory can match or exceed the uncompressed baseline.

The ablations are equally revealing. Using the true query for scoring gives the best MLVU All result, 68.1, but the default chat-template proxy reaches 66.9 and outperforms a generic text prompt at 66.7 [2508.15717]. For merging, attention-weighted merging yields 66.9, compared with 66.3 for average merging and 65.6 with no merging [2508.15717]. For filtering, the default $\delta = 0.95$ gives 66.9, while no filtering gives 65.4 [2508.15717]. On LLaVA-OneVision-7B, YaRN scaling also matters: $\lambda = 8$ gives 66.9, compared with 61.5 at $\lambda = 1$ [2508.15717].

On streaming QA, StreamMem does not dominate the upper-bound oracle ReKV, but it remains competitive. With LLaVA-OneVision-7B, StreamMem reports 57.6/3.8 on RVS-Ego and 52.7/3.4 on RVS-Movie, compared with ReKV’s 63.7/4.0 and 54.4/3.6, respectively [2508.15717]. The paper treats this gap as part of the distinction between query-agnostic compression and query-aware or offloading-based retrieval.

## 4. StreamMem within the streaming-video memory landscape

Later work uses StreamMem as a baseline category against which other memory strategies are differentiated. “FluxMem” makes the contrast explicit: StreamMem manages the LLM’s KV cache with query-agnostic policies after tokens have entered the LLM, whereas FluxMem compresses visual tokens before they reach the LLM and thereby reduces the KV burden at the source [2603.02096]. In the quantitative comparison reported there, StreamMem has VideoMME scores of Short 71.5, Medium 62.4, Long 52.3, All 62.1, and MLVU M-Avg 65.9, while FluxMem reports VideoMME All 65.3 and MLVU 73.1 [2603.02096].

“SelectStream” reframes the problem as budgeted online latent evidence allocation rather than query-agnostic KV reuse. Its memory stores projected visual embeddings consolidated into segment-level nodes, enforces a hard node budget $N$, retrieves a subgraph of size $B$, and injects only $M$ calibrated latent evidence tokens into a frozen VLM [2606.16353]. In that account, the main difference from StreamMem is not merely implementation detail but the level of representation: KV-cache activations versus fixed-capacity latent evidence nodes [2606.16353].

Other systems position themselves similarly. “SAVEMem” contrasts its semantically aware three-tier visual-token memory with StreamMem’s bounded KV-cache through continuous re-compression [2605.07897]. “CurveStream” frames the broader problem as streaming KV or memory under strict budgets and argues for a geometry-grounded admission policy based on curvature-aware scoring rather than passive cache-side management [2603.19571]. “Mosaic” reorganizes KVCache around cross-modal clusters and reports average accuracy improvements of +2.29% versus StreamMem, while noting that StreamMem is slightly faster but loses accuracy due to query-agnostic memory [2604.10060].

This later literature does not invalidate StreamMem’s original contribution. Rather, it makes StreamMem a reference architecture for one specific answer to the streaming-memory problem: compress the KV cache online, without access to the future question, and keep the memory fixed-size.

## 5. StreamMem as a benchmarked capability in personal agents

A later shift in meaning appears in “StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance” [2606.14571]. There, StreamMem is not the 2025 KV-cache mechanism itself, but “the sense most relevant to personal agents: whether memory formed from streaming egocentric observations and interactions actually improves future-oriented assistance” [2606.14571]. The benchmark is designed around the trajectory from evidence formation to initial use, feedback-driven correction, and later reuse [2606.14571].

The benchmark builds on EgoLife, using 7-day continuous recordings from 6 participants, divided into 3,347 five-minute segments and yielding 8,107 evidence anchors and 16,214 queries [2606.14571]. Each anchor produces a two-step task sequence: an initial task $T_1$ that requires hidden evidence, user feedback $F$ based on the initial response $R_1$, and a follow-up task $T_2$ that tests whether the system reuses the same anchor or corrected interaction experience [2606.14571]. The construction enforces three constraints:

$$
\mathrm{Leak}(T_i, a_k) = 0,\qquad
\mathrm{Need}(T_i, a_k) = 1,\qquad
\mathrm{Natural}(T_i) = 1.
$$

The four diagnostic metrics are Fidelity, Initial Evidence Use (IEU), Feedback Incorporation (FI), and Follow-up Reuse (FUR). The benchmark defines, for example,

$$
\mathrm{Fid}_k = \mathbf{1}[A_{\mathrm{eval}}(\Delta M_i, a_k)=\mathrm{pass}],
\qquad
\mathrm{IEU}_k = \mathbf{1}[A_{\mathrm{user}}(R_1,T_1,a_k)=\mathrm{pass}],
$$

with analogous definitions for FI and FUR, then aggregates them into pass rates $M_{\mathrm{Fid}}$, $M_{\mathrm{IEU}}$, $M_{\mathrm{FI}}$, and $M_{\mathrm{FUR}}$ [2606.14571].

The reported outcomes show why the benchmark broadens the meaning of StreamMem. Systems can store evidence yet fail to use it when it matters. A-Mem and MemoryOS achieve Fidelity 100.0† on both backbones, but their IEU is only 34.92 and 23.93 on DeepSeek-V4-Flash, and their FUR is 64.98 and 61.95, respectively [2606.14571]. MemOS is the extreme case: Fidelity 67.97, FI 77.25, but FUR 3.96 on DeepSeek-V4-Flash [2606.14571]. The benchmark’s central conclusion is that current systems often fail to use observed evidence or turn feedback into reliable follow-up behavior, even when evidence is stored or feedback is incorporated locally [2606.14571].

In this broader sense, StreamMem becomes an end-to-end property of streaming assistance rather than only a cache-management algorithm. The shift is substantive: the benchmark measures whether streaming observations, memory writes, user feedback, and later reuse form a coherent behavioral trajectory [2606.14571].

## 6. Limitations and research trajectory

The original StreamMem paper identifies several limitations. There remains a gap to true query-aware selection, since ablations show that the true question outperforms proxy queries [2508.15717]. Performance is sensitive to the filtering threshold $\delta$, YaRN scaling $\lambda$, per-layer memory budget, and chunk size [2508.15717]. Frame-level merging can lose fine-grained spatial relations if compression is too aggressive, and the best positional-extension setting is model-dependent [2508.15717]. The design also compresses visual KVs only; it does not introduce an explicit multi-turn dialogue memory beyond visual KV [2508.15717].

The benchmark literature adds a second layer of limitation. High Fidelity does not guarantee IEU or FUR, and immediate correction does not guarantee later reuse [2606.14571]. For personal agents, persistent memory also raises privacy and safety requirements: consent, minimization, access control, and user inspection, correction, and deletion mechanisms [2606.14571].

A plausible implication is that query-agnostic KV compression solves only one part of the streaming-memory problem. Later systems respond by moving the intervention point earlier or later in the pipeline, or by changing the memory unit altogether: pre-LLM spatiotemporal token compression in FluxMem [2603.02096], fixed-capacity latent evidence graphs in SelectStream [2606.16353], semantic-aware visual-token retention in SAVEMem [2605.07897], curvature-aware hierarchical visual memory management in CurveStream [2603.19571], and cross-modal clustering for KVCache organization in Mosaic [2604.10060]. These alternatives preserve StreamMem’s core constraints—causality, bounded memory, and long-horizon retrieval—while challenging its specific assumption that query-agnostic proxy attention over the KV cache is the most effective locus for streaming-memory control.

Taken together, StreamMem names both a concrete 2025 method and a broader research program. In the narrow sense, it is a query-agnostic KV cache memory mechanism that continuously compresses old and new visual tokens under a fixed budget [2508.15717]. In the broader sense that later benchmarks make explicit, it is the ability of a system to carry forward evidence formed from streaming observations and interactions and reuse it when future tasks require it [2606.14571].

Source: https://www.emergentmind.com/topics/streammem