StreamMem: Query-Agnostic KV Cache for Video Streaming

• StreamMem is a streaming video mechanism that compresses and updates a fixed-size KV cache using proxy queries and causal processing.
• It achieves competitive video QA performance by balancing memory constraints with efficient online compression of visual tokens.
• Beyond its initial design, StreamMem serves as a benchmark and reference framework, inspiring alternative memory strategies like FluxMem and SAVEMem.

StreamMem most commonly denotes a training-free, query-agnostic KV cache memory mechanism for streaming video understanding in multimodal LLMs. It was introduced to address a setting in which storing and attending to the key-value cache for long visual contexts incurs substantial memory and computational overhead, while many prior visual compression methods either require encoding the entire visual context before compression or having access to the questions in advance (Yang et al., 21 Aug 2025). In later literature, the term is also used more broadly for the streaming-memory capability itself: maintaining bounded memory under continuous observations and converting that memory into later question answering, reasoning, or future-oriented assistance (Liu et al., 12 Jun 2026). Across these usages, StreamMem is defined by three recurring constraints: causal processing, query-agnostic memory formation, and bounded memory.

1. Definition, scope, and naming

In its specific 2025 sense, StreamMem is the method titled “StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding” (Yang et al., 21 Aug 2025). Its central design claim is that new video frames can be encoded in a streaming manner while the KV cache is continuously compressed using attention scores between visual tokens and generic query tokens, thereby maintaining a fixed-size KV memory for long-video question answering under memory constraints (Yang et al., 21 Aug 2025).

Subsequent papers treat StreamMem as a reference point for a particular design family. “FluxMem” describes StreamMem as “Query-Agnostic KV Cache Memory” that operates after tokens have entered the LLM and focuses on cache-side retention and eviction rather than pre-LLM compression (Xie et al., 2 Mar 2026). “What Should a Streaming Video Model Remember?” refers to “StreamMem (Yang et al., 2025)” as a query-agnostic KV-cache mechanism that keeps a cache of key-value states from prior decoding or encoding and retrieves them to augment current queries (Ge et al., 15 Jun 2026). “Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding” similarly characterizes StreamMem as maintaining a bounded KV-cache through continuous re-compression (Wu et al., 8 May 2026).

The name is easily confused with other systems. “Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory” states explicitly that MemStream is a separate, contemporaneous work and that StreamMem is not an alias for MemStream (Agarwal et al., 20 Feb 2026). A still earlier anomaly-detection paper is also named MemStream and states that the term StreamMem does not appear there (Bhatia et al., 2021). In another domain, an earlier streaming Transformer acoustic model describes an augmented-memory self-attention mechanism as “StreamMem,” using short segments plus a bank of segment-level memories for speech recognition (Wu et al., 2020). This naming history suggests that the stable meaning of StreamMem is not a single architecture across all fields, but a memory-organizing principle for streaming inference.

2. Query-agnostic KV cache memory for streaming video

The original StreamMem paper formulates streaming video with fixed KV memory as an online update problem. At time step $t$, a new video clip $v_t$ arrives, producing raw KVs $(K_t^i, V_t^i)$ at layer $i$, while the previous compressed memory is $(K_{t-1}^{i\prime}, V_{t-1}^{i\prime})$. StreamMem computes updated compressed KVs under a global budget shared across layers (Yang et al., 21 Aug 2025):

$$K_t^{i\prime}, V_t^{i\prime} = \mathrm{Compress}(K_{t-1}^{i\prime}, K_t^i, V_{t-1}^{i\prime}, V_t^i), \qquad \sum_{i=1}^{L} \|K_t^{i\prime}\|_0 \le M.$$

In practice, the memory budget is evenly distributed as $M_i = \lfloor M/L \rfloor$ (Yang et al., 21 Aug 2025). The streaming update has five named components: input frame filtering, query-agnostic saliency via generic query tokens, KV cache compression per step, fixed-size memory, and positional embeddings via YaRN visual context window extension (Yang et al., 21 Aug 2025).

The query-agnostic scoring mechanism is the defining feature. Because the user query is unknown during streaming, StreamMem uses chat template tokens as proxy queries, specifically <|im_end|><|im_start|>assistant\n appended after visual tokens (Yang et al., 21 Aug 2025). Their cross-attention to visual tokens yields token-importance scores. For a layer with proxy-query representations $Q \in \mathbb{R}^{q \times d}$ and visual keys $K_t^i \in \mathbb{R}^{n \times d}$, the paper defines

$$A_t^i = \mathrm{Softmax}\!\left(\frac{Q (K_t^i)^\top}{\sqrt{d}}\right), \qquad A_t^i \in \mathbb{R}^{q \times n},$$

with visual-token importance aggregated as

$v_t$0

The same scoring is applied to the previous compressed memory, so old and new tokens are jointly ranked (Yang et al., 21 Aug 2025). Selection is then performed by TopK over the concatenated score vector, reserving slots for frame-level prototypes. The frame-wise merging step constructs a single prototype KV per retained frame using attention-weighted pooling:

$v_t$1

The final compressed memory is the concatenation of selected salient tokens and inserted frame prototypes, with position IDs preserved rather than reassigned (Yang et al., 21 Aug 2025).

The streaming loop is fully incremental. At each time step, StreamMem reads a new frame chunk, filters adjacent frames whose cosine similarity exceeds $v_t$2, encodes the filtered chunk with proxy queries, scores both old memory and new tokens, performs joint pruning, constructs one prototype per retained frame, and updates each layer’s memory so that $v_t$3 always holds (Yang et al., 21 Aug 2025). The default chunk size is $v_t$4 frames, the default filtering threshold is $v_t$5, and many experiments use a per-layer KV budget of $v_t$6 tokens, with additional studies at $v_t$7 and $v_t$8 (Yang et al., 21 Aug 2025).

3. Empirical behavior and ablation results

The original evaluation covers three offline long-video benchmarks—MLVU, EgoSchema, and VideoMME—and two streaming QA benchmarks, RVS-Ego and RVS-Movie (Yang et al., 21 Aug 2025). The reported results position StreamMem as state-of-the-art for query-agnostic KV cache compression and competitive with query-aware compression approaches (Yang et al., 21 Aug 2025).

Setup	StreamMem result	Comparison stated in the paper
LLaVA-OneVision-7B, KV $v_t$9/layer	MLVU 66.9, EgoSchema 63.0, VideoMME All 59.4	baseline 64.7, 60.1, 56.9
Qwen2-VL-7B, KV $(K_t^i, V_t^i)$0/layer	MLVU 65.9, EgoSchema 67.2, VideoMME All 62.1	InfiniPot-V 65.8, 65.6, 62.8
Qwen2.5-VL-3B, KV $(K_t^i, V_t^i)$1/layer	MLVU 62.3, EgoSchema 62.2, VideoMME All 59.5	InfiniPot-V 62.1, 61.8, 59.3
Qwen2-VL-7B, KV $(K_t^i, V_t^i)$2/layer	MLVU All 66.3	exceeds full KV 65.9

The budget study is one of the strongest empirical points. On Qwen2-VL-7B for MLVU, StreamMem reports 65.9 at $(K_t^i, V_t^i)$3, 66.0 at $(K_t^i, V_t^i)$4, and 66.3 at $(K_t^i, V_t^i)$5, while the full-KV setting is 65.9 (Yang et al., 21 Aug 2025). This is a compression regime in which a bounded memory can match or exceed the uncompressed baseline.

The ablations are equally revealing. Using the true query for scoring gives the best MLVU All result, 68.1, but the default chat-template proxy reaches 66.9 and outperforms a generic text prompt at 66.7 (Yang et al., 21 Aug 2025). For merging, attention-weighted merging yields 66.9, compared with 66.3 for average merging and 65.6 with no merging (Yang et al., 21 Aug 2025). For filtering, the default $(K_t^i, V_t^i)$6 gives 66.9, while no filtering gives 65.4 (Yang et al., 21 Aug 2025). On LLaVA-OneVision-7B, YaRN scaling also matters: $(K_t^i, V_t^i)$7 gives 66.9, compared with 61.5 at $(K_t^i, V_t^i)$8 (Yang et al., 21 Aug 2025).

On streaming QA, StreamMem does not dominate the upper-bound oracle ReKV, but it remains competitive. With LLaVA-OneVision-7B, StreamMem reports 57.6/3.8 on RVS-Ego and 52.7/3.4 on RVS-Movie, compared with ReKV’s 63.7/4.0 and 54.4/3.6, respectively (Yang et al., 21 Aug 2025). The paper treats this gap as part of the distinction between query-agnostic compression and query-aware or offloading-based retrieval.

4. StreamMem within the streaming-video memory landscape

Later work uses StreamMem as a baseline category against which other memory strategies are differentiated. “FluxMem” makes the contrast explicit: StreamMem manages the LLM’s KV cache with query-agnostic policies after tokens have entered the LLM, whereas FluxMem compresses visual tokens before they reach the LLM and thereby reduces the KV burden at the source (Xie et al., 2 Mar 2026). In the quantitative comparison reported there, StreamMem has VideoMME scores of Short 71.5, Medium 62.4, Long 52.3, All 62.1, and MLVU M-Avg 65.9, while FluxMem reports VideoMME All 65.3 and MLVU 73.1 (Xie et al., 2 Mar 2026).

“SelectStream” reframes the problem as budgeted online latent evidence allocation rather than query-agnostic KV reuse. Its memory stores projected visual embeddings consolidated into segment-level nodes, enforces a hard node budget $(K_t^i, V_t^i)$9, retrieves a subgraph of size $i$0, and injects only $i$1 calibrated latent evidence tokens into a frozen VLM (Ge et al., 15 Jun 2026). In that account, the main difference from StreamMem is not merely implementation detail but the level of representation: KV-cache activations versus fixed-capacity latent evidence nodes (Ge et al., 15 Jun 2026).

Other systems position themselves similarly. “SAVEMem” contrasts its semantically aware three-tier visual-token memory with StreamMem’s bounded KV-cache through continuous re-compression (Wu et al., 8 May 2026). “CurveStream” frames the broader problem as streaming KV or memory under strict budgets and argues for a geometry-grounded admission policy based on curvature-aware scoring rather than passive cache-side management (Wang et al., 20 Mar 2026). “Mosaic” reorganizes KVCache around cross-modal clusters and reports average accuracy improvements of +2.29% versus StreamMem, while noting that StreamMem is slightly faster but loses accuracy due to query-agnostic memory (Wang et al., 11 Apr 2026).

This later literature does not invalidate StreamMem’s original contribution. Rather, it makes StreamMem a reference architecture for one specific answer to the streaming-memory problem: compress the KV cache online, without access to the future question, and keep the memory fixed-size.

5. StreamMem as a benchmarked capability in personal agents

A later shift in meaning appears in “StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance” (Liu et al., 12 Jun 2026). There, StreamMem is not the 2025 KV-cache mechanism itself, but “the sense most relevant to personal agents: whether memory formed from streaming egocentric observations and interactions actually improves future-oriented assistance” (Liu et al., 12 Jun 2026). The benchmark is designed around the trajectory from evidence formation to initial use, feedback-driven correction, and later reuse (Liu et al., 12 Jun 2026).

The benchmark builds on EgoLife, using 7-day continuous recordings from 6 participants, divided into 3,347 five-minute segments and yielding 8,107 evidence anchors and 16,214 queries (Liu et al., 12 Jun 2026). Each anchor produces a two-step task sequence: an initial task $i$2 that requires hidden evidence, user feedback $i$3 based on the initial response $i$4, and a follow-up task $i$5 that tests whether the system reuses the same anchor or corrected interaction experience (Liu et al., 12 Jun 2026). The construction enforces three constraints:

$i$6

The four diagnostic metrics are Fidelity, Initial Evidence Use (IEU), Feedback Incorporation (FI), and Follow-up Reuse (FUR). The benchmark defines, for example,

$i$7

with analogous definitions for FI and FUR, then aggregates them into pass rates $i$8, $i$9, $(K_{t-1}^{i\prime}, V_{t-1}^{i\prime})$0, and $(K_{t-1}^{i\prime}, V_{t-1}^{i\prime})$1 (Liu et al., 12 Jun 2026).

The reported outcomes show why the benchmark broadens the meaning of StreamMem. Systems can store evidence yet fail to use it when it matters. A-Mem and MemoryOS achieve Fidelity 100.0† on both backbones, but their IEU is only 34.92 and 23.93 on DeepSeek-V4-Flash, and their FUR is 64.98 and 61.95, respectively (Liu et al., 12 Jun 2026). MemOS is the extreme case: Fidelity 67.97, FI 77.25, but FUR 3.96 on DeepSeek-V4-Flash (Liu et al., 12 Jun 2026). The benchmark’s central conclusion is that current systems often fail to use observed evidence or turn feedback into reliable follow-up behavior, even when evidence is stored or feedback is incorporated locally (Liu et al., 12 Jun 2026).

In this broader sense, StreamMem becomes an end-to-end property of streaming assistance rather than only a cache-management algorithm. The shift is substantive: the benchmark measures whether streaming observations, memory writes, user feedback, and later reuse form a coherent behavioral trajectory (Liu et al., 12 Jun 2026).

6. Limitations and research trajectory

The original StreamMem paper identifies several limitations. There remains a gap to true query-aware selection, since ablations show that the true question outperforms proxy queries (Yang et al., 21 Aug 2025). Performance is sensitive to the filtering threshold $(K_{t-1}^{i\prime}, V_{t-1}^{i\prime})$2, YaRN scaling $(K_{t-1}^{i\prime}, V_{t-1}^{i\prime})$3, per-layer memory budget, and chunk size (Yang et al., 21 Aug 2025). Frame-level merging can lose fine-grained spatial relations if compression is too aggressive, and the best positional-extension setting is model-dependent (Yang et al., 21 Aug 2025). The design also compresses visual KVs only; it does not introduce an explicit multi-turn dialogue memory beyond visual KV (Yang et al., 21 Aug 2025).

The benchmark literature adds a second layer of limitation. High Fidelity does not guarantee IEU or FUR, and immediate correction does not guarantee later reuse (Liu et al., 12 Jun 2026). For personal agents, persistent memory also raises privacy and safety requirements: consent, minimization, access control, and user inspection, correction, and deletion mechanisms (Liu et al., 12 Jun 2026).

A plausible implication is that query-agnostic KV compression solves only one part of the streaming-memory problem. Later systems respond by moving the intervention point earlier or later in the pipeline, or by changing the memory unit altogether: pre-LLM spatiotemporal token compression in FluxMem (Xie et al., 2 Mar 2026), fixed-capacity latent evidence graphs in SelectStream (Ge et al., 15 Jun 2026), semantic-aware visual-token retention in SAVEMem (Wu et al., 8 May 2026), curvature-aware hierarchical visual memory management in CurveStream (Wang et al., 20 Mar 2026), and cross-modal clustering for KVCache organization in Mosaic (Wang et al., 11 Apr 2026). These alternatives preserve StreamMem’s core constraints—causality, bounded memory, and long-horizon retrieval—while challenging its specific assumption that query-agnostic proxy attention over the KV cache is the most effective locus for streaming-memory control.

Taken together, StreamMem names both a concrete 2025 method and a broader research program. In the narrow sense, it is a query-agnostic KV cache memory mechanism that continuously compresses old and new visual tokens under a fixed budget (Yang et al., 21 Aug 2025). In the broader sense that later benchmarks make explicit, it is the ability of a system to carry forward evidence formed from streaming observations and interactions and reuse it when future tasks require it (Liu et al., 12 Jun 2026).