Context-Aware Memory (CAM)

• Context-Aware Memory (CAM) is a design pattern that conditions memory storage, retrieval, and update on current contextual inputs.
• CAM systems execute key operations—contextual encoding, address formation, memory read, and adaptive update—to leverage historical data effectively.
• Diverse implementations of CAM across table ranking, visual dialog, speech synthesis, and network control lead to significant performance improvements.

Context-Aware Memory (CAM) denotes a family of mechanisms that condition storage, retrieval, and update on the current input context rather than treating inference steps as isolated. In the arXiv literature, the term spans several non-identical formulations: a single-hop key–value memory for table insight ranking, Context-aware Attention and Memory for visual dialog, a dynamic read-write Visual-Textual Context Memory Unit in large vision-LLMs, a vector-based memory store for AI-native Radio Access Networks, a fixed-length latent memory for long-context speech synthesis, and a hierarchical graph memory for long-document reading comprehension (Zeng et al., 2018, Shah et al., 2020, Shen et al., 6 Sep 2025, Barros, 6 May 2025, Li et al., 20 Aug 2025, Li et al., 7 Oct 2025).

1. Terminological scope and domain-specific formulations

The literature uses the same label for related but technically distinct mechanisms. What is shared is the attempt to preserve context across time, turns, episodes, or document regions, and to make that preserved context queryable during downstream inference.

Domain	Memory formulation	Primary role
Insight ranking	Key–Value Memory Network; “single-hop, read-only key-value memory” (Zeng et al., 2018)	inject “global” table context into per-insight ranking
Visual dialog	Context-aware Attention and Memory with Context-Aware Attention and Multi-Turn Memory (Shah et al., 2020)	reason over dialog history, particularly coreference resolution
Multi-turn LVLM reasoning	Visual-Textual Context Memory Unit with Adaptive Visual Focus Guidance (Shen et al., 6 Sep 2025)	store and manage critical visual features, textual semantic representations, and cross-modal correspondences
AI-native RAN	context encoder, vector-based memory store, recall engine, policy interface (Barros, 6 May 2025)	provide contextual recall for xApps and rApps
Long-context TTS	CAM block integrating long-term memory and local context details (Li et al., 20 Aug 2025)	guide sentence-level speech synthesis within long paragraphs
Long-document reading	Constructivist Agentic Memory with a hierarchical graph of representations (Li et al., 7 Oct 2025)	activate query-relevant information for contextual response

This distribution of uses indicates that CAM is better understood as a design pattern than as a single canonical module. Depending on the domain, “context” may mean peer insights in the same table, prior dialog turns, past network episodes, preceding utterances, or higher-level summaries in a document hierarchy.

2. Core computational pattern

Despite their heterogeneity, CAM systems recur to four operations: contextual encoding, address formation, memory read, and memory update or reorganization.

In the table-ranking formulation, the memory is statically populated as

$$m=\bigl[(s_1,I_1),\;(s_2,I_2),\;\dots,\;(s_M,I_M)\bigr],$$

with query vector $q=s_i$, key addressing

$$\alpha_k=\mathrm{Softmax}(q^\top s_k)=\frac{\exp(q^\top s_k)}{\sum_{j=1}^M \exp(q^\top s_j)},$$

and value reading

$o=\sum_{k=1}^M \alpha_k I_k.$

The paper explicitly notes that it does not describe a dynamic “write” or update rule; memory is treated as populated with all $M$ insights at the start of each forward pass (Zeng et al., 2018).

In CAMVR, the Visual-Textual Context Memory Unit maintains a fixed-size memory matrix

$M_t\in\mathbb{R}^{N_m\times D_m},$

and performs a gated GRU-style update:

$g_t=\sigma\bigl(W_g[E_t;M_{t-1}]+b_g\bigr),$

$$\widetilde M_t=\tanh\bigl(W_m[E_t;M_{t-1}]+b_m\bigr),$$

$M_t=(1-g_t)\odot M_{t-1}+g_t\odot \widetilde M_t.$

Retrieval is implemented by cross-attention between the current query text and memory slots:

$Q'_t=Q_tW_Q,\quad K'_t=M_tW_K,\quad V'_t=M_tW_V,$

$q=s_i$0

The memory has a fixed capacity $q=s_i$1, all slots are updated in parallel via the gate $q=s_i$2, and there is no explicit “filling pointer” (Shen et al., 6 Sep 2025).

RAN Cortex instead uses embedding-space retrieval. A learned encoder maps state to context vector $q=s_i$3, memory is

$q=s_i$4

and retrieval is

$q=s_i$5

with cosine similarity

$q=s_i$6

The retrieved history is then fused into an augmented policy

$q=s_i$7

The design uses ANN indexing, including FAISS HNSW or IVF-PQ, and reports sub-millisecond recall for $q=s_i$8 and $q=s_i$9 (Barros, 6 May 2025).

The TTS formulation uses a fixed-length latent memory with explicit compression, retrieval, and update. The update is

$$\alpha_k=\mathrm{Softmax}(q^\top s_k)=\frac{\exp(q^\top s_k)}{\sum_{j=1}^M \exp(q^\top s_j)},$$0

where the retrieved latent $$\alpha_k=\mathrm{Softmax}(q^\top s_k)=\frac{\exp(q^\top s_k)}{\sum_{j=1}^M \exp(q^\top s_j)},$$1 is obtained by cross-attending compressed current-text latents to a fused sequence of long-term memory and local history (Li et al., 20 Aug 2025).

Constructivist CAM organizes memory not as a flat matrix or vector store but as a hierarchy of graphs

$$\alpha_k=\mathrm{Softmax}(q^\top s_k)=\frac{\exp(q^\top s_k)}{\sum_{j=1}^M \exp(q^\top s_j)},$$2

with upward mappings $$\alpha_k=\mathrm{Softmax}(q^\top s_k)=\frac{\exp(q^\top s_k)}{\sum_{j=1}^M \exp(q^\top s_j)},$$3. Query-time retrieval begins with similarity-based localization and then expands through a “Prune-and-Grow” process over same-layer neighbors and lower-layer children (Li et al., 7 Oct 2025).

Taken together, these formulations suggest that CAM is less about a particular storage substrate than about conditioning memory access on the current task state and preserved historical structure.

3. Early neural formulations: table insight ranking and visual dialog

One of the earliest explicit CAM formulations appears in “Text Assisted Insight Ranking Using Context-Aware Memory Network” (Zeng et al., 2018). There, CAM is a Key–Value Memory Network whose purpose is to inject “global” table context into the per-insight ranking process. Each memory slot stores a key–value pair $$\alpha_k=\mathrm{Softmax}(q^\top s_k)=\frac{\exp(q^\top s_k)}{\sum_{j=1}^M \exp(q^\top s_j)},$$4, where $$\alpha_k=\mathrm{Softmax}(q^\top s_k)=\frac{\exp(q^\top s_k)}{\sum_{j=1}^M \exp(q^\top s_j)},$$5 is the “semantic key” derived from header words and $$\alpha_k=\mathrm{Softmax}(q^\top s_k)=\frac{\exp(q^\top s_k)}{\sum_{j=1}^M \exp(q^\top s_j)},$$6 is the full insight representation. The context-aware vector $$\alpha_k=\mathrm{Softmax}(q^\top s_k)=\frac{\exp(q^\top s_k)}{\sum_{j=1}^M \exp(q^\top s_j)},$$7 is fed directly into a small multi-layer perceptron,

$$\alpha_k=\mathrm{Softmax}(q^\top s_k)=\frac{\exp(q^\top s_k)}{\sum_{j=1}^M \exp(q^\top s_j)},$$8

and training uses a per-insight squared-error loss summed in a list-wise fashion over all insights in the same table. On the Financial Report dataset, adding CAM to TAR$$\alpha_k=\mathrm{Softmax}(q^\top s_k)=\frac{\exp(q^\top s_k)}{\sum_{j=1}^M \exp(q^\top s_j)},$$9 yields an absolute gain in Precision@1 from 0.162 to 0.170, mAP@5 from 0.756 to 0.772, and NDCG@5 from 0.815 to 0.829. The textual context is not fed into the memory network directly; it is used offline to produce weak-supervision scores.

In visual dialog, CAM appears as an augmentation to the MAC network of Hudson and Manning. “Reasoning Over History: Context Aware Visual Dialog” introduces two additions: Multi-Turn Memory, in which the MAC memory is not reset between turns, and Context-Aware Attention, in which the control unit at step $o=\sum_{k=1}^M \alpha_k I_k.$0 in turn $o=\sum_{k=1}^M \alpha_k I_k.$1 attends over all previous control outputs from turns $o=\sum_{k=1}^M \alpha_k I_k.$2 and steps $o=\sum_{k=1}^M \alpha_k I_k.$3 (Shah et al., 2020). The memory update across turns is defined so that the first step of turn $o=\sum_{k=1}^M \alpha_k I_k.$4 begins from $o=\sum_{k=1}^M \alpha_k I_k.$5 rather than from zero. Context-aware self-attention is computed over stacked control vectors, with lower-triangular masking, and fused using a gated combination of the original control and its attended summary. Experimentally, vanilla MAC reaches 65.9% accuracy on CLEVR-Dialog, MAC+CAA+MTM reaches 97.98%, and the best single-model configuration, MAC+CQ+CAA+MTM, reaches 98.25%. The paper reports that gains are largest on coreference-heavy question types and exceed 20% absolute at turns 9–10 compared to vanilla MAC. It also states a clear limitation: CLEVR-Dialog is synthetic and uses single-word answers.

These two early formulations already expose a central bifurcation in CAM design: one line treats context as a static set of peer items to be softly addressed; the other treats context as a temporally evolving reasoning trace.

4. Dynamic multimodal CAM in LVLMs and speech synthesis

In “Context-Aware Multi-Turn Visual-Textual Reasoning in LVLMs via Dynamic Memory and Adaptive Visual Guidance,” CAM is instantiated as the Visual-Textual Context Memory Unit (VCMU) and coupled to Adaptive Visual Focus Guidance (AVFG) (Shen et al., 6 Sep 2025). Raw visual features are projected to

$o=\sum_{k=1}^M \alpha_k I_k.$6

query text is embedded as

$o=\sum_{k=1}^M \alpha_k I_k.$7

and a small multi-modal encoder fuses $o=\sum_{k=1}^M \alpha_k I_k.$8 into a turn embedding

$o=\sum_{k=1}^M \alpha_k I_k.$9

AVFG consumes raw spatial features $M$0 and pooled memory context

$M$1

then produces an attention map

$M$2

The modulated features

$M$3

are flattened and passed onward. The paper states that coarse or fine spatial resolutions such as $M$4, $M$5, and $M$6 can be used, and that $M$7 works best in practice. For decoding, $M$8, $M$9, and $M_t\in\mathbb{R}^{N_m\times D_m},$0 are projected into a common decoder space and concatenated:

$M_t\in\mathbb{R}^{N_m\times D_m},$1

The ablation on LLaVA-1.5 isolates the contribution of the memory unit. Base performance is VisDial 76.5 CIDEr, A-OKVQA 61.2% Accuracy, MTIF 52.8% IFSR, and CCS 0.72. Adding VCMU only raises these to 77.8, 62.9%, 54.7%, and 0.75. AVFG only yields smaller gains. Full CAMVR reaches VisDial 78.9, A-OKVQA 64.3%, MTIF 56.5%, and CCS 0.78.

In “Long-Context Speech Synthesis with Context-Aware Memory,” CAM is a latent block inserted between the current sentence’s text encoder and the long-form LLM (Li et al., 20 Aug 2025). It fuses long-term memory $M_t\in\mathbb{R}^{N_m\times D_m},$2, local context $M_t\in\mathbb{R}^{N_m\times D_m},$3, and compressed current text $M_t\in\mathbb{R}^{N_m\times D_m},$4. Compression is implemented by a Perceiver Resampler, retrieval by cross-attention over a HistoryEncoder output, and update by a learned scalar gate. The paper introduces two updated memories, one for text and one for speech, which are used as prefix tokens. The prefix mask enables bidirectional attention on prefix tokens while maintaining unidirectional generation. On a 15 000 h Mandarin audiobook corpus, compared with baseline CosyVoice-LM, MMCE-Qformer[5 sent.], and CLAP-RAG[~900-token prompt], CAM raises Paragraph MOS from 3.468 to 3.796, CoMOS from 3.460 to 3.992, SpeechBERTScore from 77.776 to 80.448, reduces CER from 5.85% to 4.14%, and raises speaker similarity from 85.051% to 85.685%. It uses only 1 prior sentence + 64 memory tokens. Removing text memory drops MOS to 3.604 and raises CER to 5.04%; removing speech memory yields MOS 3.516 and CER 5.50%; reverting to a standard causal mask yields MOS 3.661 and CER 4.63%.

These multimodal variants move CAM beyond static lookup. Memory is not merely read; it actively reshapes future perception or generation, either by modulating visual attention or by altering the token-level conditioning regime of a decoder-only LM.

5. Retrieval-augmented and agentic CAM

“RAN Cortex: Memory-Augmented Intelligence for Context-Aware Decision-Making in AI-Native Networks” formulates CAM as a modular layer composed of four elements: a context encoder, a vector-based memory store of past network episodes, a recall engine, and a policy interface (Barros, 6 May 2025). The memory stores triples $M_t\in\mathbb{R}^{N_m\times D_m},$5 of past embedding, action, and outcome or reward. New entries are streamed into the ANN index as soon as outcomes are available, and the system supports rolling-window retention, Product Quantization, and memory shards rolled over by time or cell cluster. The architecture is designed for O-RAN compatibility: Near-RT RIC deployments target recall latency below 1 ms and total additional latency below 10 ms, while Non-RT RIC deployments target 100 ms–1 s. In the stadium-traffic mitigation example, the recall engine retrieves $M_t\in\mathbb{R}^{N_m\times D_m},$6 past episodes and the policy recommends a multi-carrier scheduling boost and provisional TDD reconfiguration. The empirical projection from pilot simulations reports 30% reduction in packet drops, 20% uplift in user-perceived throughput, and a 40% cut in SLA violation rate, without retraining the scheduler model.

“CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension” reframes CAM through Jean Piaget’s Constructivist Theory, emphasizing “structured schemata, flexible assimilation, and dynamic accommodation” (Li et al., 7 Oct 2025). At layer 0, incoming chunks are inserted into a weighted semantic graph using a score

$M_t\in\mathbb{R}^{N_m\times D_m},$7

Ego-centric disentanglement replicates nodes according to connected components of ego-networks, and incremental label propagation updates clusters on the active replica subgraph. Higher layers are formed by LLM-generated summaries of clusters, producing a hierarchical graph of representations. Query-time retrieval first localizes top-$M_t\in\mathbb{R}^{N_m\times D_m},$8 nodes by cosine similarity and then performs “Prune-and-Grow” associative exploration. Across six long-text benchmarks, the paper reports NovelQA ACC-L = 52.3% versus best baseline 47.8%, QMSum ROUGE-1 = 37.2 and ACC-L = 57.6%, MultiHop-RAG F1 = 77.5%, and ODSum-Story ROUGE-1 = 39.2. For efficiency, CAM remains under 0.25 h even for batch insertion of 500 chunks and converges toward its offline 0.3 h cost, while offline methods such as RAPTOR and GraphRAG take approximately 1–1.8 h regardless of batch size.

These systems show a further shift in CAM design: memory becomes a systems primitive that can be attached to existing policies or LLMs, rather than only an internal latent state of a task-specific network.

6. Comparative interpretation, common confusions, and unresolved issues

A common source of confusion is the assumption that CAM denotes a single standardized memory architecture. The literature does not support that interpretation. The table-ranking CAM is statically populated and read-only; the visual-dialog CAM stores prior control states and carries MAC memory across turns; CAMVR uses a fixed-size memory matrix with gated overwrite and no explicit “filling pointer”; RAN Cortex uses a vector-based memory store with ANN retrieval and rolling-window policies; constructivist CAM uses hierarchical graph construction and online clustering updates (Zeng et al., 2018, Shah et al., 2020, Shen et al., 6 Sep 2025, Barros, 6 May 2025, Li et al., 7 Oct 2025).

A second confusion concerns the role of “context.” In some systems, context is encoded as latent vectors rather than explicit text. VCMU stores visual features, textual semantic representations, and cross-modal correspondences; visual-dialog CAM attends over control states; TTS CAM stores fixed-length latent memory plus local text and speech history; RAN Cortex stores embeddings, actions, and outcomes rather than raw telemetry (Shen et al., 6 Sep 2025, Shah et al., 2020, Li et al., 20 Aug 2025, Barros, 6 May 2025). Conversely, in the 2018 ranking model, textual context is not stored in memory at all; it is used offline to derive weak-supervision scores (Zeng et al., 2018).

The limitations also differ sharply. Visual-dialog CAM is validated on a synthetic benchmark with single-word answers and may face cost growth for very long dialogs or large reasoning depth (Shah et al., 2020). Constructivist CAM notes that LLM summaries may invent details, that inconsistent sources remain an open challenge, and that its current policies use fixed hyperparameters and prompt templates (Li et al., 7 Oct 2025). CAMVR’s VCMU works under fixed capacity $M_t\in\mathbb{R}^{N_m\times D_m},$9 and implicit slot allocation, which suggests a trade-off between bounded memory and explicit control over slot usage (Shen et al., 6 Sep 2025). In long-context TTS, the published method uses only the cross-entropy loss on predicted speech tokens and no auxiliary memory-specific loss (Li et al., 20 Aug 2025).

Taken together, these works position CAM as a broad research direction centered on context-conditioned persistence. The unifying idea is not a particular tensor layout or indexing scheme, but the claim that strong performance in multi-turn reasoning, long-context generation, decision continuity, and long-document comprehension depends on preserving and selectively reactivating historical structure. The divergence of implementations across ranking, dialog, LVLMs, RAN control, speech synthesis, and reading comprehension suggests that future CAM research will likely continue to be domain-shaped rather than architecture-universal.