Semantics-Aware Memory Hierarchy

• Semantics-aware memory hierarchy is a design that organizes memory based on semantic structure rather than flat byte storage.
• It leverages physical and operational signals—from bounded state-space dynamics to token surprisal—to guide multi-tier placement and dynamic retrieval.
• Researchers use these hierarchies to boost efficiency, reduce memory overhead, and improve inference in language and multi-modal systems.

A semantics-aware memory hierarchy is a class of memory organizations in which units are not managed as a flat pool of bytes, tokens, or pages, but are segmented, placed, compressed, migrated, or retrieved according to meaning-bearing structure. In the literature, the relevant semantic signal varies by domain: bounded state-space structure and engine performance in a physical account of semantics; delimiters, semantic chunks, spans, or document trees in language systems; task vectors and semantic manifolds in multi-modal in-context learning; cumulative attention in reasoning traces; semantic block types and transition context in KV-cache serving; temporal and topic structure in agentic memory; and object identity, bounds, and lifetime in descriptor-based architectures (Koleva, 2010, Wu et al., 15 Mar 2026, Chen et al., 10 Jun 2026, Abaskohi et al., 30 Jun 2026, Yuan et al., 10 May 2026, Ganjihal, 19 Apr 2026, Tian et al., 13 Jan 2026, Tong, 31 Oct 2025).

1. Physical and conceptual foundations

One early formulation treats semantics itself as a physical property, but only under the hypothesis of boundedness, defined as bounded local accumulation of matter/energy and bounded exchange rate with the environment (Koleva, 2010). Under that hypothesis, the state space of an open system is divided into basins of attraction; adjacent states are the only allowed transitions; trajectories form a dense transitive set of orbits; and motion on inter-basin orbits becomes non-Markovian. In that setting, a semantic unit is identified with an inter-basin orbit, while the average point at which memory is “lost” functions as a separator analogous to a space bar (Koleva, 2010).

This account assigns every semantic unit an irreducible two-fold representation: as a specific sequence of letters and as the performance of a specific engine (Koleva, 2010). The sequence is the algorithmic or syntactic side; the engine’s useful work and efficiency encode semantic content. The resulting hierarchy is multi-layered: a unit at one level is a sequence of lower-level units, but on the next level that whole unit becomes a “letter” for a higher structure, with meaning again represented by another engine (Koleva, 2010). The same framework attributes three physical signatures to semantics—non-extensivity, permutation sensitivity, and Zipf’s law—while maintaining hierarchy by matter-wave-mediated non-local feedback (Koleva, 2010).

This physical account is not a computer-memory design in the conventional sense. It does, however, provide an unusually explicit statement of the general principle later seen in systems work: semantic hierarchy is not merely storage of symbols, but an organization in which state transitions, boundaries, and higher-level composition are part of the memory substrate itself (Koleva, 2010). A plausible implication is that later engineering systems inherit the same structural ambition even when their “semantics” are operational rather than ontological.

2. What “semantics-aware” denotes across subfields

The term is polysemous. In large-scale GPU inference serving, “semantics-aware” means that KV cache should not be treated as a generic blob of bytes; the system uses what the cache entry means—its attention architecture, semantic block type, transition context, reuse probability, and positional locality—to decide how large it is, where it should live, when it should move, and what to evict (Ganjihal, 19 Apr 2026). This is architecture-variant-aware and workload-aware semantics rather than linguistic semantics.

In long-context and multimodal inference, the semantic signal is usually content structure. SemantiCache aligns compression with the semantic hierarchical nature of language by using delimiter-based semantic chunking, Greedy Seed-Based Clustering, semantic cores, and Proportional Attention (Wu et al., 15 Mar 2026). SeKV defines semantic spans from token surprisal and stores each span as a GPU-resident summary plus a CPU-resident low-rank SVD basis (Abaskohi et al., 30 Jun 2026). H$^{2}$MT builds a tree whose nodes correspond to semantically coherent units such as sections, subsections, paragraphs, and chunks, and stores a memory embedding for each node (Haghifam et al., 24 May 2026). SwiftMem organizes agent memory along the two axes that many queries follow—time and meaning/topic—through a temporal index, a semantic DAG-Tag index, and embedding-tag co-consolidation (Tian et al., 13 Jan 2026).

In reasoning LLMs, semantics-awareness can refer to the observation that “cold” chain-of-thought tokens are not safely disposable. The four-tier hierarchy of HBM, DDR, compressed, and evicted tiers ranks tokens by cumulative attention, but treats low-importance tokens as preservable state rather than garbage (Yuan et al., 10 May 2026). In older paging work, the semantic content is weaker but still present: the N-level Aging policy uses reference-history semantics—recency, frequency, and inactivity encoded in counters—to guide migration across tiers (Oren, 2017). In data-aware microarchitecture, semantics extends to program and system behavior itself: PCs, deltas, page number, offsets, values, bandwidth usage, cache pollution, and stable address/value repeatability drive prefetching, off-chip prediction, coordination, and load elimination (Bera, 8 Mar 2026).

Descriptor-based object-aware memory systems use yet another sense of the term. Here the semantic unit is the software-defined object, and the descriptor carries object identity, bounds, permissions, address translation metadata, lifetime, and optional semantic tags across the hardware/software boundary (Tong, 31 Oct 2025). This suggests that semantics-aware memory hierarchy is not a single technique, but a design family defined by whether placement and retrieval decisions are conditioned on interpreted structure rather than anonymous storage addresses.

3. Canonical organizational patterns

Across the literature, semantics-aware hierarchies recur in a small set of organizational patterns: semantic segmentation, semantic aggregation, tiered placement, recoverable compression, query-conditioned routing, and fault- or feedback-driven reactivation.

System	Semantic unit or signal	Hierarchy
TASM	task vector; visual manifold	Core Memory / Latent Bank
SemantiCache	delimiters, chunks, clusters	delimiters $\rightarrow$ chunks $\rightarrow$ semantic cores
SeKV	surprisal-defined spans	GPU summaries / CPU SVD bases
H$^{2}$MT	document-tree nodes	offline semantic tree / coarse-to-fine routing
SwiftMem	temporal ranges; DAG tags	temporal index / semantic DAG / consolidated embedding store
Reasoning KV hierarchy	cumulative attention on tokens	HBM / DDR / compressed / evicted

Semantic segmentation identifies the unit of memory management. In SemantiCache, the cache is partitioned into an alternating sequence of chunks and delimiters, with the delimiter set defined as $[ ".", ",", "?", "!", ";", ":", " ", "\t", "\n"]$ (Wu et al., 15 Mar 2026). In SeKV, boundaries are induced by high surprisal, with anchor tokens retained when $H_t > \mu + \alpha \sigma$ (Abaskohi et al., 30 Jun 2026). In H$^{2}$MT, the unit is the node of a rooted semantic tree $\mathcal{T}=(V,E,r)$ (Haghifam et al., 24 May 2026).

Semantic aggregation compresses a unit into a representation that can participate in higher-level retrieval. SemantiCache mean-pools each cluster into a semantic core and reweights attention by cluster size through Proportional Attention (Wu et al., 15 Mar 2026). H$^{2}$MT computes leaf memories from a write/read-tokenized local text sequence and computes internal-node memories bottom-up by post-order aggregation over child memories (Haghifam et al., 24 May 2026). SeKV stores both a normalized summary key for routing and surprisal-weighted mean keys and values for coarse attention (Abaskohi et al., 30 Jun 2026).

Tiered placement then separates active memory from latent or cheaper storage. TASM makes this explicit with a compact GPU-resident Core Memory and a larger CPU-resident Latent Bank (Chen et al., 10 Jun 2026). The reasoning hierarchy separates T0 HBM, T1 DDR, T2 compressed, and T3 evicted tokens (Yuan et al., 10 May 2026). The predictive multi-tier serving system generalizes the idea to six tiers—GPU HBM, CPU DRAM, CXL 3.0 memory, NVMe via GPUDirect Storage, RDMA fabric, and parallel filesystem—with architecture-aware block sizes and Bayesian reuse prediction (Ganjihal, 19 Apr 2026).

Recoverable compression distinguishes several recent systems from earlier token pruning. SeKV stores a truncated SVD basis per span on CPU and reconstructs token-level keys and values on demand (Abaskohi et al., 30 Jun 2026). TASM preserves low-score visual tokens by semantics-aware token merging via bipartite graph matching rather than destructive pruning (Chen et al., 10 Jun 2026). Pichay pages out addressable content but keeps a backing store so that page faults can restore it later (Mason, 9 Mar 2026).

Finally, query-conditioned routing and reactivation govern when lower tiers are touched. TASM uses Jensen-Shannon divergence between the current and previous attention distributions over core memory to trigger retrieval from the latent bank (Chen et al., 10 Jun 2026). H$^{2}$MT routes queries coarse-to-fine by per-parent top-$\rightarrow$0 traversal over node memories (Haghifam et al., 24 May 2026). SwiftMem maps a query to relevant tags with cosine similarity, expands breadth-first through a semantic DAG, and narrows temporal queries through $\rightarrow$1 timeline search (Tian et al., 13 Jan 2026). Pichay detects a page fault when the model re-requests content that was previously evicted, and pins pages that have faulted back in (Mason, 9 Mar 2026).

4. Long-context, reasoning, and multimodal instantiations

SemantiCache is a semantics-first KV compression framework for long-context inference. Its three-stage pipeline—delimiter-based semantic chunking, Greedy Seed-Based Clustering, and clustered merging into semantic cores with Proportional Attention—was evaluated on LongBench and Needle-in-a-Haystack using Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2 (Wu et al., 15 Mar 2026). On LongBench, the reported averages for Llama-3-8B are 30.01 / 32.17 / 32.34 for SemantiCache versus 29.23 / 31.22 / 31.86 for D2O and 34.88 for the full model; for Mistral-7B, SemantiCache reports 39.68 / 40.09 / 40.87 versus 39.05 / 39.56 / 40.02 for D2O and 42.01 for the full model (Wu et al., 15 Mar 2026). Under a 20% KV cache budget on Llama-3-8B with 32k context, SemantiCache reports TTFT 4.25 s, TPOT 0.031 s, and memory 15.94 GB, with up to 2.61× speedup in TPOT (Wu et al., 15 Mar 2026).

TASM addresses dynamic multi-modal in-context learning by replacing rigid token removal with task-vector guided compression, semantics-aware token merging via bipartite graph matching, and a hierarchy of Core Memory and Latent Bank (Chen et al., 10 Jun 2026). Its experiments report up to 80–85% memory reduction while preserving performance; on ImageNet-100 with 200 examples it reduces average context length from 16264 to 3485, a 78.6% reduction; and on V-NIAH it remains far more robust than pruning methods because lost details are kept in the latent bank and retrieved when needed (Chen et al., 10 Jun 2026). The paper attributes gains on ScreenSpot and RefCOCOg to preservation of the visual manifold and reports that token merging significantly helps spatial tasks, while dynamic retrieval adds a large gain for temporal reasoning (Chen et al., 10 Jun 2026).

SeKV introduces a resolution-adaptive semantic KV cache in which each semantic span retains a lightweight summary vector on GPU and a low-rank SVD basis on CPU for token-level reconstruction (Abaskohi et al., 30 Jun 2026). The base LLM is fully frozen; only the routing projections, zoom thresholds, and shared rank-gate predictor are trained, totaling approximately 4.3M added parameters, about 0.05% of the base model (Abaskohi et al., 30 Jun 2026). Across LongBench, RULER, InfiniteBench, Needle-in-a-Haystack, and many-shot GSM8K, SeKV improves over SentenceKV by 5.9% on average at the same 10% GPU KV budget and reduces GPU memory by 53.3% compared to FullKV at 128K context (Abaskohi et al., 30 Jun 2026). The reported gains are especially strong on retrieval-heavy tasks, including +5.68 on NIAH and +3.63 on RULER (Abaskohi et al., 30 Jun 2026).

For reasoning LLMs, the central result is that offloading and eviction are not equivalent. The four-tier HBM/DDR/compressed/evicted hierarchy scores tokens by cumulative attention, periodically repartitions every $\rightarrow$2 decoding steps, protects all prompt tokens, the first $\rightarrow$3 reasoning tokens, and the most recent $\rightarrow$4 tokens, and proves a “Zero-Error DDR Offloading” proposition: if all T1 tokens are prefetched to GPU before attention computation, attention output is independent of whether they were stored in HBM or DDR (Yuan et al., 10 May 2026). In a controlled 3×3 grid over HBM ratios 30%, 50%, 70% and eviction ratios 3%, 5%, 10%, accuracy varies only slightly across HBM settings within a fixed eviction ratio, while changing far more strongly across eviction ratios (Yuan et al., 10 May 2026). At 3% eviction, the hierarchy retains 91.5% of the full-cache GSM8K baseline and 71% on MATH-500; at 14B scale it reports 90.0% versus an 86.0% full-cache baseline while offloading half the KV cache; and a real GPU–CPU prototype reports transfer overhead of only 5–7% of total inference time (Yuan et al., 10 May 2026).

H$\rightarrow$5MT addresses a different part of the long-context problem. Rather than compressing a flat token stream, it builds a reusable semantic hierarchy offline and routes queries coarse-to-fine at inference (Haghifam et al., 24 May 2026). On the TTFT axis, the paper reports large reductions relative to HMT: for example, Qwen2.5-14B averages 2897.49 ms for HMT versus 468.92 ms for H$\rightarrow$6MT, Qwen2.5-7B reports 6398.96 ms versus 276.89 ms, and Mistral-7B reports 8111.14 ms versus 398.85 ms (Haghifam et al., 24 May 2026). Peak GPU memory also decreases in the reported averages, such as 56.85 GB to 42.90 GB for Qwen2.5-14B and 31.17 GB to 24.33 GB for Llama3.1-8B (Haghifam et al., 24 May 2026).

5. Operating-system, storage, and hardware interpretations

Pichay reframes context management as memory management. It treats the context window as L1 cache, defines L2 as the working set, L3 as session history compressed into model-authored summaries with declared losses, and L4 as cross-session persistent memory (Mason, 9 Mar 2026). In production measurements over 857 sessions, 54,170 API calls, and 4.45 billion effective input tokens, it reports 21.8% structural waste, broken into unused tool schemas (11.0%), duplicated content (2.2%), and stale tool results (8.7%) (Mason, 9 Mar 2026). Offline replay over 1.393 million simulated evictions reports 354 page faults and a fault rate of 0.0254%; live deployment reports context reduction by up to 93% from 5,038 KB to 339 KB; and under sustained pressure it exhibits repeated fault-in of evicted content, explicitly identified as thrashing (Mason, 9 Mar 2026). The proxy acts as an MMU analogue: it evicts stale content, detects faults when the model re-requests it, and pins working-set pages identified by fault history (Mason, 9 Mar 2026).

At the serving-system level, predictive multi-tier KV management extends the hierarchy to six physical tiers and adds semantics-aware sizing and reuse prediction (Ganjihal, 19 Apr 2026). The sizing engine distinguishes MHA, GQA/MQA, and MLA; for DeepSeek-V3 it computes MLA per-token per-layer cache as 1,152 bytes versus 65,536 bytes for MHA-equivalent sizing, a 57× compression (Ganjihal, 19 Apr 2026). On an 80 GB H100 with 30 GB KV budget and $\rightarrow$7, the reported maximum batch size for DeepSeek-V3 rises from 15 under MHA-equivalent sizing to 104 under MLA-aware sizing, which is the source of the up to 7.4× higher batch sizes claim (Ganjihal, 19 Apr 2026). The full six-tier hierarchy extends effective KV cache capacity from 40 GB to over 38 TB per node, while trace replay reports Bayesian hit rates of 69.8% on ShareGPT, 84.2% on LMSYS-Chat-1M, and 80.5% on an agentic workload (Ganjihal, 19 Apr 2026). Analytical projections report 1.4–2.1× TTFT reduction, 1.7–2.9× throughput improvement, and 47% cost reduction relative to stated baselines (Ganjihal, 19 Apr 2026).

SwiftMem applies semantics-aware hierarchy to agentic long-term memory. Its temporal index provides $\rightarrow$8 range queries and $\rightarrow$9 episode metadata access, while the semantic DAG-Tag index supports query cost $\rightarrow$0 (Tian et al., 13 Jan 2026). On LoCoMo, SwiftMem reports LLM score 0.704, F1 0.429, BLEU-1 0.467, and search latency 11 ms versus 522 ms for Zep, 835 ms for Nemori, and 19,829 ms for LangMem (Tian et al., 13 Jan 2026). The paper states a 47× speedup and further reports that co-consolidation preserves evidence recall at 90.5% while improving LLM Judge score from 64.3% to 78.6% and reducing latency from 10.2 ms to 7.4 ms (Tian et al., 13 Jan 2026).

Longer historical lineages used weaker semantic signals but pursued the same tier-matching principle. The automated Memory Allocation Manager work adapts NRU, FIFO, and especially Aging to N-level memory hierarchies and reports that the adapted Aging algorithm yields the best hit/miss ratio by mapping the number of leading zeros in the aging counter to a destination memory level (Oren, 2017). The graph-layout literature treats graph connectivity and traversal behavior as the relevant semantics: the Hierarchical Blocking Algorithm takes locality units $\rightarrow$1 as input and rearranges nodes by hierarchical BFS-based copying, reporting about 54% query-time reduction on a depth-25 binary search tree, about 29% speedup in Jikes for all-level HBA, and up to 21.31× for BFS on a 4-ary tree (Roy, 2012).

At the hardware/software boundary, descriptor-based object-aware systems make semantics explicit by elevating the descriptor to a first-class architectural abstraction carrying object identity, bounds, permissions, translation metadata, lifetime, and optional semantic tags (Tong, 31 Oct 2025). The survey positions this as foundational for future cache hierarchies, unified virtual memory, and even 128-bit architectures (Tong, 31 Oct 2025). Data-aware microarchitecture extends semantics-awareness further downward: Pythia, Hermes, Athena, and Constable use program and system data to drive prefetching, off-chip prediction, coordination, and load elimination, with reported average performance improvements of 22.4% for Pythia, 11.5% for Hermes-O, and 5.1% for real Constable, while 34.2% of dynamic loads are reported as global-stable on average (Bera, 8 Mar 2026).

6. Limits, misconceptions, and open questions

A major theoretical limit is that organizing memory by meaning creates interference. For semantically continuous kernel-threshold memories—systems whose retrieval score is a monotone function of an inner product in a semantic feature space with finite local intrinsic dimension—the literature derives four structural results: semantically useful representations have finite effective rank, finite local dimension implies positive competitor mass, retention decays to zero under growing memory, and false recall for $\rightarrow$2-convex associative lures cannot be eliminated by threshold tuning alone (Barman et al., 28 Mar 2026). The retention expression

$\rightarrow$3

formalizes the claim that as competitors accumulate, retention goes to zero (Barman et al., 28 Mar 2026). Experiments reported across vector retrieval, graph memory, attention-based context, BM25 retrieval, and parametric memory show that pure semantic systems exhibit smooth forgetting and false recall, while reasoning-augmented systems can hide the vulnerability but convert graceful degradation into catastrophic failure (Barman et al., 28 Mar 2026).

Several recurrent misconceptions follow from this. Zipf behavior is not evidence of meaning by itself: bounded irregular sequences, including meaningless bounded sequences, can exhibit the same $\rightarrow$4 spectral form and inverse-rank statistics, so the significance of Zipf’s law in the physical account is that boundedness yields a universal statistical skeleton on which semantic hierarchy can be built (Koleva, 2010). Likewise, compression is not equivalent to lossless tiering: the reasoning-cache hierarchy proves that moving tokens between HBM and DDR causes zero approximation error if T1 tokens are prefetched before attention, and that the only source of error is permanent eviction of T3 tokens (Yuan et al., 10 May 2026). This distinction recurs in SeKV’s recoverable SVD bases, TASM’s latent bank, and Pichay’s addressable page faults (Abaskohi et al., 30 Jun 2026, Chen et al., 10 Jun 2026, Mason, 9 Mar 2026).

Open questions are explicit in the literature. Pichay identifies cross-session memory as the remaining frontier, notes that permanent pinning may cause monotonic growth of the working set, and proposes cost-weighted pin decay as a future refinement (Mason, 9 Mar 2026). The descriptor-based survey argues that explicit cross-layer communication of object semantics is a foundational research direction for next-generation cache hierarchies, unified virtual memory, and 128-bit architectures (Tong, 31 Oct 2025). The interference theory identifies three escape routes—abandon semantic continuity, add an external symbolic verifier or exact episodic record, or push semantic effective rank to infinity—but treats the last option as unrealistic for natural language and practical memory systems (Barman et al., 28 Mar 2026). This suggests that future semantics-aware hierarchies are likely to remain hybrid systems: partly semantic, partly symbolic, and explicitly tiered between active working sets, latent stores, and recoverable or exact records.