Hierarchical Paging Techniques

• Hierarchical paging is a multi-tier memory management approach that automatically migrates data blocks based on temperature metrics to optimize performance.
• It extends traditional two-level models with N-level aging algorithms, significantly reducing page faults and enhancing hit ratios across diverse systems.
• The technique is implemented in both classical architectures and emerging LLM systems using advanced methods like multi-level page tables and reinforcement learning.

Hierarchical paging is a generalization of traditional two-level virtual memory and caching schemes in which multiple memory/storage layers, with distinct performance and capacity characteristics, are organized into a managed hierarchy. This paradigm is essential in both classical operating systems and emerging domains such as storage-class memory (SCM), large-scale LLM context management, and complex agent systems, where resource constraints and performance requirements dictate the need for multi-tiered, automated page management. Hierarchical paging enables automatic and efficient migration, eviction, and promotion of data blocks ("pages") across storage levels—from ultra-fast on-chip memory to persistent disk—to match access patterns, maximize hit rates, and minimize latency.

1. Memory and Storage Hierarchy Motivations

Traditional virtual memory models treat main memory (DRAM) and secondary storage (disk/SSD) as two distinct levels, separated by orders of magnitude in latency (10,000×) and bandwidth (∼200×) (Oren, 2017). Storage-Class Memory (SCM)—such as phase-change memory, MRAM, or Nano-Ionic RAM—introduces persistent, byte-addressable devices with latency between DRAM and disk, creating the need for at least three genuine performance tiers that cannot be efficiently collapsed into a flat address space. Similarly, in LLM systems, the context window functions as high-cost, low-capacity L1 cache, lacking the hierarchical demand paging found in classical systems (Mason, 9 Mar 2026). Hierarchical paging organizes and automates data movement across such multilayered structures; access and locality patterns are leveraged to improve effective cache capacity and performance.

2. General Principles and N-Level Paging Algorithmics

Hierarchical paging algorithms extend classical paging methods—such as Least Recently Used (LRU), Aging, or Belady's optimum—from flat two-level models to N-level hierarchies. In an N-level system, pages are migrated between tiers based on “temperature” metrics (recency, frequency, or predictive utility), with the goal of keeping “hot” pages near the top (e.g., DRAM), and demoting “cold” pages downward as their apparent value decays (Oren, 2017).

A canonical example is N-level Aging. Each page maintains an m-bit shift register $A_i$ as a recency counter, updated on each timer tick:

$$A_i^{(t+1)} = (A_i^{(t)} \gg 1) + R_i^{(t)} \cdot 2^{m-1}$$

Eviction and demotion decisions use the number $Z_0$ of leading zeros in $A_i$ to map to a target tier $\ell^*$:

$$\ell^* = \min\Big(N, \Big\lceil \frac{Z_0}{m} N \Big\rceil\Big)$$

This smoothly demotes "cold" pages further down the hierarchy, achieving significantly improved HitRatio over flat paging (Oren, 2017).

3. Hierarchical Paging in Classical Architectures

On x86-64 and similar architectures, hierarchical paging is realized via multi-level radix-tree page tables, typically four levels deep for 4 KB pages, and hardware-managed page walks on Translation Lookaside Buffer (TLB) misses (Patil, 2020). Each virtual address is partitioned into indices selecting entries from PML4, PDPT, PD, and PT tables, with each table lookup potentially causing a cascade of memory accesses on a miss.

Extensions to the TLB and page-walk mechanisms—including superpages, multi-level page-walk caches, and dynamic TLB sizing—are employed to align virtual memory management with the underlying physical hierarchy (including die-stacked DRAM caches and persistent memory). Quantitative simulation demonstrates that, even as on-chip caches grow into gigabyte regimes, address-translation (paging) overheads persist unless TLB reach and page-walk bandwidth are explicitly scaled. No combination of LLC size and TLB organization alone suffices; instead, deeper, more flexible hierarchical translation and caching are prerequisite for sustained low-latency operation (Patil, 2020).

4. Hierarchical Paging in Machine Learning Systems

In LLM systems, context windows are currently managed as single-level L1 caches, resulting in waste (21.8% structural input-token overhead across large-scale production workloads) (Mason, 9 Mar 2026). The "Pichay" demand-paging proxy reinstates a full memory hierarchy:

• L1 (context window): small, fast, attended each API call.
• L2 (working set): pages that have faulted in and are now “pinned.”
• L3 (history): compacted, summarized session data.
• L4 (cross-session): persistent semantic indices and embeddings.

Eviction uses age-based FIFO with thresholds; page-faults trigger retrieval from persistent storage and one-fault-one-pin promotion to prevent repeated faults on reused content. Across 1.4 million simulated evictions, the page-fault rate is 0.0254%. Context consumption reductions up to 93% have been measured in live production deployment (Mason, 9 Mar 2026).

Theoretical formalization and empirical analysis in "Neural Paging" generalize the paging concept to Turing-complete agent architectures:

• The Context Paging Problem (CPP) is defined as optimizing cumulative reward (prediction accuracy minus eviction and fetch costs) given a bounded window $K$ and unbounded external memory (Chen et al., 11 Feb 2026).
• The Page Controller seeks to approximate the offline-optimal (Semantic Belady) eviction, maintaining a distribution over {KEEP, EVICT, PREFETCH} actions, trained via PPO.
• Complexity analysis demonstrates reduction from $O(N^2)$ to $O(NK^2)$ for long-horizon inference, with formal robustness bounds under policy-dependent access.
• Synthetic trace experiments confirm practical competitive ratios far below $K_b$, indicating learnable policies can outperform classic heuristics in realistic regimes (Chen et al., 11 Feb 2026).

5. Simulation Methodologies and Key Performance Metrics

The efficacy of hierarchical paging schemes is systematically validated via synthetic trace-based simulators:

• DeMemory (Oren, 2017) implements multi-level frame management and measures per-level hit/miss frequencies, providing HitRatio and MissRatio as primary metrics.
• Similar synthetic evaluation frameworks benchmark LRU, LFU, FIFO, and learned policies (e.g., PPO-trained controllers) against Belady’s theoretical optimum (Chen et al., 11 Feb 2026).

Empirical findings (Oren, 2017):

• 3-level Aging achieves up to 3× lower miss ratio than two-level LRU/Aging.
• Direct mapping of demotion targets via $$A_i^{(t+1)} = (A_i^{(t)} \gg 1) + R_i^{(t)} \cdot 2^{m-1}$$0 index yields an additional 10–20% HitRatio improvement over non-hierarchical variants.
• In regimes where DRAM cannot hold the full working set, N-level schemes offset the performance penalty of cheaper intermediate tiers (e.g., SCM).

In LLM context paging, key metrics include page-fault rate (0.0254% in large scale (Mason, 9 Mar 2026)), context amplification factor, and actual reduction in context utilization.

6. Practical Implications and Future Directions

Hierarchical paging enables:

• Automatic, near-optimal placement of data blocks across modern memory/storage hierarchies without explicit programmer or application intervention.
• Significant effective cache expansion (2–3×) in HPC workloads, simply by inserting SCM as an intermediate tier and adopting hierarchical management (Oren, 2017).
• Sustained context window performance in LLMs and agent systems as working-set sizes (and global model capacities) increase (Mason, 9 Mar 2026, Chen et al., 11 Feb 2026).

Future advances, as identified by recent studies, are focused on:

• Dynamic TLB resizing and multi-dimensional page-walk optimization for classical hardware (Patil, 2020).
• Neural approximation of semantic-optimal paging policies, leveraging reinforcement learning with robust performance guarantees (Chen et al., 11 Feb 2026).
• Modular user-level paging interfaces for memory-aware data structures and agent frameworks, enabling fine-grained control over tier residency in exascale and agent-oriented computing (Oren, 2017).

7. Summary Table: Hierarchical Paging Across Domains

Domain	Hierarchy Example	Key Policy / Metric
Classical OS, HPC	DRAM → SCM → Disk	N-level Aging, HitRatio
x86-64 Hardware	Multi-level radix tree, TLB, die-stacked	TLB miss, Pagewalk latency
LLM/AI Systems	L1 context → working-set → summaries → disk	FIFO+pinning, Page-fault rate
Agent Architectures	L1/L2 context cache → external memory	RL-trained page controller

These applications underscore hierarchical paging’s critical role across system design, hardware-software interface, and emerging machine learning contexts, providing fundamental structure for scalable, adaptive memory management in contemporary and future architectures (Oren, 2017, Patil, 2020, Mason, 9 Mar 2026, Chen et al., 11 Feb 2026).