ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories

Abstract: The rapid shift toward agentic and long-context workloads in LLMs is pushing the industry beyond the capacity of individual servers toward disaggregated shared storage to handle TB-scale context states. This movement has led to the emergence of specialized shared context layers designed to externalize and share cumulative inference states across distributed clusters. While offloading to a data processing unit (DPU) within just-a-bunch-of-flash (JBOF) architectures accelerates NVMe-over-fabrics (NVMe-oF) target processing, the need for sophisticated software-level optimization and cost-efficiency burdens remain significant. Consequently, the ideal architecture for scaling this shared context infrastructure is still an active area of exploration. In this paper, we propose ITME (Inference Tiered Memory Expansion), which leverages a CXL-hybrid memory to present a massive, TB-scale byte-addressable remote memory expansion. This approach enables cost-efficient scaling and simplifies the software stack through direct byte-addressability, effectively addressing the challenges of shared context infrastructure. Our key insight is that the deterministic access patterns of voluminous model weights and prefix caches enable the system to proactively manage data movement across the memory-storage hierarchy. We validate ITME by evaluating its performance potential with production-grade SK Hynix CMM and PCIe Gen5 NVMe SSDs, while further demonstrating its functional feasibility through an FPGA-based hardware prototype. Overall, ITME enhances conventional CPU-offloading by providing additional remote memory expansion to accommodate large KV cache footprints beyond host memory limits, achieving up to a 35.7\% throughput improvement.

• The paper presents ITME, a novel architecture that leverages disaggregated CXL-hybrid memory to scale LLM inference memory capacity.
• It employs pipelined, layer-wise prefetch and hardware-managed scheduling to mitigate I/O latency and optimize data flow.
• Performance evaluations demonstrate up to 1.81× TTFT speedup and a 35.7% improvement in multi-turn conversation throughput.

ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories

Motivation and Problem Context

The exponential growth of LLM size and the increasing prevalence of agentic and long-context workloads have fundamentally shifted the primary bottleneck in inference infrastructure from compute to memory capacity and data movement. As models such as Llama-3.1 70B and OPT-175B require hundreds of gigabytes for weights and terabyte-scale key-value (KV) cache footprints during multi-turn, long-context execution, traditional server-centric memory architectures—comprising high-bandwidth GPU memory and host DRAM—are inadequate and economically unscalable.

Contemporary inference hierarchies rely on a tiered, multi-device memory/storage structure, spanning from GPU HBM, host DRAM, local and remote NVMe SSDs, and shared storage. However, new challenges arise in efficient context reuse, amortization of storage, and latency management, especially for long-context and multi-turn conversational workloads. JBOF architectures utilizing DPUs and NVMe-oF provide raw capacity but incur high overhead from complex, heavily software-managed stacks, creating significant engineering and total cost of ownership burdens. Thus, industry proposals such as the NVIDIA CMX context memory platform and CXL-based disaggregation are attempting to reshape the ecosystem for scalable, low-latency, shared context management.

Architectural Overview: CXL-Hybrid Memory for Tiered Expansion

The core proposal is ITME: an infrastructure that inserts a CXL-hybrid memory based tier between host DRAM and backend SSDs. ITME exposes remote, byte-addressable, terabyte-scale CXL-hybrid memory pools to inference nodes via standard RDMA/NICs, thereby decoupling memory scaling from server boundaries and bypassing the complexity of DPU-centric storage stacks.

Figure 1: A detailed structural comparison of classic and ITME-enabled memory hierarchies, the latter featuring an intermediate shared CXL-hybrid tier.

Internally, the CXL-hybrid memory device couples high-density NAND flash with a hardware-managed DRAM cache, accessed via a PCIe Gen5 interface, supporting direct CPU/GPU memory access through cache-coherent CXL protocols. The deterministic access patterns of LLM weights and long-context prefix KV caches offer a unique opportunity: by exploiting known, layer-consistent data requests, ITME can coordinate pipelined DMA prefetching, moving large, predictable blocks between tiers with minimal latency exposure.

Figure 2: CXL-Hybrid Memory Architecture integrating SSDs and DRAM as a hardware-managed cache, exposed to host CPUs and GPUs as byte-addressable memory via CXL.

The explicit user-level prefetch APIs provided by ITME’s software stack enable LLM inference frameworks to trigger hardware-managed prefetches, decoupling I/O latency from runtime critical paths and eliminating software-level paging and translation overheads.

Figure 3: Example walkthrough of user-directed prefetching through ITME’s low-overhead API which interacts directly with device registers.

ITME Data Path and Prefetch Pipeline

The ITME I/O pipeline leverages a multitiered prefetch and eviction mechanism, orchestrating movement of model weights and KV blocks between GPU memory, host CPU staging buffers, and remote CXL-hybrid memory. Two key design points underpin performance:

1. Pipelined Layer-wise Prefetching: Weights are transferred in large, sequential blocks. During the execution of layer $L$, weights for $L+1$ are proactively staged, so I/O and compute overlap. Prefix KV caches, characterized by multi-turn reuse and append-dominated access, are likewise fetched in aggregated units keyed to expected execution order.
2. Hardware-managed Scheduling and Bandwidth Alignment: To avoid read stalls, ITME regulates asynchronous writes to NAND so that high-priority read requests always drain first. On cache misses, ITME prefers recomputation to remote retrieval to avoid performance collapse due to I/O amplification.

   Figure 4: Evaluation of sustained bandwidth across DMA paths, showing the scaling to near-hardware limits for large block sizes critical in LLM workloads.

The staging buffer size and the DRAM cache on the CXL-hybrid device are tuned to optimize hit rates, minimize network and backend SSD contention, and match the throughput of PCIe interfaces.

System Implementation

The ITME implementation integrates with the vLLM serving framework, extending the existing tiered cache management mechanisms. Two specialized pinned-memory staging buffers in host DRAM handle asynchronous block aggregation and prefetch, with programmable granularity (64-512MB) for sequential streaming compatible with RDMA and storage characteristics. The logical dataflow ensures that blocks evicted from GPU HBM are organized to maximize retrieval locality, especially during prefill and multi-turn sessions.

Figure 5: ITME system architecture showing the end-to-end integration from GPU, through host, to the CXL-hybrid tier and its offload/prefetch scheduling.

As evictions propagate towards the remote tier, context writing is throttled and interleaved with compute phases that offer I/O slack, thereby preventing SSD write amplification and long-tail contentions.

Figure 6: Step-by-step illustration of ITME’s multi-tiered block movement and proactive prefetching aligned with inference phases.

Evaluation

The evaluation deploys ITME on a real hardware testbed: dual-socket Xeon hosts, 80GB HBM NVIDIA A100 GPUs, SK Hynix CMM, Gen5 NVMe SSDs, and 100Gbps RDMA links. Comparative baselines include GPU-caching, host memory offloading (CPU-offload), and highly-tuned local and remote NVMe-oF architectures.

Figure 7: Multi-node evaluation stack: local GPU server and remote CXL-hybrid memory server (T3.5) hardware.

Performance Characterization

• GPU Tier Comparison: Relative to GPU-only recomputation, ITME achieves up to 1.81× TTFT speedup on Llama-3.1 70B and closes nearly 60% of the gap to an ideal, all-in-GPU-memory, 80GB HBM configuration.

  Figure 8: ITME maintains a steady GPU memory footprint by tiering ~10GB to host/CXL, compared to excessive all-in-caching and recomputation approaches.

• Host Memory Tier Comparison: As cumulative KV cache grows (across 35-turn, 256-conversation ShareGPT workloads), ITME surpasses CPU-offload. While CPU-offload collapses under DRAM exhaustion, ITME maintains throughput and attains a 35.7% improvement in conversation throughput under multiterabyte context stress.

  Figure 9: ITME outperforms CPU-offload in multi-turn benchmarks once host DRAM is saturated, leveraging the scalability of remote CXL-hybrid memory.

• Comparison with Ideal NVMe-oF: Even against a local, network-free NVMe-oF upper bound, ITME demonstrates superior hit rates and retrieval latency due to hardware prefetch and effective read-priority scheduling. NVMe-oF suffers from blocking on heavy writes and lacks effective scheduling for deterministic, bursty LLM access.

  Figure 10: ITME exceeds even an ideal, locality-optimized NVMe-oF backend in amortized inference latency across escalating conversation length.

• Sensitivity to Weight Prefetch and Buffer Size: For model weights, ITME reaches within 1-5% of in-host-DRAM upper bounds, even for 105GB models exceeding the DRAM cache, provided sufficient pipeline depth is maintained.

  Figure 11: Weight prefetching analysis and sensitivity to DRAM cache capacity demonstrate near-host-memory throughput when pipeline depth matches execution pattern granularity.

• Hardware Feasibility: The FPGA-based ITME prototype achieves 18GB/s reads, 12GB/s writes to DRAM cache—20-25% below CMM reference, consistent with lower-level hardware overheads but validating the architectural design.

  Figure 12: ITME hardware prototype sustains robust bandwidth under real-world access patterns, validating feasibility of pattern-aware prefetch.

  

  Figure 13: CMM-based setup demonstrates higher read/write bandwidth than the initial FPGA implementation, with the performance gap attributed to overheads in hardware logic.

Implications and Future Directions

The ITME architecture demonstrates that CXL-hybrid memory can function as an efficient, high-bandwidth, low-latency, byte-addressable expansion tier for LLM inference. ITME challenges the reliance on software-heavy DPU/JBOF stacks and presents a path to economically scaling inference context infrastructure without GPU memory overprovisioning.

Practically, ITME allows data center operators to scale LLM endpoints elastically by centralizing context storage, efficiently amortizing hardware investment. Theoretically, ITME motivates a new class of memory hierarchy-aware inference scheduling and prefetch policies, in which hardware/software co-design maximally exploits the deterministic structure of transformer inference.

The composability of this architecture with emerging paging, context sharing, and global cache mechanisms (e.g., LMCache (Cheng et al., 8 Oct 2025), vLLM (Kwon et al., 2023), Mooncake (Tairin et al., 10 Mar 2025)) points to a convergent trend where backend memory and context layers are increasingly decoupled from server boundaries and efficiently federated across cluster-scale deployments.

Conclusion

ITME establishes a practically deployable pathway for scaling LLM inference memory through CXL-hybrid disaggregated memory tiers. By exploiting deterministic access patterns, hardware-managed prefetching, and multi-tier bandwidth alignment, ITME delivers significant improvements over CPU-offload and DPU/JBOF architectures, verifying its architectural soundness across both FPGA prototypes and production hardware. The work will inform future research into inference-tier composability, workload-aware memory tiering, and fine-grained scheduling in extreme-scale AI systems (2606.12556).