Papers
Topics
Authors
Recent
Search
2000 character limit reached

Cache-Resident LLM Inference in GB-Scale Last-Level Caches

Published 24 Jun 2026 in cs.AR | (2606.25353v1)

Abstract: LLM inference is increasingly dominated by data movement across the memory hierarchy. Recent 3D-stacked cache technologies have enabled GB-scale last-level caches in modern server CPUs, making it possible to keep reusable model weights on chip and exploit cache bandwidth and latency. Achieving this regime is not straightforward: deeper pipelining for weight residency increases in-flight requests and KV-cache footprint, while cache-resident operators make operator-boundary synchronization a visible bottleneck. We present a cache-resident execution model for inference on hierarchical-memory clustered systems. The model separates weight-centric operators from attention and KV-cache management into dedicated resource domains, keeping reusable weights cache-resident while scaling KV capacity independently of pipeline depth. It also relaxes synchronization from operator boundaries to true sub-operator dependencies, reducing coordination overhead in the cache-resident regime. We instantiate this model on a multi-socket CPU cluster with a weight-attention decoupled architecture, locality-aware placement, and a specialized static runtime. The prototype substantially outperforms equally provisioned llama.cpp. On deployed Llama-3.2-3B and Llama-2-7B configurations, it achieves 2.04x-11.51x speedup on time-per-output-token (TPOT). Under a validated analytical model, it further reaches up to 13.9x TPOT speedup across model sizes, context lengths, and batch sizes. These results show that commodity CPUs with GB-scale last-level caches can support efficient LLM inference when execution is organized around cache residency, decoupled state management, and dependency-aware coordination.

Summary

  • The paper introduces a novel cache-resident LLM inference architecture that decouples weight-centric and attention operations to overcome LLC bottlenecks.
  • The paper employs bank-aware weight placement and static thread pools to optimize cache locality and reduce synchronization overhead.
  • The paper demonstrates substantial performance gains with up to 11.51× speedup, validating the scalability of cache-resident inference on commodity CPUs.

Cache-Resident LLM Inference in GB-Scale Last-Level Caches: Technical Summary

Motivation and Architectural Challenges

LLM inference on modern server clusters is bottlenecked by memory hierarchy data movement, especially as model sizes and context lengths grow. Advances in 3D-stacked cache technologies have enabled GB-scale last-level caches (LLCs) on commodity CPUs, theoretically allowing model weights to stay resident on-chip and potentially exploit cache bandwidth and latency. The underlying challenge is achieving feasible cache residency for both static model weights and dynamically growing per-request KV (key-value) caches, which have distinct access and reuse patterns. Conventional pipeline parallelism, which shards layers across compute domains, increases in-flight requests and KV-cache footprint proportional to pipeline depth, thereby contesting LLC capacity needed for weight residency. Increasing pipeline depth for weight sharding does not reduce per-node KV-cache pressure, creating a scalability paradox.

Decoupling Weight and Attention Resource Domains

This work introduces a cache-resident execution architecture that separates weight-centric operators (QKV projection, FFN) and attention/KV-cache management into distinct resource domains. By routing outputs of weight-centric operators to dedicated attention nodes, the architecture supports scaling KV capacity independently of pipeline depth, mitigating cache contention. Each decoding sequence is mapped to an attention node that owns its KV state, while weight-centric computation nodes prioritize model weight residency, effectively decoupling memory pressure between phases.

Synchronization is relaxed from conventional operator-centric (global barriers) to true data dependencies, aligning with per-head independence in multi-head attention, and propagated via readiness signals rather than operator boundaries. Tree-based synchronization and bank-aware weight shard initialization further minimize cache-line bouncing and maximize LLC locality.

Implementation: Multi-Socket CPU Clusters and Static Thread Pools

The architecture is instantiated on AMD EPYC 9684X multi-socket CPU nodes with 1152MB LLC per socket. Pipeline parallelism is mapped across nodes, and tensor parallelism is realized across cores within a socket. Weight shards are placed bank-aware and warmed up for optimal LLC residency. Customized static thread pools avoid dynamic runtime overhead inherent in generic frameworks (e.g., OpenMP), maintaining deterministic shard-to-core mapping and reducing synchronization latency. Inter-node communication uses InfiniBand RDMA with fixed queue pairs and pre-registered buffers for microsecond-scale activation exchange.

Evaluation and Numerical Results

The prototype is benchmarked on Llama-3.2-3B and Llama-2-7B configurations against equally provisioned llama.cpp baselines on identical hardware and quantization. The cache-resident model achieves substantial improvements:

  • Measured end-to-end speedups: Up to 11.51× token-per-output-token (TPOT) speedup at batch size 1 (Llama-3.2-3B); up to 10.43× TPOT speedup (Llama-2-7B), with geomean throughput and TPOT speedups in the 3.7×–5.0× and 5.3×–6.7× range, respectively, across model sizes and workloads.
  • Thread pool optimization: Static thread pool reduces per-transformer-block latency by tens to over 100 μs in the small-batch regime (up to 1.56× speedup), with diminishing impact as batch size increases.
  • Weight-attention separation: Provides modest but systematic gains (up to 1.16× for larger models), highly dependent on cache pressure, especially in longer-context, larger-model workloads.
  • Cache-resident regime validation: Analytical models closely match empirical results, supporting scalability modeling to larger configurations (e.g., Llama-2-70B), with up to 13.9× TPOT speedup in extrapolated settings.

Practical and Theoretical Implications

The system demonstrates that commodity CPUs with GB-scale LLCs can support low-latency, high-throughput LLM inference by careful resource decoupling and cache-aware scheduling, without reliance on scarce accelerators (GPUs). The proposed separation between weight and attention resource domains directly informs system design for deployment in geopolitically constrained or supply-limited environments. The architecture also offers a template for scalable cache residency in accelerators with bounded on-chip memory, suggesting that similar locality-preserving and dependency-aware execution models can mitigate memory bottlenecks.

From a performance engineering perspective, alignment of synchronization granularity with actual semantic dependencies provides measurable reductions in coordination overhead, especially as compute kernels shorten under cache residency. The design exposes tradeoffs between latency (favorable for weight-attention separation at critical cache pressure) and throughput efficiency per resource, supporting adaptive resource allocation for latency-critical serving.

Future Directions

Further work could extend continuous batching with dynamic KV-cache reclamation, topology-aware expert routing for mixture-of-experts models, and extremely low-latency serving for interactive workloads via additional socket specialization and ultra-lightweight synchronization. The architectural principles may be adapted to large-scale cache-resident inference on accelerators, wafer-scale processors, and hybrid CPU-GPU clusters.

Conclusion

Cache-resident LLM inference exploiting GB-scale LLCs on CPUs is possible through decoupled resource domains, locality-aware placement, and dependency-driven coordination. The architecture systematically reduces memory bandwidth bottlenecks and synchronization overhead, achieving strong numerical gains against conventional baselines. The results are directly relevant to scalable deployment of LLMs in memory-dominated serving environments and provide guidance for future cache-aware inference engines across commodity and specialized hardware platforms.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Cache-Resident LLM Inference in GB-Scale Last-Level Caches — Explained Simply

What is this paper about?

This paper is about making LLMs run faster on regular server CPUs by reducing how much data they move around. The big idea: keep the most-used parts of the model inside the CPU’s fast “on-chip” memory (called cache) and organize the work so less time is wasted waiting and shuffling data.

Think of it like doing homework:

  • If your books (the model’s “weights”) are always on your desk (the fast cache), you don’t have to keep going to your locker (slow main memory).
  • Your notes that grow as you work (the “KV cache,” which stores attention history) need space too—but they shouldn’t push your books off the desk.

What questions did the researchers ask?

The paper focuses on two simple questions:

  1. How can we keep the model’s reusable data (weights) in the CPU’s fast cache without running out of space for the growing attention memory (KV cache)?
  2. How can we stop the computer from pausing too often—like teammates waiting at checkpoints—so it spends more time computing and less time waiting?

How did they do it?

They reorganized how LLMs run on multi-socket CPUs with very large last-level caches (LLC), using two key ideas.

Key ideas

  • Separate “weights” from “attention”: Weights are fixed rules used for every token. The “KV cache” is the model’s short-term memory that grows with the input length. These two have different behavior, so the authors put them on different CPU “stations”:
    • Weight nodes: handle weight-heavy steps (like projections and feed-forward layers), keeping weights hot in cache.
    • Attention nodes: handle attention steps and store the KV cache, which grows over time.
    • This avoids a tug-of-war for the same cache space and lets you add more attention capacity without messing up the weight side.
  • Sync only when truly needed: Usual systems make everything wait at the end of each operator (step), even if parts don’t depend on each other. Here, they let independent parts (like different attention heads) run ahead and only synchronize when there’s a real dependency. That’s like letting each teammate finish their subtask before doing a quick, small-group check-in, instead of a full team meeting after every tiny step.

How they built it (in everyday terms)

  • Hardware: They used servers with two-socket AMD CPUs that have a very large on-chip cache (about 1 GB per socket).
  • Placement: One socket is the “weights station,” the other is the “attention station.” Across multiple servers, they connect these stations in a pipeline (like an assembly line), passing small activation vectors between stages.
  • Inside each socket: They split work across CPU cores so each core handles a slice of the calculation, keeping small, frequently reused inputs in the fastest cache (L1) and streaming larger data (weights or KV) from the big cache (LLC).
  • Smarter syncing: Instead of full stop-and-wait at every step, they use light, tree-like, local-first synchronization, which reduces slowdowns from all cores fighting over the same lock.
  • Networking: Between servers, they use a low-latency method (RDMA) so sending activations is fast and predictable.

What did they find, and why is it important?

Here’s what changed when they organized work around cache and smarter scheduling:

  • Faster generation per token:
    • Measured speedups over llama.cpp (a strong CPU baseline) on real models:
    • Llama-3.2-3B and Llama-2-7B: 2.04× to 11.51× faster time-per-output-token (biggest gains at small batch sizes).
    • Predicted (with a validated trend model) up to 13.9× faster per-token time and up to 12.5× higher throughput across different model sizes, context lengths, and batch sizes.
  • Why this matters:
    • Less waiting, more computing: By keeping weights in cache and syncing only when necessary, CPUs spend more time doing math and less time moving data or waiting.
    • Works on widely available hardware: Many places don’t have enough GPUs. This approach shows that commodity CPUs with big caches can run LLMs efficiently if you organize the work well.
    • Scales better with long contexts: Separating attention memory from weights lets you handle longer texts without knocking weights out of cache.

What does this mean for the future?

  • Better use of CPUs: If you structure LLM inference around cache residency, you can get impressive speedups on CPUs—useful when GPUs are scarce or expensive.
  • Lower cost and latency: Faster per-token generation means snappier responses and potentially lower cloud costs.
  • A general principle: The main ideas—separating reusable data from growing state and syncing only on real dependencies—aren’t CPU-specific. They can guide designs on other hardware too (like future GPUs or custom chips with large on-chip memory).
  • More accessible AI: With smarter software organization, powerful AI can run well on more common servers, helping bring AI to more users and places.

Key terms in plain words

  • Weights: The learned numbers that define how the model thinks—fixed during inference.
  • KV cache: The model’s short-term memory of previous tokens (keys and values) used by attention; it grows with context length.
  • Cache (L1, L2, LLC): Tiny to large “desks” inside the CPU. Smaller caches are faster but hold less; LLC is slower than L1/L2 but still much faster than main memory.
  • Pipeline parallelism: Splitting layers across machines like stations on an assembly line. Deeper pipelines mean more “in-flight” tokens (more memory pressure).
  • Tensor parallelism: Splitting a single big operation across cores/machines so each handles a slice.
  • Synchronization: Making different parts of the system wait and coordinate so results line up correctly. Fewer, smarter syncs = less waiting.
View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The paper introduces a cache-resident execution model with weight–attention decoupling and sub-operator asynchronicity, but several aspects remain missing or underexplored. The following list outlines concrete gaps and questions to guide future research:

  • Prefill performance and design: The work prioritizes decoding and explicitly leaves a detailed treatment of the prefill phase to future work; it is unclear how cache residency, WA separation, and sub-operator coordination should be adapted for prefill, especially given larger GEMM-style kernels and different reuse patterns.
  • Generality beyond AMD EPYC 3D V-Cache: The prototype relies on AMD’s per-core L3 banking and first-touch placement; the portability of cache placement, bank-aware warmup, and hierarchical synchronization to Intel, ARM, or other CPU microarchitectures (with different LLC organizations) is not evaluated.
  • Applicability to accelerators and other memory hierarchies: Although the principles are stated to be hardware-agnostic, there is no implementation or evaluation on GPUs, NPUs, or CPUs with HBM/L4; it remains open how to map WA separation and sub-operator asynchronicity to those hierarchies and their distinct synchronization and cache models.
  • Scaling beyond four pipeline nodes: End-to-end results cover up to four nodes with RDMA; behavior under deeper pipelines (more nodes), higher hop counts, and interconnect contention is not characterized, nor are techniques for mitigating growing communication latency or variability.
  • Operation without RDMA: The system relies on InfiniBand RDMA to keep per-hop overhead in the tens of microseconds; performance and design adaptations for commodity Ethernet-only deployments (common in clouds) are not explored.
  • Load balancing and routing to attention nodes: The scheme assigns each sequence to a specific attention node, but the paper does not describe or evaluate policies for dynamic routing, hotspot avoidance, or rebalancing when context lengths vary widely across sequences.
  • KV-cache overflow management: When KV working sets exceed LLC, degradation is said to be confined to attention nodes, but there is no quantification of thresholds, degradation curves, or exploration of mitigation strategies (e.g., KV compression/quantization, windowing, eviction, paging to DRAM, or selective recomputation).
  • Impact of modern attention variants (MQA/GQA): Sub-operator independence is motivated by head independence, yet many current models use grouped/multi-query attention with shared K/V; the implications for sub-operator asynchronicity, synchronization structure, and KV layout are not analyzed or evaluated.
  • Extensibility to other model architectures: The approach is validated on standard decoder-only Transformers (LLaMA/Qwen). Its applicability to MoE models, encoder–decoder models, recurrence (e.g., RWKV), or linear/long-context attention variants remains unexplored.
  • Quantization accuracy and robustness: All evaluations use INT8 (including KV), citing prior work for accuracy, but there is no in-paper quality evaluation (e.g., perplexity, task metrics) across models and contexts, nor an analysis of sensitivity to quantization schemes, outlier channels, or KV quantization errors.
  • Compute vs memory-bound transition: The design assumes decoding is memory-bound; the behavior near the crossover (e.g., at larger batch sizes or different precisions where kernels become compute-bound) and whether synchronization overheads re-emerge are not examined.
  • Energy and cost efficiency: The paper reports latency/throughput but lacks measurements of power, tokens-per-joule, and TCO comparisons against alternative CPU/GPU deployments; it is unclear whether cache-resident CPU inference is economically competitive at scale.
  • Robustness under OS noise and virtualization: The runtime assumes strict per-core pinning and low jitter; effects of OS interference, SMT configurations, hypervisor scheduling, and noisy neighbors (multi-tenant settings) on cache residency and synchronization are not evaluated.
  • Coherence and synchronization overhead quantification: While hierarchical synchronization is motivated, there is no direct measurement of coherence traffic, cache-line bouncing, or lock contention, nor a model predicting sync costs as core counts and head counts scale.
  • Autotuning and adaptivity: The system uses static partitioning and thread placement; no autotuner is provided to choose pipeline depth, WA separation, batch size, or head-level sync topology based on SLOs, model size, and context length at runtime.
  • Network-aware scheduling: The analytical model uses fixed 5 μs per hop and ignores congestion; dynamic scheduling that co-optimizes compute placement and network paths/topology (e.g., NUMA and link-aware mapping) is not developed.
  • Baseline completeness and fairness: Comparisons are limited to llama.cpp; the paper does not benchmark against other optimized CPU inference engines (e.g., vLLM CPU backends, FasterTransformer-like CPU ports) or alternative operator fusion/scheduling strategies.
  • Validation at larger scales: Results are measured on 3B/7B and extrapolated for 70B; without hardware measurements at >7B, it remains uncertain how cache residency, WA separation, and inter-node effects behave at 10–80B scales and larger contexts.
  • Prefetching and cache policy control: The design counts on “streaming from LLC” but lacks an investigation of hardware prefetchers, software prefetching, line sizes, and replacement policies; methods to further reduce LLC miss penalties and bank conflicts remain unexplored.
  • Scheduling for heterogeneous and bursty workloads: The steady-state decoding analysis does not address arrival variability, SLO-driven batching/deferral, or policies that trade latency and throughput; mechanisms to avoid pipeline bubbles under bursty traffic are not presented.
  • Failure handling and elasticity: The pipeline model assumes all stages remain healthy; recovery from node failure, dynamic addition/removal of nodes, and state handoff for KV ownership in attention nodes are not considered.
  • Integration with ML frameworks: The bespoke runtime and sub-operator dependencies are not integrated into mainstream frameworks (PyTorch, XLA, TVM); a general API or IR to express sub-operator readiness and WA separation for graph compilers is not provided.
  • Security and isolation with RDMA and shared memory: Using pre-registered buffers and shared memory across sockets/nodes raises questions about memory safety, isolation, and multi-tenant security; mitigations and policies are not discussed.
  • Effects of long-context features: While context lengths up to 4096 are evaluated, practical deployments often require 32k–200k tokens; the performance and memory impacts of positional encodings (e.g., RoPE scaling), chunked attention, and long-context optimizations in the cache-resident regime are not quantified.
  • Detailed kernel-level choices: The “Flash-style” attention kernel and GEMV design are outlined but not compared against alternative kernels (e.g., low-precision KV layouts, block-sparse/factorized kernels) within the LLC-resident constraints; optimal tile sizes and dataflows for different head counts and dimensions are left open.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

The following use cases can be deployed today using GB-scale LLC multi-socket CPUs and the paper’s execution model (weight–attention decoupling, cache residency, and sub-operator scheduling), with integration effort primarily on the software/runtime side.

  • CPU-only LLM serving with near-GPU-like responsiveness at small batches
    • Sector: cloud/data center, software
    • What: Serve 3–8B parameter models (e.g., Llama-3.2-3B, Llama-2-7B, Qwen-3-8B) on dual/multi-socket 3D V-Cache CPUs to cut time-per-output-token (TPOT) by 2–11× vs conventional CPU stacks, especially for B=1–4.
    • Tools/products/workflows: A CPU runtime implementing WA decoupling, hierarchical sync, and cache-aware placement; drop-in CPU backend/plugin for vLLM/TGI/LLM-as-a-service gateways; “cache warm” service lifecycle that pins threads and pre-touches weight shards.
    • Assumptions/dependencies: Servers with GB-scale LLC (e.g., AMD EPYC 3D V-Cache); INT8 quantization (including KV-cache) with acceptable accuracy; decoding-heavy workloads; moderate context lengths (or attention-node capacity to handle long contexts).
  • On-prem, regulated deployments when GPUs are scarce or restricted
    • Sector: healthcare, finance, government
    • What: Compliant, low-latency inference for chatbots, document summarization, coding assistants using CPU-only clusters in controlled facilities.
    • Tools/products/workflows: Kubernetes Operator that maps “weight sockets” and “attention sockets” into distinct node pools; policy templates for pinning and NUMA/CCD affinity; observability dashboards for cache residency.
    • Assumptions/dependencies: Dual-socket (or more) servers; OS support for CPU pinning and NUMA/first-touch; acceptable INT8 accuracy on target models; potentially RDMA-capable interconnect inside the cluster.
  • SLA-oriented small-batch serving for interactive assistants
    • Sector: enterprise SaaS, customer support, IDE co-pilots
    • What: Prioritize low latency at B=1–4 using cache-resident weights; reduce tail latency without batching penalties.
    • Tools/products/workflows: Serving policies that route latency-critical requests to “weight-resident” CPU tiers; admission control that caps batch size to protect TPOT.
    • Assumptions/dependencies: Decoding dominates; prefill path may still need optimization; workload skew toward interactive latency rather than raw throughput.
  • Long-context request handling via attention-node scaling
    • Sector: legal, eDiscovery, RAG applications, analytics
    • What: Scale KV capacity by attaching dedicated attention nodes, sustaining longer contexts without evicting reusable weights.
    • Tools/products/workflows: “KV islands” attention servers; sharded KV-cache placement keyed by sequence/session; workflow to pin sessions to their attention home.
    • Assumptions/dependencies: Sufficient DRAM and LLC at attention nodes; careful KV tiling (Flash-style kernels); network latency budget for weight→attention routing.
  • RDMA-based inter-stage activation transport to remove networking overhead
    • Sector: cloud networking, telecom/NFV
    • What: Reduce per-hop send overhead to tens of microseconds between pipeline stages; keep communication off the kernel stack.
    • Tools/products/workflows: Pre-registered buffer pools and persistent QPs; gRPC bypass plugin for intra-cluster hop-to-hop embedding transfer.
    • Assumptions/dependencies: NICs supporting RDMA (e.g., ConnectX-6/7), RDMA-enabled fabric configuration; static pipeline topology.
  • Static thread-pool and hierarchical synchronization library for microsecond kernels
    • Sector: software infrastructure, HPC
    • What: A reusable C/C++ runtime that outperforms general-purpose OpenMP-like models for short kernels by providing per-core pinning, CCD-local barriers, and sub-operator signaling.
    • Tools/products/workflows: Drop-in primitives for GEMV-heavy codes and other cache-resident kernels; bindings for PyTorch custom ops/TVM externs.
    • Assumptions/dependencies: Developers adopt static scheduling; per-socket thread pools with fixed placement; platform exposes CCD/LLC bank topology.
  • Cache-aware weight placement and warmup utilities
    • Sector: DevOps/SRE tooling
    • What: Tools to deterministically map shards to LLC banks and verify cache residency; automated “first-touch” warmup on service start.
    • Tools/products/workflows: CLI/agent that pins threads, prefetches weights, validates LLC hits; eBPF or perf-based monitors for L1/L3 hit ratios and coherence traffic.
    • Assumptions/dependencies: OS affinity APIs; low-level performance counters; stable shard-to-core mapping.
  • Cost and supply-chain mitigation when GPUs are constrained
    • Sector: policy, procurement, cloud capacity planning
    • What: Replace or hedge GPU capacity with cache-rich CPU fleets for common inference tiers, reducing capex/lead-time risk.
    • Tools/products/workflows: TCO calculators comparing CPU cache-resident serving vs GPU at target SLAs/batches; procurement guides emphasizing V-Cache CPUs.
    • Assumptions/dependencies: Model sizes compatible with CPU throughput targets; power/cooling budgets; INT8 acceptance by stakeholders.
  • Academic benchmarking and pedagogy on cache-resident ML systems
    • Sector: academia
    • What: Use the paper’s analytical model and prototype to teach memory-bound inference, pipeline scheduling, and synchronization granularity.
    • Tools/products/workflows: Open benchmark harnesses for per-layer latency vs cache pressure; labs reproducing TPOT speedups and thread-pool ablations.
    • Assumptions/dependencies: Access to multi-socket 3D V-Cache servers or simulators; open-source release of kernels/runtime.

Long-Term Applications

These use cases are enabled by the paper’s principles but require further research, hardware support, or ecosystem changes.

  • Hardware co-design: native support for weight–attention separation
    • Sector: semiconductors
    • What: Sockets with dedicated, software-addressable KV SRAM/LLC partitions, hardware attention engines, or KV-aware coherence policies; on-die fabric QoS between “weight” and “attention” domains.
    • Tools/products/workflows: ISA hints for head-level dependencies; prefetchers tuned for GEMV streaming vs KV tiling.
    • Assumptions/dependencies: Vendor roadmap alignment; standardized APIs across OS/runtime stacks.
  • Compiler and IR support for sub-operator DAGs
    • Sector: ML compilers (TVM, XLA, MLIR), frameworks (PyTorch)
    • What: First-class representation of attention-head independence and bounded-fan-in reductions; schedulers that emit per-head readiness signals instead of operator barriers.
    • Tools/products/workflows: MLIR dialects or TVM schedules for “head-DAGs”; correctness checks for numerics and determinism.
    • Assumptions/dependencies: Upstream acceptance; performance-portable implementations across CPU/GPU.
  • Generalization to GPU/TPU/NPU backends
    • Sector: AI accelerators
    • What: Map WA decoupling to HBM partitions/MIG slices with “KV islands” and head-level kernel graphs; reduce all-reduce barriers inside attention/FFN.
    • Tools/products/workflows: CUTLASS/cuDNN schedule templates for sub-operator sync; GPUDirect RDMA pipelines for inter-stage embeddings.
    • Assumptions/dependencies: Vendor runtime cooperation (CUDA, ROCm, XLA); mechanisms for memory partitioning and stream-level signaling.
  • Elastic KV memory tiers for ultra-long contexts
    • Sector: memory systems, storage
    • What: Hierarchies that spill KV from LLC→DRAM→CXL/NVMe while keeping weights hot; predictive pinning based on sequence access patterns.
    • Tools/products/workflows: KV-object stores with tiling/compression; prefetch/evict policies integrated with attention kernels.
    • Assumptions/dependencies: CXL memory pooling maturity; acceptable latency penalties for far-memory KV blocks.
  • Multi-tenant isolation via attention-node tenancy
    • Sector: security, compliance
    • What: Hard isolation of per-tenant KV state on attention nodes, simplifying data governance and incident response.
    • Tools/products/workflows: Attestation of KV residency and wipe-on-release; audit trails linking sessions to KV shards.
    • Assumptions/dependencies: Strong OS/container isolation; side-channel defenses for shared LLC; formal policies.
  • Energy-aware, cache-residency-first schedulers
    • Sector: energy/green computing
    • What: Schedulers that minimize DRAM traffic and coherence churn to reduce joules/token; carbon-aware placement of weight and attention work.
    • Tools/products/workflows: Telemetry-driven DVFS tied to cache hit-rate; energy models embedded in admission control.
    • Assumptions/dependencies: Fine-grained power counters; SLAs tolerating energy-optimal batching windows.
  • Real-time and edge robotics with compact LLMs
    • Sector: robotics, industrial control
    • What: Use sub-operator sync and cache-resident kernels to bound inference latency of small LLMs for language-mediated control or diagnostics.
    • Tools/products/workflows: Real-time thread pool variants; safety-certified inference components.
    • Assumptions/dependencies: Hard real-time OS support; verified worst-case execution times; tightly bounded contexts.
  • Ultra–low-latency financial inference in colocation facilities
    • Sector: finance
    • What: Deploy microsecond-scale CPU pipelines with RDMA to classify or summarize streams with minimal added latency.
    • Tools/products/workflows: Co-lo appliances integrating WA runtime; integration with market data buses.
    • Assumptions/dependencies: Compliance and audit requirements; predictable network jitter; carefully tuned NUMA.
  • Knowledge-work platforms with very long windows on CPUs
    • Sector: legal tech, software engineering, biomedical research
    • What: 128k–1M token workflows for code review, contract analysis, or literature synthesis on CPU clusters using KV sharding.
    • Tools/products/workflows: Session pinning to KV islands; retrieval pipelines coupling chunking with KV tile locality.
    • Assumptions/dependencies: Models that maintain quality at long contexts under INT8; sufficient DRAM/CXL pools.
  • OEM “cache-resident inference appliance”
    • Sector: hardware systems
    • What: Turnkey servers exposing WA-separated sockets, RDMA fabric, and cache-aware orchestration as a managed product.
    • Tools/products/workflows: Management plane for warmup/pinning; SLA profiles (latency-first vs throughput-first).
    • Assumptions/dependencies: Market demand; standardized benchmarks demonstrating TPOT gains (e.g., MLPerf extensions).
  • Standards and policy for cache-resident AI benchmarking and procurement
    • Sector: standards bodies, public sector IT
    • What: Define metrics (e.g., TPOT at small batches, DRAM bytes/token) and procurement profiles emphasizing cache residency and GPU alternatives.
    • Tools/products/workflows: Benchmark suites; guidance for export-control–constrained environments.
    • Assumptions/dependencies: Cross-vendor participation; reproducible methodologies.

Notes on feasibility across applications:

  • Performance advantages are strongest when decoding dominates and batch sizes are small-to-moderate; at very large batches, memory reuse narrows the gap.
  • The current prototype prioritizes decoding; prefill optimizations may be required for workloads with heavy prefill.
  • INT8 quantization (including KV) is assumed; some domains may require higher precision.
  • Benefits depend on accurate thread pinning, cache-bank–aware placement, and stable shard-to-core mappings; orchestration must preserve these properties.
  • RDMA is beneficial for inter-node hops but not strictly required intra-node; Ethernet adds software overhead that can erode gains.
View Paper Prompt  View All Prompts 

Glossary

  • 3D-stacked cache: A packaging technology that stacks memory layers to greatly increase on-chip cache capacity and bandwidth. "Recent advances in 3D-stacked cache technologies have enabled GB-scale last-level caches in modern server CPUs."
  • Analytical model: A simplified performance model used to estimate latency/throughput trends from measured components. "We build an analytical model to characterize latency and throughput under PP scaling."
  • Arithmetic intensity: The ratio of compute operations to data movement; higher values imply better reuse of data per byte moved. "the effective arithmetic intensity improves only modestly and remains relatively low, as shown in Fig. 2."
  • Bank-aware placement: Organizing data so cores primarily access nearby cache banks to reduce latency and contention. "weight shards are partitioned and initialized in a bank-aware manner so that each core predominantly accesses LLC slices that are physically close to it."
  • Cache-line bouncing: Repeated transfer of cache-line ownership among cores due to synchronization or shared updates, causing coherence overhead. "Such atomic operations introduce non-trivial contention and cache-line bouncing across cores, especially in this highly concurrent setting."
  • Cache residency: Keeping the active working set in on-chip caches so computation is limited by cache bandwidth/latency instead of main memory. "After cache residency is achieved, synchronization overhead is no longer negligible."
  • CCD (Core Complex Die): A multi-core chiplet unit in AMD CPUs with local caches used to structure locality-aware synchronization. "Each CCD maintains a local atomic counter shared only by cores within that CCD."
  • Context length: The number of tokens of past state the model attends over; it drives KV-cache size and attention cost. "As model sizes and context lengths grow, the re- peated decoding incurs substantial data movement across the memory hierarchy"
  • Fan-in: The number of parallel participants converging on a single synchronization point or reduction, affecting coordination cost. "Such synchronization cost scales with fan-in: the more tensor- parallel shards converge on the same completion state, the higher the coordination overhead"
  • Feed-forward network (FFN): The per-token MLP sublayer in a transformer block applied after attention. "Fig. 1(b)-(d) zooms into a single transformer block, which typically consists of three phases: QKV projection, attention, and feed-forward network (FFN)."
  • First-touch NUMA placement: A policy where memory is allocated on the NUMA node of the core that first accesses it, improving locality. "this first touch follows the same local- ity principle as first-touch NUMA placement [9]"
  • Flash-style kernel: An attention implementation that tiles KV blocks to avoid materializing large intermediates (inspired by FlashAttention). "Attention computation is executed on the attention socket using a Flash-style kernel [6]."
  • GEMV (matrix-vector multiplication): A linear algebra kernel multiplying a matrix by a vector; dominant in LLM decoding. "LLM decoding is dominated by matrix-vector multiplica- tion (GEMV) with low data reuse in both linear projection and attention."
  • Hierarchical-memory clustered system: A distributed setup with multiple nodes, each having a cache/memory hierarchy, organized for scalable inference. "a cache-resident execution model for infer- ence on hierarchical-memory clustered systems."
  • Hierarchical synchronization: A multi-level coordination scheme that synchronizes locally (e.g., within a CCD) before global aggregation to reduce contention. "we employ a hierarchical synchronization structure built on lightweight atomic integers."
  • InfiniBand RDMA: A low-latency, kernel-bypass network transport used to move activations between pipeline stages. "the prototype adopts Infini- Band RDMA to transfer activation tensors between consecu- tive pipeline stages."
  • INT8: An 8-bit integer quantization format used to store weights/activations to reduce memory and increase throughput. "All models are evaluated in a fully INT8 configuration, in- cluding the KV cache."
  • KV cache: The per-sequence key/value tensors stored for attention to enable autoregressive decoding across context. "Attention state (KV cache) must be maintained for every in-flight token at each stage."
  • Last-level cache (LLC): The largest shared on-chip cache level that backs private caches before main memory. "The hierarchy consists of private caches and access to a shared last- level cache (LLC), backed by main memory."
  • Multi-head attention: An attention mechanism with multiple independent heads operating on disjoint projections. "In multi-head attention, each head operates on its own shard of projections and KV state."
  • Operator-centric execution: A framework model where operators are the units of scheduling and synchronization barriers. "The operator-centric abstraction has been widely adopted in machine learning frameworks"
  • Pipeline parallelism (PP): Partitioning model layers across devices as pipeline stages to fit weights and increase throughput. "Layer-wise partition- ing, a.k.a. pipeline parallelism (PP), shards the model by lay- ers"
  • RDMA queue pair: A connection context for RDMA verbs pre-established to avoid per-transfer setup overhead. "RDMA queue pairs are established during initialization and reused throughout ex- ecution."
  • Resource domain: A group of compute units that share part of the memory hierarchy and have high intra-domain bandwidth. "A group of computing units sharing portions of the mem- ory hierarchy forms a resource domain."
  • SmoothQuant: A quantization method enabling accurate INT8 LLM inference by smoothing activation ranges. "Prior work such as SmoothQuant [32] shows that INT8 LLM inference can preserve accuracy suf- ficiently for practical deployment."
  • Sub-operator asynchronicity: Scheduling and synchronization at finer-than-operator granularity (e.g., per head), avoiding global barriers. "To implement sub-operator asynchronicity, we employ a hierarchical synchronization structure built on lightweight atomic integers."
  • Tensor parallelism (TP): Splitting an operator’s computation across devices/cores and synchronizing at operator boundaries. "intra-layer partitioning, a.k.a. tensor paral- lelism (TP), splits individual operators within a layer across different computing domains"
  • Thread pool (static per-socket): A fixed set of pinned threads per socket used to minimize scheduling overhead and enforce locality. "We therefore im- plement a static thread pool runtime instead of relying on existing frameworks such as OpenMP [5]."
  • Time-per-output-token (TPOT): The latency metric measuring time required to generate one token during decoding. "it achieves 2.04x-11.51x speedup on time-per- output-token (TPOT)."
  • Tree-based synchronization: A reduction scheme that merges partial results via a bounded fan-in tree to limit contention. "we perform a tree-based synchronization that bounds fan-in at each merge"
  • VNNI: Intel/AMD x86 vector neural network instructions that accelerate low-precision dot products. "This setting enables x86 VNNI acceleration for dot-product-heavy kernels."
  • WA separation (Weight-Attention separation): Decoupling weight-centric operators from attention to isolate KV-cache from weight residency and reduce contention. "WA sepa- ration is optional: when KV-cache pressure is still modest, a colocated design remains more socket-efficient;"
  • Weight-centric operator: Operators (e.g., QKV projection, FFN) that depend only on static model weights and benefit from residency. "Weight-centric operators are grouped into weight nodes that prioritize reusable weight residency, and no KV cache is maintained there."
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 36 likes about this paper.

Sign Up for Free