---
title: 'PagedAttention: Efficient Memory Management for LLMs'
url: https://www.emergentmind.com/topics/pagedattention
type: topic
---

# PagedAttention: Efficient Memory Management for LLMs

PagedAttention is a memory management and caching framework for efficient key-value (KV) cache allocation in transformer-based large language model (LLM) inference and training. Inspired by operating system virtual memory paging, PagedAttention fragments each sequence’s KV cache into fixed-size blocks that are dynamically allocated and referenced through per-sequence block tables. This approach dramatically reduces the internal and external fragmentation caused by monolithic allocation schemes, thereby enabling higher throughput, greater effective batch sizes, and support for long-context decoding on resource-constrained hardware.

## 1. Principles of PagedAttention

PagedAttention maps the logical KV cache for each sequence into fixed-size, physically noncontiguous blocks (or "pages"). Each block contains KV pairs for a fixed number of tokens, typically 16–128, and may reside anywhere in device memory. A per-sequence table maintains the mapping from logical (token position) to physical (memory address) blocks, analogous to page tables in operating systems [2309.06180]. 

Key steps:
- When a new token fills up the active block, a new physical block is allocated and registered in the page table.
- Attention kernels gather KV states for a sequence by traversing page tables and fetching the appropriate blocks.
- When a sequence completes or requires memory reclamation, all blocks registered in its table can be promptly released.

This structure enables on-demand allocation and fine-grained reclamation, resulting in minimal idle memory. Additionally, prefix blocks can be shared between sequences (for e.g., beam search, parallel sampling), leveraging copy-on-write semantics for block modifications [2309.06180, 2511.17593].

## 2. Mathematical and Algorithmic Formulation

Let $B$ be the number of concurrent sequences, $L$ the maximum sequence length, $d$ the hidden dimension, $H$ the number of attention heads, and $b$ the block size (tokens per block). Instead of statically reserving $B \times L \times H \times d$ memory per batch, PagedAttention only allocates memory for active (populated) blocks:

\[
M_{\mathrm{paged}} = 2 \cdot (\text{\#active blocks}) \cdot b \cdot d \cdot H \cdot \text{bytes/element}
\]

In steady-state generation, internal fragmentation per sequence is limited to $b-1$ tokens. Memory efficiency is quantified as the ratio of utilized vs. reserved cache:

\[
F_{\mathrm{paged}} = \frac{B \cdot (b-1) \cdot S_{\mathrm{token}}}{U + B \cdot (b-1) \cdot S_{\mathrm{token}}}
\]
where $S_{\mathrm{token}}$ is the per-token storage cost, and $U$ is bytes in use [2405.04437].

The allocation and retrieval algorithms operate as follows:

```python
# Allocate a new block on page boundary
if ctx_len[seq] % b == 0:
    block_id = alloc_block()
    page_table[seq].append(block_id)
# Write KV to current block
offset = ctx_len[seq] % b
K_blocks[block_id][offset] = current_K
V_blocks[block_id][offset] = current_V
```

PagedAttention supports rapid block (page) allocation and deallocation, and batch attention kernels gather all relevant blocks using the page table indirection.

## 3. Integration with Serving Systems and Attention Kernels

PagedAttention is centrally featured in the vLLM LLM serving system [2309.06180], wherein the block engine manages a global pool of free device memory blocks and per-sequence block tables. The system supports:
- KV block sharing across multiple sequences and requests via reference counting.
- Copy-on-write semantics for block mutation in branched decoding (beam search, few-shot, RAG).
- Custom CUDA attention kernels that accept a list of block pointers and treat the concatenated blocks as input to fused matrix multiplication routines.

PyTorch’s FlexAttention model offers a programmable interface for implementing “paged” attention via `mask_mod` and `score_mod` hooks that accept arbitrary page-table–based masking and scoring. The FlexAttention kernel fuses these hooks into Triton or TorchInductor-generated GPU code, optimizing block-wise data movement and reducing the cost of indirection [2412.05496, 2506.07311]. Empirical results show that FlexAttention with paging adds less than 2% kernel overhead and achieves up to 30% reduction in peak GPU memory at 16K–64K contexts compared to standard full-KV caching [2412.05496].

## 4. Quantitative Performance and Memory Efficiency

PagedAttention realigns memory usage to closely match actual token demand with minimal reserve overhead. Evaluations demonstrate the following:

| Model          | Baseline Tput | PagedAttn Tput | TGI Tput | Peak Mem (GB) | Latency (s p99) |
|----------------|---------------|----------------|----------|---------------|-----------------|
| LLaMA-2-7B     | 9,170         | 15,243         | 4,156    | 24.3          | 14.2            |

PagedAttention achieves up to 24× throughput improvement over HuggingFace TGI for high-concurrency workloads, reduces peak GPU KV memory by 20%, and consistently scales with batch size and sequence length [2511.17593]. Benchmarks on FlexAttention kernels indicate decoding latency per token remains nearly flat as sequence length scales up to 64K (see Table 1 in [2412.05496]).

Memory gains are more pronounced for heterogeneous-length batches, long-context tasks, and when sharing prefixes. For batch average length $S_{\mathrm{avg}} \ll S_\mathrm{max}$, the theoretical reduction ratio is $1 - (S_{\mathrm{avg}}/S_\mathrm{max})$ (e.g., up to 88% if $S_{\mathrm{avg}}=0.125 \, S_\mathrm{max}$) [2511.17593].

## 5. Extensions: Eviction, Compression, and Quantization

### Block-Wise Eviction
PagedEviction augments PagedAttention with structured block-wise KV cache pruning: entire pages are evicted based on token or block "importance" scores (e.g., $S_j = (1/B) \sum_{i \in \mathrm{block\,}j} ||V_i||_2/||K_i||_2$). This enables cache budgets with negligible accuracy loss under tight GPU constraints and without kernel changes [2509.04377].

### Variable Compression
KV-Compress introduces variable compression rates per attention head, evicting contiguous KV blocks determined by per-head "least important" metrics. The memory footprint after eviction is $M_\mathrm{after} = B \times d \times \sum_{h=0}^{H-1}(L-e_h)$; further block rounding aligns to full pages. In vLLM integration, up to $5.18\times$ throughput increase and up to $64\times$ compression with >90% accuracy retention is reported [2410.00161]. The entire compression scheduling executes as custom on-GPU kernels.

### Low-Precision KV Caching
PagedAttention memory layout is amenable to aggressive quantization. In practical systems, each KV page (block) can be quantized to 4 bits with group-wise min-max or Hessian-aware scaling [2505.24584]. Streaming attention kernels decode and reconstitute these quantized pages on-the-fly with negligible (<0.1%) accuracy loss while quadrupling effective cache capacity.

## 6. Trade-Offs, Alternatives, and Limitations

PagedAttention requires noncontiguous tensor layouts, necessitating custom CUDA kernels or programmable GPU attention kernels with block pointer support. This design adds complexity for kernel development, maintenance, and integration with new kernel optimizations (e.g., grouped-query, FlashInfer, etc.) [2405.04437]. For each attention algorithm or optimization, new paged-compatible kernels may be required, increasing engineering overhead.

Alternatives such as vAttention employ CUDA virtual memory APIs to decouple virtual from physical memory, retaining contiguous layouts for the KV cache and enabling unmodified attention kernel reuse [2405.04437]. This approach achieves similar near-zero fragmentation, often with lower runtime and programming complexity at a mild cost in practical flexibility (e.g., porting to non-NVIDIA backends).

PagedAttention introduces minor kernel overheads (10–30% in some benchmarks [2405.04437]) due to additional indirection, especially as block size increases. CPU management of block tables can impact tail latencies. Block size tuning ($b \in [16,128]$) is critical to balancing internal waste, GPU utilization, and eviction/reactivity overheads [2309.06180, 2509.04377].

## 7. Applications and Deployment Considerations

PagedAttention is deployed broadly in high-throughput, memory-bound LLM serving scenarios, including:
- vLLM (primary reference implementation), Foundation Model Stack (FMS), and other open-source LLM inference systems [2309.06180, 2511.17593, 2506.07311].
- Multi-user API endpoints and batch generators, maximizing served request rates under finite device memory.
- RAG, beam search, and parallel sampling tasks where prefix sharing yields additional memory compaction.
- Industrial agentic pipelines integrating quantized paged KV caches with speculative decoding and multi-pass inference for constrained environments [2505.24584].

PagedAttention is most beneficial as context lengths continue to increase and models adopt more advanced attention variants or high-concurrency serving patterns. Ongoing development includes further extensions for multi-tier cache hierarchies, background page eviction, efficient quantization, and integration with dynamic attention pruning frameworks [2509.04377, 2410.00161]. PagedAttention remains a foundational mechanism for scalable LLM inference in contemporary production and research settings.

Source: https://www.emergentmind.com/topics/pagedattention