---
title: 'PagedAdamW: Memory-Efficient AdamW Variant'
url: https://www.emergentmind.com/topics/pagedadamw
type: topic
---

# PagedAdamW: Memory-Efficient AdamW Variant

Searching arXiv for the specified paper to ground the article and citation metadata.
PagedAdamW is an 8-bit, memory-efficient variant of AdamW, used in the case study "Profiling LoRA/QLoRA Fine-Tuning Efficiency on Consumer GPUs: An RTX 4060 Case Study" to profile parameter-efficient fine-tuning on an NVIDIA RTX 4060 with 8 GB VRAM [2509.12229]. In that study, the term refers specifically to the bitsandbytes implementation described as “PagedAdamW (8-bit, memory-efficient),” deployed during LoRA/QLoRA fine-tuning of Qwen/Qwen2.5-1.5B-Instruct. Its defining characteristic is not a different optimization rule, but a different optimizer-state storage and residency strategy: optimizer states are kept in 8-bit form and managed through paging, reducing GPU memory pressure so that longer sequence lengths and larger batches can fit within a constrained VRAM budget [2509.12229].

## 1. Definition and optimizer role

In the reported setup, PagedAdamW is an implementation choice within a broader LoRA/QLoRA training stack rather than a distinct optimizer family with altered update equations [2509.12229]. The model is Qwen/Qwen2.5-1.5B-Instruct, the fine-tuning method is parameter-efficient fine-tuning via PEFT for LoRA integration, and the software stack includes Transformers v4.55.0 via the Trainer API, PEFT, bitsandbytes for quantized optimizer support, and Python 3.12. The hardware platform is a single NVIDIA RTX 4060 with 8 GB GDDR6 and a board power cap of 115 W, hosted on an AMD Ryzen 9 7900X system with 96 GB RAM running Ubuntu 24.04, NVIDIA driver 575.64.03, CUDA 12.9 on the system, and a PyTorch 2.2.0 CUDA 12.1 backend [2509.12229].

The study uses PagedAdamW in two of three profiled runs. The baseline run instead uses AdamW (torch). The reported distinction is primarily one of memory/storage strategy. Standard AdamW in PyTorch maintains 32-bit or 16-bit optimizer state, including momenta and variances, on the GPU. PagedAdamW, by contrast, stores optimizer states in 8-bit form and pages them so they do not need to fully reside on the GPU at all times [2509.12229]. Within the paper’s framing, this makes PagedAdamW especially relevant for consumer GPUs where optimizer-state memory competes directly with parameters and activations for a fixed 8 GB VRAM envelope.

This role is operational rather than purely theoretical. The study attributes the feasibility of sequence length 2048 with batch size 2 on the RTX 4060 to parameter-efficient methods combined with the memory-efficient optimizer design. It further reports that the paged configuration improved throughput relative to the baseline AdamW run while staying within the 8 GB budget [2509.12229].

## 2. Mathematical relation to AdamW

The study treats PagedAdamW as an efficiency variant of AdamW rather than a new optimizer with modified dynamics [2509.12229]. Accordingly, the AdamW update equations apply:

$$
m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t
$$

$$
v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^{\odot 2}
$$

$$
\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}
$$

$$
w_{t+1} = w_t - \eta \left( \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} + \lambda w_t \right)
$$

Within this formulation, PagedAdamW changes how optimizer states are stored and moved, not the mathematical update itself [2509.12229]. The paper therefore presents PagedAdamW as preserving decoupled weight decay and the standard AdamW state evolution while altering the implementation substrate through 8-bit state representation and paging.

A common misconception is to interpret “PagedAdamW” as implying a different optimization law. The case study does not support that interpretation. It explicitly characterizes the method as an implementation that is “8-bit, memory-efficient,” and states that the mathematical update is the same as AdamW. The practical difference lies in storage precision and state residency management, not in alternative moment estimation or weight-decay mechanics [2509.12229].

The paper does not report numerical values for betas, epsilon, weight decay, or learning rate, and it does not provide fused kernel settings or further paged-configuration specifics beyond the description “8-bit, memory-efficient” [2509.12229]. This omission limits any attempt to decompose performance differences into optimizer hyperparameters versus systems-level effects.

## 3. Interaction with LoRA and QLoRA under VRAM constraints

The study situates PagedAdamW inside a LoRA/QLoRA workflow designed for memory-constrained fine-tuning [2509.12229]. LoRA trains small low-rank adapters. QLoRA, as described conceptually in the paper, quantizes base model weights, commonly to 4-bit, while keeping adapters in 16-bit. The backend notes “BitsAndBytes (bnb) with 16-bit adapter layers,” consistent with QLoRA practice. The paper does not enumerate quantization bits per individual run, and all runs are reported under the umbrella of “LoRA/QLoRA fine-tuning” [2509.12229].

In this context, PagedAdamW addresses optimizer-state pressure rather than adapter design. On an 8 GB GPU, activations, parameters, gradients, and optimizer states all contend for limited memory. The reported benefit of PagedAdamW is that optimizer states are stored in 8-bit and paged so they do not fully reside on GPU memory, reducing VRAM pressure enough to make more aggressive configurations feasible [2509.12229]. The study specifically states that this helped the stress-test configuration fit batch size 2 at sequence length 2048 tokens with a peak VRAM usage of 8062 MB and without out-of-memory failure.

This should not be read as implying that PagedAdamW always lowers absolute peak VRAM below all non-paged runs. In the reported results, the PagedAdamW runs have higher peak VRAM than the baseline: 8062 MB for the 2048-token fp16 stress test and 7949 MB for the 1024-token bf16 intermediate run, compared with 6234 MB for the baseline AdamW run [2509.12229]. The reason given is that the paged runs also use larger batches and longer sequence lengths. The relevant claim is therefore one of feasibility at higher workload intensity, not simple peak-memory minimization.

The paper does not provide per-parameter memory accounting or closed-form memory formulas for weights, gradients, or optimizer states [2509.12229]. A plausible implication is that its focus is empirical profiling rather than analytical memory modeling.

## 4. Experimental context and profiled configurations

The case study is a controlled profiling experiment rather than a general benchmark suite [2509.12229]. Training uses a 5k-sample Alpaca subset consisting of instruction-response pairs, tokenized at maximum sequence lengths of 512, 1024, and 2048. Gradient accumulation is fixed at 1, gradient checkpointing is enabled, evaluation and checkpoint saving are disabled, the seed is 42, and each run includes a 60 s warmup followed by steady-state measurements. Identical preprocessing and environment settings are maintained across runs [2509.12229].

Three configurations are reported:

| Run | Configuration | Headline outcome |
|---|---|---|
| Run-1 | AdamW (torch), batch size 1, seq 512, fp16 | Baseline |
| Run-2 | PagedAdamW, batch size 2, seq 2048, fp16 | Stress Test |
| Run-3 | PagedAdamW, batch size 2, seq 1024, bf16 | Intermediate |

The detailed results are as follows. Run-1 has wall time 3494 s, tokens processed 1,717,908, throughput 500.3 tok/s, time per 10k tokens 19.99 s, VRAM peak 6234 MB, and energy estimates of 0.19 J/token with \(E_{10k} = 1900\) J at 95 W, or 0.23 J/token with \(E_{10k} = 2298\) J at 115 W [2509.12229]. Run-2 has wall time 972.6 s, tokens processed 573,206, throughput 628.1 tok/s, time per 10k tokens 15.93 s, VRAM peak 8062 MB, and energy estimates of 0.151 J/token with \(E_{10k} = 1513\) J at 95 W, or 0.183 J/token with \(E_{10k} = 1832\) J at 115 W [2509.12229]. Run-3 has wall time 1651.4 s, tokens processed 573,206, throughput 360.2 tok/s, time per 10k tokens 27.76 s, VRAM peak 7949 MB, and energy estimates of 0.264 J/token with \(E_{10k} = 2640\) J at 95 W, or 0.319 J/token with \(E_{10k} = 3193\) J at 115 W [2509.12229].

These configurations are central to interpreting PagedAdamW in the paper. They show it not as an isolated optimizer swap, but as part of a systems configuration involving sequence length, batch size, precision mode, LoRA/QLoRA, and checkpointing under a strict VRAM ceiling.

## 5. Throughput, time efficiency, and energy characteristics

The headline empirical finding is that the fp16 PagedAdamW configuration achieves up to approximately 25% higher throughput than the AdamW baseline: 628.1 tok/s versus 500.3 tok/s, corresponding to a reported gain of +25.5% [2509.12229]. Time per 10k tokens decreases from 19.99 s to 15.93 s, a reduction of 20.3%. These gains are reported while simultaneously enabling a much longer sequence length, 2048 tokens instead of 512, and a larger batch size, 2 instead of 1 [2509.12229].

The study also normalizes energy by tokens processed. Because direct NVML telemetry was unavailable, energy is estimated using board-level power assumptions, with 95 W as a central steady-state estimate and 80–115 W as bounds [2509.12229]. The formulas used are:

$$
R = \frac{N}{t}
$$

$$
T_{10k} = \frac{10{,}000}{R}
$$

$$
E = \sum_{i=1}^{k} P_i \cdot \Delta t_i \;\; \approx P \cdot t
$$

$$
E_{\text{token}} = \frac{E}{N}
$$

$$
E_{10k} = 10{,}000 \times E_{\text{token}}
$$

Under the 95 W assumption, Run-2 yields 0.151 J/token compared with 0.19 J/token for Run-1 and 0.264 J/token for Run-3 [2509.12229]. Under the 115 W assumption, the corresponding values are 0.183, 0.23, and 0.319 J/token. The study interprets this as a throughput-energy linkage: higher throughput correlates with lower joules per token, making PagedAdamW with fp16 the most energy-efficient configuration among the three profiled runs [2509.12229].

The reported energy advantage should, however, be read within the measurement methodology. The values are estimates derived from assumed board power rather than direct telemetry. The paper therefore supports comparative energy analysis within its own setup more strongly than absolute device-level power characterization.

## 6. Precision effects, practical recommendations, and limitations

The paper reports a clear precision effect on this hardware class: bf16 degrades efficiency relative to fp16 on the RTX 4060 [2509.12229]. In the paged configurations, Run-3 uses bf16 and reaches 360.2 tok/s, whereas Run-2 uses fp16 and reaches 628.1 tok/s. At the 95 W estimate, energy per token is 0.264 J/token for bf16 versus 0.151 J/token for fp16. The authors attribute this difference to “bf16 overheads” on this consumer GPU class and conclude that bf16 offers no advantage versus fp16 in their setup [2509.12229]. They do not provide deeper architectural reasons such as memory bandwidth or kernel differences.

For 8 GB GPUs, the practical guidance in the case study is explicit. PagedAdamW is preferred when longer contexts up to 2048 tokens or larger batches must fit within the memory budget, and when higher throughput and lower energy per token are desired relative to the baseline AdamW run [2509.12229]. The recommended configuration from this case study is fp16 mixed precision with batch size 2 and sequence lengths up to 2048, using PagedAdamW, gradient checkpointing, bitsandbytes with 16-bit LoRA adapters, the Hugging Face Transformers Trainer, and PEFT for LoRA integration. The profiling setup keeps gradient accumulation at 1, disables evaluation and checkpoint saves, uses a 60 s warmup, and fixes the seed at 42 [2509.12229].

The caveats are equally important. Paging can introduce overheads and shift memory pressure to CPU RAM; the profiled host has 96 GB RAM [2509.12229]. The optimizer comparison is confounded by simultaneous changes in batch size and sequence length between the baseline and the stress test. The study therefore does not isolate optimizer effects in a strictly ceteris paribus sense. Its reported throughput improvement with PagedAdamW and fp16 is described as a strong signal in this setting, but exact gains may vary with model size, LoRA rank, data pipeline, and GPU architecture [2509.12229].

External validity is limited by scope. Results are reported for a single RTX 4060 with 8 GB VRAM, a 1.5B-parameter model, and a 5k Alpaca subset [2509.12229]. Other consumer GPUs, such as the RTX 3060 or 4070, may show different paging and precision trade-offs; larger models may not fit within 8 GB even with LoRA/QLoRA and paging; and the observed bf16 degradation should not be generalized to datacenter GPUs. The paper also does not provide code snippets or CLI commands. It notes only that, in practice, enabling PagedAdamW with Hugging Face Trainer would typically involve selecting the bitsandbytes optimizer class, such as `bitsandbytes.optim.PagedAdamW`, while not supplying an implementation example [2509.12229].

Taken together, the case study defines PagedAdamW as an implementation-level adaptation of AdamW for memory-constrained fine-tuning. Its significance in this setting lies in preserving AdamW’s mathematical update while altering optimizer-state precision and residency so that longer contexts, higher throughput, and lower estimated energy per token become feasible on a consumer GPU with 8 GB VRAM [2509.12229].

Source: https://www.emergentmind.com/topics/pagedadamw