Adacc: Adaptive Memory Management for LLMs

• Adacc is a memory management framework that integrates adaptive compression and activation checkpointing to reduce GPU memory footprint during LLM training.
• It employs layer-specific compression with outlier handling and a MILP-based scheduling policy to balance memory savings against recomputation overhead.
• Empirical evaluations on GPT models show significant improvements in batch size and throughput while maintaining model accuracy comparable to baseline.

Searching arXiv for the primary "Adacc" topic and closely related LLM memory-management work. arXiv search query: "Adacc adaptive compression activation checkpointing LLM memory management" Adacc is a memory management framework for training LLMs that combines adaptive compression and activation checkpointing to reduce GPU memory footprint while controlling recomputation overhead. In the reported formulation, Adacc comprises three modules—layer-specific compression algorithms, an MILP-based scheduling policy, and an adaptive policy evolution mechanism—and is evaluated on GPT-117M, GPT-345M, and GPT-4.7B trained on The Pile. The framework is designed to improve the memory–compute trade-off relative to no optimization, full recomputation, and naïve FP16→INT4 quantization, while maintaining model accuracy comparable to the Baseline (Chen et al., 1 Aug 2025).

1. Problem setting and system structure

Adacc addresses a standard constraint in LLM training: reducing activation memory without incurring the full runtime penalty associated with recomputation. The motivating observation is that recomputation can introduce up to 30% overhead in real-world scenarios, while direct low-bit compression can damage training dynamics if it ignores the tensor statistics specific to Transformer activations (Chen et al., 1 Aug 2025).

The framework is organized around three tightly coupled components:

Component	Role	Mechanism
Layer-specific compression	Reduce tensor footprint	Outlier-aware 4-bit compression with layer-dependent schemes
MILP-based scheduling	Choose memory action per tensor	Recompute, compress, or retain via binary decision variables
Adaptive policy evolution	Update policy during training	Recompute statistics at tracking iterations $1,2,4,8,\dots$

This design implies that Adacc is not a pure quantization method and not a pure checkpointing method. Its defining feature is the joint treatment of compression, retention, and recomputation as alternative actions under a single scheduling policy. A plausible implication is that the framework is intended to operate at the systems level rather than as an isolated numerical compression primitive.

2. Layer-specific compression and outlier handling

The compression module is built on the observation that many activation tensors in Transformers contain a small fraction of outlier channels whose magnitudes far exceed the bulk of the data. Adacc therefore avoids blindly quantizing all entries from FP16 to INT4. Instead, for each channel $i$ in a tensor $X\in\mathbb{R}^{B\times L\times D}$, it computes the Z-score

$Z_i \;=\;\frac{\sum_{j}|X_{j,i}| \;-\;\mu}{\sigma}$

where $\mu,\sigma$ are the mean and standard deviation of $\sum_j |X_{j,i}|$. Channels with $|Z_i|>3$ are marked as outliers, left in full precision, and zeroed out in the residual tensor (Chen et al., 1 Aug 2025).

After outlier removal, Adacc applies 4-bit quantization to the remaining entries in place, using either symmetric or asymmetric quantization. The framework chooses schemes on a per-layer basis: symmetric quantization is used on Q/K/V matrices, asymmetric quantization on Softmax, and a special 1-bit compaction is used for dropout masks. This per-layer selection is central to the method’s claim of preserving model accuracy while still reducing memory pressure.

The compression module is therefore explicitly heterogeneous across tensor types. A common misunderstanding is to treat Adacc as a fixed INT4 activation-quantization scheme. In the reported design, however, quantization policy depends on layer semantics and on channel-level outlier detection, and full-precision preservation is part of the compression path rather than an exception.

3. MILP-based scheduling policy

Adacc’s second module formulates tensor handling as an optimization problem. Given $N$ forward-pass operators $\{op_i\}$, it defines binary decision variables

$$R_{i,0}=1 \;\text{(recompute)},\quad R_{i,1}=1 \;\text{(compress)},\quad R_{i,2}=1 \;\text{(retain)},$$

with $i$0. Let $i$1 denote the size of tensor $i$2, $i$3 the post-compression size fraction, $i$4 the recomputation time, and $i$5 the compress and decompress times. The optimization problem is

$i$6

subject to three conditions: each tensor receives exactly one action; total GPU memory satisfies

$i$7

and dependencies are respected, so that recomputation is not attempted without the necessary inputs (Chen et al., 1 Aug 2025).

Because Transformer layers are assumed to be identical, the framework solves one block’s MILP offline via a standard solver, specifically PuLP/Gurobi, in less than $i$8 s and then tiles the solution across layers. This assumption is material: it reduces scheduler cost substantially, but it also restricts the direct applicability of the method to architectures with repeated homogeneous blocks.

4. Adaptive policy evolution during training

Adacc’s third module responds to the fact that outlier statistics, and hence $i$9, drift during training. The framework marks iterations as tracking iterations at exponentially growing intervals $X\in\mathbb{R}^{B\times L\times D}$0, recomputes the Z-scores and updated $X\in\mathbb{R}^{B\times L\times D}$1, and then checks whether the MILP’s optimal $X\in\mathbb{R}^{B\times L\times D}$2 has changed. If the optimum changes, the framework hot-swaps in the new policy (Chen et al., 1 Aug 2025).

The stated purpose of this mechanism is to accommodate early-stage volatility while ensuring diminishing overhead over time. Since tracking intervals grow exponentially, the policy-adjustment cost decreases as training proceeds, even though the method remains adaptive. This suggests that Adacc treats policy selection as a dynamic control problem rather than a one-time compile-time decision.

Operationally, the adaptive loop binds the other two modules together. Compression quality determines $X\in\mathbb{R}^{B\times L\times D}$3; $X\in\mathbb{R}^{B\times L\times D}$4 enters the MILP; and the MILP output feeds back into runtime tensor treatment. The adaptive mechanism therefore acts as the framework’s coordination layer.

5. Memory model, compute overhead, and empirical results

Adacc expresses the baseline memory footprint as

$X\in\mathbb{R}^{B\times L\times D}$5

and the optimized memory footprint as

$X\in\mathbb{R}^{B\times L\times D}$6

The total per-step overhead is

$X\in\mathbb{R}^{B\times L\times D}$7

so that step time is $X\in\mathbb{R}^{B\times L\times D}$8, and throughput is $X\in\mathbb{R}^{B\times L\times D}$9 (Chen et al., 1 Aug 2025).

The reported experiments use 8× NVIDIA V100 32 GB with NVLink and 2× Xeon Gold. Models are GPT-117M, GPT-345M, and GPT-4.7B, trained on The Pile. Baselines include no optimization (“Baseline”), full recomputation (Megatron-style), and naïve FP16→INT4 quantization.

Measure	Reported result	Comparison
Maximum batch size	$Z_i \;=\;\frac{\sum_{j}|X_{j,i}| \;-\;\mu}{\sigma}$0	vs. Baseline
Maximum batch size	up to $Z_i \;=\;\frac{\sum_{j}|X_{j,i}| \;-\;\mu}{\sigma}$1	vs. quant
Throughput gain	$Z_i \;=\;\frac{\sum_{j}|X_{j,i}| \;-\;\mu}{\sigma}$2	vs. full recompute
Throughput gain	$Z_i \;=\;\frac{\sum_{j}|X_{j,i}| \;-\;\mu}{\sigma}$3	vs. quant
Validation-loss gap	$Z_i \;=\;\frac{\sum_{j}|X_{j,i}| \;-\;\mu}{\sigma}$4	relative to baseline
Downstream zero-shot average drop	$Z_i \;=\;\frac{\sum_{j}|X_{j,i}| \;-\;\mu}{\sigma}$5	relative to baseline

The paper also states that activation memory is reduced by up to $Z_i \;=\;\frac{\sum_{j}|X_{j,i}| \;-\;\mu}{\sigma}$6 relative to naïve quantization and $Z_i \;=\;\frac{\sum_{j}|X_{j,i}| \;-\;\mu}{\sigma}$7 relative to baseline. The experimental interpretation offered by these figures is that Adacc attempts to dominate the two obvious single-strategy baselines: full recomputation, which saves memory but increases runtime, and naïve low-bit compression, which saves memory but risks destabilizing training.

6. Assumptions, limitations, and nomenclature

The reported implementation makes three explicit assumptions. First, Transformer layers are identical, so the MILP solution for one block can be tiled across layers. Second, the Z-score threshold is fixed at $Z_i \;=\;\frac{\sum_{j}|X_{j,i}| \;-\;\mu}{\sigma}$8. Third, compression always uses 4-bit quantization, with no finer granularity (Chen et al., 1 Aug 2025). These assumptions simplify the optimization problem and the runtime implementation, but they also delimit the method’s design space.

The paper identifies several extensions. One is adaptation to heterogeneous blocks, including MoE and CNN + Transformer hybrids, by clustering layers into “types.” Another is the use of new hardware such as Ampere FP8 and NVComp lossless GPU codecs for more aggressive compression. A third is joint optimization of pipeline or parallel strategies together with Adacc’s memory plan for multi-node scaling. These proposed directions indicate that the framework is presented as a systems substrate rather than a closed-form solution to all memory-management scenarios.

The term Adacc also admits domain-specific ambiguity. In machine learning, “AdaCC” denotes a parameter-free cost-sensitive boosting method for imbalanced classification that dynamically adjusts misclassification costs over boosting rounds (Iosifidis et al., 2022). In astronomical instrumentation, “Adacc” is used as shorthand for an advanced apochromatic triplet atmospheric dispersion corrector designed for the Magellan Adaptive Optics system and VisAO camera (Kopon et al., 2013). These are unrelated to the LLM memory-management framework described here, and the capitalization difference is not always sufficient to prevent confusion in cross-domain citation or indexing.