---
title: 'Adacc: Adaptive Memory Management for LLMs'
url: https://www.emergentmind.com/topics/adacc
type: topic
---

# Adacc: Adaptive Memory Management for LLMs

Searching arXiv for the primary "Adacc" topic and closely related LLM memory-management work.
arXiv search query: "Adacc adaptive compression activation checkpointing LLM memory management"
Adacc is a memory management framework for training large language models that combines adaptive compression and activation checkpointing to reduce GPU memory footprint while controlling recomputation overhead. In the reported formulation, Adacc comprises three modules—layer-specific compression algorithms, an MILP-based scheduling policy, and an adaptive policy evolution mechanism—and is evaluated on GPT-117M, GPT-345M, and GPT-4.7B trained on *The Pile*. The framework is designed to improve the memory–compute trade-off relative to no optimization, full recomputation, and naïve FP16→INT4 quantization, while maintaining model accuracy comparable to the Baseline [2508.00806].

## 1. Problem setting and system structure

Adacc addresses a standard constraint in LLM training: reducing activation memory without incurring the full runtime penalty associated with recomputation. The motivating observation is that recomputation can introduce up to 30% overhead in real-world scenarios, while direct low-bit compression can damage training dynamics if it ignores the tensor statistics specific to Transformer activations [2508.00806].

The framework is organized around three tightly coupled components:

| Component | Role | Mechanism |
|---|---|---|
| Layer-specific compression | Reduce tensor footprint | Outlier-aware 4-bit compression with layer-dependent schemes |
| MILP-based scheduling | Choose memory action per tensor | Recompute, compress, or retain via binary decision variables |
| Adaptive policy evolution | Update policy during training | Recompute statistics at tracking iterations \(1,2,4,8,\dots\) |

This design implies that Adacc is not a pure quantization method and not a pure checkpointing method. Its defining feature is the joint treatment of compression, retention, and recomputation as alternative actions under a single scheduling policy. A plausible implication is that the framework is intended to operate at the systems level rather than as an isolated numerical compression primitive.

## 2. Layer-specific compression and outlier handling

The compression module is built on the observation that many activation tensors in Transformers contain a small fraction of outlier channels whose magnitudes far exceed the bulk of the data. Adacc therefore avoids blindly quantizing all entries from FP16 to INT4. Instead, for each channel \(i\) in a tensor \(X\in\mathbb{R}^{B\times L\times D}\), it computes the Z-score

\[
Z_i \;=\;\frac{\sum_{j}|X_{j,i}| \;-\;\mu}{\sigma}
\]

where \(\mu,\sigma\) are the mean and standard deviation of \(\sum_j |X_{j,i}|\). Channels with \(|Z_i|>3\) are marked as outliers, left in full precision, and zeroed out in the residual tensor [2508.00806].

After outlier removal, Adacc applies 4-bit quantization to the remaining entries in place, using either symmetric or asymmetric quantization. The framework chooses schemes on a per-layer basis: symmetric quantization is used on Q/K/V matrices, asymmetric quantization on Softmax, and a special 1-bit compaction is used for dropout masks. This per-layer selection is central to the method’s claim of preserving model accuracy while still reducing memory pressure.

The compression module is therefore explicitly heterogeneous across tensor types. A common misunderstanding is to treat Adacc as a fixed INT4 activation-quantization scheme. In the reported design, however, quantization policy depends on layer semantics and on channel-level outlier detection, and full-precision preservation is part of the compression path rather than an exception.

## 3. MILP-based scheduling policy

Adacc’s second module formulates tensor handling as an optimization problem. Given \(N\) forward-pass operators \(\{op_i\}\), it defines binary decision variables

\[
R_{i,0}=1 \;\text{(recompute)},\quad
R_{i,1}=1 \;\text{(compress)},\quad
R_{i,2}=1 \;\text{(retain)},
\]

with \(\sum_t R_{i,t}=1\). Let \(M_i\) denote the size of tensor \(i\), \(CRate_i\) the post-compression size fraction, \(T_{\rm comp}^i\) the recomputation time, and \(T_c^i, T_{dc}^i\) the compress and decompress times. The optimization problem is

\[
\min_{R}\quad
\sum_i\bigl[R_{i,0}\,T_{\rm comp}^i + R_{i,1}(T_c^i+T_{dc}^i)\bigr]
\]

subject to three conditions: each tensor receives exactly one action; total GPU memory satisfies

\[
M_{\rm static}+\sum_i M_i\,[R_{i,2}+R_{i,1}\,CRate_i]\le M_{\rm cap};
\]

and dependencies are respected, so that recomputation is not attempted without the necessary inputs [2508.00806].

Because Transformer layers are assumed to be identical, the framework solves one block’s MILP offline via a standard solver, specifically PuLP/Gurobi, in less than \(0.5\) s and then tiles the solution across layers. This assumption is material: it reduces scheduler cost substantially, but it also restricts the direct applicability of the method to architectures with repeated homogeneous blocks.

## 4. Adaptive policy evolution during training

Adacc’s third module responds to the fact that outlier statistics, and hence \(CRate_i\), drift during training. The framework marks iterations as tracking iterations at exponentially growing intervals \(1,2,4,8,\dots\), recomputes the Z-scores and updated \(CRate_i\), and then checks whether the MILP’s optimal \(\{R_{i,t}\}\) has changed. If the optimum changes, the framework hot-swaps in the new policy [2508.00806].

The stated purpose of this mechanism is to accommodate early-stage volatility while ensuring diminishing overhead over time. Since tracking intervals grow exponentially, the policy-adjustment cost decreases as training proceeds, even though the method remains adaptive. This suggests that Adacc treats policy selection as a dynamic control problem rather than a one-time compile-time decision.

Operationally, the adaptive loop binds the other two modules together. Compression quality determines \(CRate_i\); \(CRate_i\) enters the MILP; and the MILP output feeds back into runtime tensor treatment. The adaptive mechanism therefore acts as the framework’s coordination layer.

## 5. Memory model, compute overhead, and empirical results

Adacc expresses the baseline memory footprint as

\[
M_{\rm tot}^{\rm base}=M_{\rm static}+\sum_i M_i,
\]

and the optimized memory footprint as

\[
M_{\rm tot}^{\rm Adacc}
=
M_{\rm static}
+\sum_i M_i\bigl[R_{i,2} + R_{i,1}\,CRate_i\bigr].
\]

The total per-step overhead is

\[
T_{\rm over}
=\sum_i\Bigl[R_{i,0}\,T_{\rm comp}^i + R_{i,1}\,(T_c^i+T_{dc}^i)\Bigr],
\]

so that step time is \(T_{\rm step}=T_{\rm fp}+T_{\rm bp}+T_{\rm over}\), and throughput is \(\Phi=\tfrac{1}{T_{\rm step}}\) [2508.00806].

The reported experiments use 8× NVIDIA V100 32 GB with NVLink and 2× Xeon Gold. Models are GPT-117M, GPT-345M, and GPT-4.7B, trained on *The Pile*. Baselines include no optimization (“Baseline”), full recomputation (Megatron-style), and naïve FP16→INT4 quantization.

| Measure | Reported result | Comparison |
|---|---|---|
| Maximum batch size | \(\uparrow 2.4\!-\!7.6\times\) | vs. Baseline |
| Maximum batch size | up to \(3.2\times\) | vs. quant |
| Throughput gain | \(1.01\!-\!1.37\times\) | vs. full recompute |
| Throughput gain | \(1.09\!-\!1.28\times\) | vs. quant |
| Validation-loss gap | \(\le 0.5\%\) | relative to baseline |
| Downstream zero-shot average drop | \(\le 0.5\%\) | relative to baseline |

The paper also states that activation memory is reduced by up to \(3.2\times\) relative to naïve quantization and \(7.6\times\) relative to baseline. The experimental interpretation offered by these figures is that Adacc attempts to dominate the two obvious single-strategy baselines: full recomputation, which saves memory but increases runtime, and naïve low-bit compression, which saves memory but risks destabilizing training.

## 6. Assumptions, limitations, and nomenclature

The reported implementation makes three explicit assumptions. First, Transformer layers are identical, so the MILP solution for one block can be tiled across layers. Second, the Z-score threshold is fixed at \(3\). Third, compression always uses 4-bit quantization, with no finer granularity [2508.00806]. These assumptions simplify the optimization problem and the runtime implementation, but they also delimit the method’s design space.

The paper identifies several extensions. One is adaptation to heterogeneous blocks, including MoE and CNN + Transformer hybrids, by clustering layers into “types.” Another is the use of new hardware such as Ampere FP8 and NVComp lossless GPU codecs for more aggressive compression. A third is joint optimization of pipeline or parallel strategies together with Adacc’s memory plan for multi-node scaling. These proposed directions indicate that the framework is presented as a systems substrate rather than a closed-form solution to all memory-management scenarios.

The term *Adacc* also admits domain-specific ambiguity. In machine learning, “AdaCC” denotes a parameter-free cost-sensitive boosting method for imbalanced classification that dynamically adjusts misclassification costs over boosting rounds [2209.08309]. In astronomical instrumentation, “Adacc” is used as shorthand for an advanced apochromatic triplet atmospheric dispersion corrector designed for the Magellan Adaptive Optics system and VisAO camera [1308.4844]. These are unrelated to the LLM memory-management framework described here, and the capitalization difference is not always sufficient to prevent confusion in cross-domain citation or indexing.

Source: https://www.emergentmind.com/topics/adacc