---
title: Adaptive Mixed-bit Activation Quantization
url: https://www.emergentmind.com/topics/adaptive-mixed-bit-activation-quantization-amaq-8e039608-f7eb-447f-97fb-2fb0abc3e65b
type: topic
---

# Adaptive Mixed-bit Activation Quantization

Searching arXiv for the cited papers and closely related mixed-precision activation quantization work.
Adaptive Mixed bit Activation Quantization (AMAQ) denotes a class of quantization methods in which activation precision is assigned non-uniformly rather than fixed globally, typically under an explicit communication, memory, or bit-budget constraint. In the most direct usage of the term, AMAQ refers to the server-client split-learning method introduced for collaborative parameter-efficient fine-tuning of large language models, where activations and gradients are progressively compressed from higher precision to low precision and the effective bit-width is adaptively allocated across features using learnable gates and bit regularization [2510.05468]. In a broader research sense, AMAQ also describes a family of activation-quantization strategies that vary precision across layers, tokens, windows, or neurons, often to manage activation outliers, heterogeneous sensitivity, or communication bottlenecks [2606.13054] [2507.19131] [2311.06798].

## 1. Concept and scope

AMAQ is concerned specifically with **activation precision**, not merely weight quantization. This distinction is important because many mixed-precision schemes for neural networks and large language models are weight-only, even when they are activation-aware in calibration or bit allocation. By contrast, AMAQ methods explicitly vary the precision used for activations themselves, either during inference, training, or both [2510.05468] [2606.13054].

The core premise is that activations are heterogeneous. Their quantization difficulty varies across layers, channels, tokens, windows, or neurons because of differences in dynamic range, outliers, sparsity, and downstream sensitivity. A fixed activation bit-width therefore tends to be inefficient: it over-allocates bits to robust components and under-allocates them to fragile ones. AMAQ methods seek to correct this by assigning more bits where precision matters and fewer bits where redundancy or robustness permits it [2507.19131] [2605.25054] [2506.01352].

This general idea appears in several distinct operational regimes. In split or collaborative training, AMAQ is used to reduce communication of transmitted activations and gradients while maintaining optimization stability [2510.05468] [2506.01352]. In post-training quantization for LLM inference, AMAQ appears as layer-wise mixed-precision activation assignment under a global bit budget, often to make very low-bit activation quantization feasible after suppressing outliers [2606.13054]. In vision transformers, AMAQ can be intra-layer and input-dependent, assigning different activation bit-widths to different windows based on their importance [2507.19131]. In quantization-aware training, the adaptation may occur at layer, global-network, or neuron level [2404.16876] [2605.25054].

## 2. Formal structure of adaptive mixed-bit activation assignment

Most AMAQ methods share three ingredients: a decision variable for activation precision, a quality proxy or task loss, and a resource constraint or regularizer. The decision variable may be discrete layer bits, continuous relaxed bit parameters, per-token routing, or per-neuron thresholded precision strengths.

A direct and explicit formulation appears in TWLA’s activation module, Inter-Layer Aware Activation Mixed Precision (ILA-AMP), which assigns each layer $\ell$ an activation bit-width $b_\ell \in \mathcal{B}$ with
$$
\mathcal{B}=\{2,4,6,8\}.
$$
The assignment vector is $\mathbf{b}=(b_1,\ldots,b_L)$, and the quality signal is validation negative log-likelihood:
$$
v_{\mathrm{NLL}(\mathbf{b}) = \mathbb{E}_{(\mathbf{x},t)\sim\mathcal{D} \Big[ -\log p_{\boldsymbol{\theta}(x_{t+1}\mid \mathbf{x}_{\le t}; \mathbf{b}) \Big].
$$
The global allocation objective is
$$
\min_{\{b_\ell\} \sum_{\ell=1}^L C_\ell(b_\ell) + \sum_{\ell=2}^L K_{\ell-1,\ell}(b_{\ell-1}, b_\ell), \quad \text{s.t. }\sum_{\ell=1}^L b_\ell \le B,
$$
where $C_\ell(b)$ is isolated layer sensitivity and $K_{\ell-1,\ell}(b',b)$ captures adjacent-layer interaction costs [2606.13054]. This is a canonical AMAQ formulation: discrete, globally budgeted, and explicitly interaction-aware.

A different formulation appears in collaborative split learning. In AMAQ for collaborative PEFT, effective bit-width is controlled by learnable gates:
$$
\text{Bit-width} = min + (max - min)\times \sigma(\alpha \cdot Q).
$$
The corresponding bit regularization is
$$
\text{Bits\_Loss} = \frac{1}{n}\sum_{i=1}^{n} \sigma(\alpha \cdot q_i)^2,
$$
and the total loss is
$$
\text{Loss} = \text{QAT\_Loss} + \beta \times \text{Bits\_Loss}.
$$
This yields a training-time adaptive mechanism in which the task loss implicitly preserves higher precision for important features while $\beta$ pushes the average bit-width downward [2510.05468].

In neuron-level mixed-precision QAT, each neuron has an activation precision strength $s_j^{(a)}$ mapped to a discrete bit-width from
$$
B_a = \{4,8,16\},
$$
and activation quantization is
$$
z_{i,j}^{(q)} = Q^{(b(s_j^{(a)}))}\big(\operatorname{clip}(z_{i,j},0,a_j)\big).
$$
Here $a_j$ is a neuron-specific clipping range, and the adaptation is static after training but learned jointly with the model [2605.25054].

These variants differ in granularity and optimization, but all instantiate the same AMAQ principle: activation precision becomes an adaptive variable rather than a fixed hyperparameter.

## 3. Principal methodological patterns

AMAQ methods can be divided by the granularity at which bits are adapted and by the signal used to drive adaptation.

### 3.1 Layer-wise mixed precision

Layer-wise AMAQ is the most common form in LLM post-training quantization. TWLA’s ILA-AMP is exemplary because it does not only measure isolated sensitivity; it also models adjacent-layer coupling to avoid failure cascades in very low-bit settings [2606.13054]. This is particularly relevant when an upstream transform changes activation quantizability unevenly across layers.

MetaMix is also layer-wise, though in a QAT/search setting rather than PTQ. It searches per-layer activation precision while holding a single global weight precision fixed, and it emphasizes that activation bit exploration destabilizes activation statistics unless the search is performed on a mixed-precision-aware meta-state [2311.06798]. AdaQAT is related but much coarser: it learns a single global activation bit-width for the whole network rather than layer-wise mixed precision [2404.16876]. AdaBits and the 2021 adaptive-bit training work likewise support multiple activation precisions at runtime, but as whole-network switchable modes rather than simultaneous per-layer mixed assignment [1912.09666] [2103.01435].

### 3.2 Intra-layer or token/window-wise mixed precision

A stricter AMAQ interpretation often emphasizes heterogeneous precision within a layer. MixA-Q is a direct instance: in Swin-based vision transformers, windows are scored by $L_2$ norm and split into high-precision and low-precision branches within the same block. Less important windows are processed at lower activation precision, while important windows remain at higher precision [2507.19131]. The method therefore performs mixed-bit activation quantization at window granularity and can be dynamic with respect to the input.

STaMP is similar in spirit but uses a sequence-domain transform first. It applies an invertible sequence transform, typically DWT, to concentrate energy into early transformed tokens, then keeps a small number of those transformed tokens at 8 bits and quantizes the remainder at 4 bits. Its theoretical bound,
$$
L_{Q;T}\le \frac{d}{2}\sum_{i=1}^s \frac{e_i}{(2^{b_i}-1)^2},
$$
with $e_i=\mathbb{E}[\|\hat x_i\|^2]$, formalizes the idea that transformed tokens with larger energy should receive more bits [2510.26771]. This is AMAQ-like, though the deployed selection rule is mostly static after choosing the transform and high-precision token count.

TAH-Quant applies a different intra-sequence strategy for pipeline-parallel training. It computes token entropy
$$
p_k = \frac{|a_{i,j,k}|}{\|\mathbf{a}_{i,j}\|_1+\epsilon},
\qquad
\mathcal{H}\left(\mathbf{a}_{i,j}\right) = \sum_{k=1}^C p_k \log\left(p_k+\varsigma\right),
$$
then assigns top-$p\%$ tokens to INT4 and the remainder to INT3. Quantization parameters are tile-wise within each token, so bit allocation is token-wise while scale and zero-point are finer-grained [2506.01352].

### 3.3 Neuron-wise or very fine-grained adaptation

NMP-QAT pushes AMAQ to neuron granularity. Each neuron independently learns activation precision and clipping range, starting from 4-bit activations and increasing to 8 or 16 bits only if training signals justify it [2605.25054]. This suggests a fine-grained AMAQ regime in which activation heterogeneity is modeled within layers rather than only across them.

A plausible implication is that the main unresolved systems challenge for AMAQ is no longer only algorithmic bit allocation but also hardware support: neuron-wise mixed-bit activation execution is substantially harder to realize efficiently than layer-wise or token-wise schemes. The paper itself notes that sub-8-bit arithmetic and mixed-precision-capable accelerators are required for practical gains [2605.25054].

## 4. Importance signals and allocation criteria

AMAQ methods differ sharply in what they treat as the relevant measure of activation importance or difficulty.

A first class uses **task-level loss proxies**. TWLA uses validation NLL measured under candidate bit assignments, both for unary sensitivity and pairwise adjacent-layer interaction terms [2606.13054]. This is arguably the most principled among current AMAQ approaches because the allocation target is directly tied to model quality.

A second class uses **activation statistics**. TAH-Quant uses token entropy as a proxy for information distribution [2506.01352]. MixA-Q uses window $L_2$ norms to identify important windows [2507.19131]. AD-based mixed-precision quantization uses activation density,
$$
AD = \frac{\# \text{ nonzero activations}}{\# \text{ total activations}},
$$
and updates per-layer bit-width by
$$
k_l^{(\text{iter})} = round\left(k_l^{(\text{iter}-1)} * AD_l\right),
$$
quantizing both activations and weights of a layer to the same bit-width [2101.04354]. This is activation-driven but more heuristic and coupled than later AMAQ formulations.

A third class uses **learned latent importance**. Collaborative AMAQ does not define a handcrafted importance score; instead, the gates $Q$ learn which features should retain more bits under task loss plus bit regularization [2510.05468]. NMP-QAT similarly lets gradient signals move precision strengths across thresholds rather than assigning bits from a closed-form saliency metric [2605.25054].

A fourth class uses **stability-aware training surrogates**. MetaMix does not compute importance directly from activation magnitudes; instead, it first learns a mixed-precision-aware meta-state by averaging losses across candidate activation bit-widths,
$$
\mathcal{L}(w) = \frac{1}{B}\sum_{i=1}^{B}{\mathcal{L}(w, b_i)},
$$
then searches per-layer bits on frozen weights [2311.06798]. This treats activation instability itself as the primary obstacle to reliable mixed-bit assignment.

## 5. Activation outliers, transforms, and quantizability shaping

A persistent obstacle for AMAQ is the heavy-tailed or outlier-dominated nature of activations. Many activation-mixed-precision methods therefore include a transformation or shaping step before bit allocation.

TWLA’s KOTMS learns a Kronecker-structured orthogonal rotation
$$
\mathbf{R} = \mathbf{R}_1\otimes\mathbf{R}_2
$$
to reshape weights into a ternary-friendly distribution and, through the shared orthogonal transform, statistically suppress activation outliers. The paper formalizes the activation-side effect via the peak-to-RMS ratio
$$
\kappa(\mathbf{x}) := \frac{\sqrt{m}\,\|\mathbf{x}\|_\infty}{\|\mathbf{x}\|_2},
$$
and derives a bound showing that orthogonal mixing reduces the maximum coordinate relative to total energy with high probability [2606.13054]. AMAQ then becomes more effective because the activation distributions are less dominated by a few extreme channels.

STaMP performs a sequence-domain transform rather than a feature-domain rotation. Its contribution is to show that mixed-bit activation assignment becomes easier when activations are represented in a domain where importance is concentrated and ordered [2510.26771].

TAH-Quant uses a Hadamard transform with pivot swapping only on tiles whose outlier ratio
$$
r = \frac{|\alpha^{(1)}|}{|\alpha^{(2)}|+\varrho}
$$
exceeds a threshold $\tau=2.0$. This selective transform reduces tile dynamic range before asymmetric quantization [2506.01352].

AQuant, though not mixed-bit, is highly relevant because it treats activation quantization itself as adaptive at fixed bit-width. It replaces the standard rounding border $0.5$ by a value-dependent function. The ideal element-wise border is
$$
B^E(x) = \frac{\Delta w}{w+\Delta w}x + \frac{1}{2},
$$
and the practical coarse-grained border is
$$
B^{o_c}(x) = b_1 \cdot x + b_0.
$$
This minimizes output error more directly than scalar nearest rounding and is especially useful in ultra-low-bit activation settings [2208.11945]. Although AQuant is not AMAQ in the bit-allocation sense, it shows that activation adaptivity can occur inside a fixed bit-width as well as across different bit-widths.

## 6. Representative systems and empirical evidence

The most explicit AMAQ paper in the narrow sense is the collaborative split-learning method titled "AMAQ: Adaptive Mixed-bit Activation Quantization for Collaborative Parameter Efficient Fine-tuning" [2510.05468]. Under the same bit budgets, it reports about 2.5% higher generation accuracy and about 1.3% better classification accuracy than fixed-precision approaches for models such as LLaMA3-8B and Qwen2.5-7B [2510.05468]. In distributed experiments, AMAQ 4 improves over AQ-SGD 4 while adding only modest transmission overhead: for a LoRA-only scenario, transmission increases from 6.8 MB to 7.4 MB per batch while GSM8K accuracy improves from 53.15 to 54.28 [2510.05468].

TWLA provides some of the strongest evidence for layer-wise AMAQ under aggressive PTQ for LLM inference. In the W1.58A4 setting with mixed-precision activations, it reports WikiText2 perplexities such as 8.31 for LLaMA2-7B, 6.68 for LLaMA2-13B, and 4.77 for LLaMA2-70B [2606.13054]. The activation-side ablation is especially striking: on LLaMA2-13B at A4, E2M-ATQ+KOTMS without ILA-AMP gives C4 perplexity 25.03 / MMLU 27.52, while adding ILA-AMP improves to 10.07 / 38.17 [2606.13054]. This indicates that in the very low-bit regime, adaptive mixed-bit activation assignment is not ancillary but necessary.

MixA-Q shows that intra-layer AMAQ can improve both efficiency and robustness in vision transformers. On COCO detection with Mask R-CNN and Swin-Tiny under W4A4 QAT, MixA-Q achieves a lossless roughly 1.24–1.25× speedup at effective 3.2-bit activations and 1.35× speedup with less than 1% mAP loss, outperforming uniform W4A3 at similar cost [2507.19131]. It also improves the W4A4 baseline mAP by 0.7%, reducing quantization degradation by 24% [2507.19131].

MetaMix demonstrates that static per-layer AMAQ can outperform fixed activation precision at similar average activation bit-widths. For MobileNet-v2, MetaMix at 3.98/4 activation/weight precision reaches 72.60% top-1 on ImageNet, exceeding several fixed 4/4 baselines [2311.06798]. Its central empirical claim is that fixed-meta-state search yields better layer assignments and much lower search cost than conventional NAS-style mixed-precision search [2311.06798].

NMP-QAT provides evidence for fine-grained activation adaptivity in tabular and telecom settings. In the weights+activations regime, average learned activation bits are often around 4–5 bits on several tasks, such as 4.855 on KVS and 4.115 on Higgs, well below 8-bit fixed baselines while improving or preserving accuracy [2605.25054]. On attention-heavy TabFormer tasks, adaptive activation precision dramatically outperforms uniform activation quantization baselines, which degrade severely [2605.25054].

TAH-Quant shows that token-adaptive mixed 3/4-bit activation quantization is effective in distributed pipeline training. It reports up to 4.3× end-to-end speedup without compromising convergence, and its adaptive bit allocation ablation shows faster loss decrease than non-adaptive variants [2506.01352].

## 7. Relation to adjacent research and common misconceptions

One common misconception is that any activation-aware quantization method is AMAQ. This is not correct. Several important papers are activation-aware but not activation-mixed-bit methods. CMPQ uses activation distributions to assign **weight channel** precision, yet activations themselves are not quantized [2410.13056]. RAMP uses activation statistics and Scale Folding to guide **weight-only** mixed-precision PTQ, while exported inference retains FP16 activations [2603.17891]. Such methods are related conceptually but should not be classified as AMAQ proper.

A second misconception is that adaptive activation quantization necessarily means per-layer mixed precision. In fact, the literature spans multiple granularities. AdaQAT adapts a single global activation precision during training and is therefore only a coarse precursor to richer AMAQ formulations [2404.16876]. AdaBits and related switchable-bit networks support multiple activation precisions at runtime, but as global network modes rather than simultaneous mixed-bit assignments [1912.09666] [2103.01435]. By contrast, MixA-Q, TAH-Quant, STaMP, and NMP-QAT allocate activation precision within a model configuration across windows, tokens, or neurons [2507.19131] [2506.01352] [2510.26771] [2605.25054].

A third misconception is that the essential difficulty is bit allocation alone. Several papers indicate that activation stability and distribution shaping are equally fundamental. MetaMix identifies activation instability during bit search as a distinct optimization pathology [2311.06798]. TWLA shows that orthogonal activation smoothing is a prerequisite for effective low-bit mixed precision [2606.13054]. AQuant shows that even with fixed bits, adaptive rounding can substantially reduce activation-induced output error [2208.11945].

## 8. Limitations and open questions

Despite rapid progress, AMAQ remains constrained by several unresolved issues.

The first is **systems realizability**. Fine-grained mixed-bit activations are difficult to execute efficiently on current accelerators, especially when precision varies at neuron, token, or window level. Papers such as NMP-QAT acknowledge that realizing theoretical savings requires mixed-precision-capable hardware [2605.25054]. MixA-Q and STaMP report BOPs or moderate kernel overheads, but wall-clock gains depend heavily on implementation support [2507.19131] [2510.26771].

The second is **metadata and routing overhead**. Token-wise or window-wise AMAQ needs selectors, masks, or branch-specific quantizer metadata. TAH-Quant and MixA-Q demonstrate practicality, but detailed accounting of all control overhead is still limited in many works [2506.01352] [2507.19131].

The third is **calibration dependence** in PTQ settings. TWLA’s ILA-AMP uses 128 WikiText2 calibration samples of sequence length 2048, and the learned mixed-bit assignments are model-specific rather than transferable [2606.13054]. This suggests that AMAQ for PTQ remains sensitive to calibration data quality and target-model idiosyncrasies.

The fourth is **objective mismatch**. Many methods optimize bit budgets or average precision rather than hardware-grounded latency or energy. TWLA uses a global bit budget $\sum_\ell b_\ell \le B$ [2606.13054]. Collaborative AMAQ regularizes gate activations rather than an explicit transmission-cost formula [2510.05468]. A plausible implication is that future AMAQ systems will need tighter coupling between bit allocation and actual hardware/runtime cost models.

The fifth is **granularity selection**. Layer-wise AMAQ is simpler and easier to deploy; token- or neuron-wise AMAQ can be more expressive but harder to train and implement. Current literature does not yet establish a universally preferred granularity across model families and hardware environments.

## 9. Summary

Adaptive Mixed bit Activation Quantization is the general strategy of assigning activation precision non-uniformly in response to heterogeneous importance, sensitivity, or communication cost. In its narrowest usage, AMAQ refers to collaborative split-learning methods that progressively compress activations and gradients from 6–8 bits toward 3–4 bits, using trainable gates and bit regularization to allocate precision across features [2510.05468]. In the broader literature, AMAQ encompasses layer-wise NLL-budgeted activation assignment for low-bit LLM inference [2606.13054], window-wise mixed-precision routing in vision transformers [2507.19131], token-adaptive activation communication compression [2506.01352], and neuron-level learned activation precision in QAT [2605.25054].

Across these variants, a consistent picture emerges. Activation quantization is difficult primarily because activations are heterogeneous, dynamic, and often outlier-dominated. Effective AMAQ methods therefore combine adaptive bit allocation with stabilization mechanisms such as transforms, interaction-aware optimization, progressive scheduling, or learned clipping and gating [2606.13054] [2311.06798] [2510.26771] [2208.11945]. The result is a shift from viewing activation precision as a fixed design choice to treating it as a structured optimization variable. In current research, this shift is one of the main enablers of practical low-bit activation quantization in both training and inference.

Source: https://www.emergentmind.com/topics/adaptive-mixed-bit-activation-quantization-amaq-8e039608-f7eb-447f-97fb-2fb0abc3e65b