Inter-Layer Aware Activation Mixed Precision

• The paper introduces ILA-AMP, which explicitly models adjacent layer activation quantization errors to optimize mixed-precision bit allocation.
• It employs dynamic programming to assign layer-wise bit-widths under a global budget, mitigating error cascades in transformer networks.
• Empirical results on LLaMA2 and Qwen models demonstrate significant improvements in perplexity and MMLU scores over uniform precision schemes.

Inter-Layer Aware Activation Mixed Precision (ILA-AMP) is the third component of TWLA, a post-training quantization framework for LLMs that targets the joint regime of ternary weights and low-bit activations. Within TWLA, ILA-AMP is the activation-side mechanism that makes 4-bit (or mixed 2–6-bit) activations work reliably together with ternary (1.58-bit) weights by explicitly modeling how activation quantization errors propagate across adjacent layers, allocating layer-wise activation bit-widths under a global bit budget, and preventing performance cascades caused by a small number of weak layers that remain hard to quantize after orthogonal reshaping (Zhao et al., 11 Jun 2026).

1. Position within TWLA

TWLA comprises three tightly coupled parts. E2M-ATQ improves weight ternarization by a Euclidean-to-manifold procedure that minimizes layer output error given ternary codes. KOTMS introduces a Kronecker-structured orthogonal rotation that reshapes weights into a ternary-friendly tri-modal distribution and, via the same orthogonal transform, statistically suppresses activation outliers. ILA-AMP then operates on the activation side: given that KOTMS improves activation statistics unevenly across layers, it explicitly models second-order interactions between adjacent layers in the validation NLL, chooses layer-wise activation bit-widths under a global bit budget, and thereby prevents performance cascades triggered by a few weak layers (Zhao et al., 11 Jun 2026).

The central motivation is that, even after KOTMS, activation “quantizability” is heterogeneous across the Transformer stack. Some layers become much easier to quantize, while others retain residual outliers or high sensitivity. Uniform activation quantization, such as assigning all layers to A4, therefore fails because the remaining weak layers can dominate the final error. ILA-AMP addresses not only per-layer sensitivity but also local error propagation across layers, so the relevant object is not an isolated layer score but an inter-layer-aware mixed-precision schedule.

In this sense, ILA-AMP is the part of TWLA that converts the uneven activation-side gains induced by KOTMS into a usable deployment configuration. E2M-ATQ stabilizes ternary weights at the layer-output level; KOTMS globally rotates weights and activations; ILA-AMP allocates activation precision so that those two preceding transformations do not collapse under uniform low-bit activation quantization.

2. Objective function and second-order surrogate

ILA-AMP considers a Transformer with $L$ layers and assigns each layer $\ell$ an activation bit-width $b_\ell$ from the discrete set

$\mathcal{B} = \{2,4,6,8\}.$

For an activation configuration $\mathbf{b}=(b_1,\ldots,b_L)$, the value function is the per-token negative log-likelihood on a validation set $\mathcal{D}$: $$v_{\mathrm{NLL}}(\mathbf{b}) = \mathbb{E}_{(\mathbf{x},t)\sim\mathcal{D}} \Big[ -\log p_{\boldsymbol{\theta}}(x_{t+1}\mid \mathbf{x}_{\le t}; \mathbf{b}) \Big].$$ Rather than searching over all $\mathbf{b}\in\mathcal{B}^L$, ILA-AMP builds a second-order surrogate of how $v_{\mathrm{NLL}}$ changes as layer precisions are reduced (Zhao et al., 11 Jun 2026).

The reference point is the high-precision activation configuration

$$b_{\max}=8,\qquad \mathbf{b}_{\max}=(b_{\max},\dots,b_{\max}),$$

with baseline

$\ell$0

For each layer $\ell$1 and candidate bit $\ell$2, the first-order cost is

$\ell$3

Operationally, this quantizes only layer $\ell$4 while keeping all others at 8 bits, and measures the increase in NLL.

ILA-AMP then adds explicit adjacent second-order interaction terms. For each neighboring pair $\ell$5 and each $\ell$6,

$\ell$7

This removes the baseline and the two independent unary effects, leaving a pure interaction term. When $\ell$8, the joint degradation is worse than the sum of the two isolated degradations, indicating error amplification through propagation.

The resulting surrogate is

$\ell$9

which is a chain-structured quadratic surrogate with unary and adjacent pairwise terms. Under a global bit budget,

$b_\ell$0

ILA-AMP solves

$b_\ell$1

The paper explicitly interprets this construction as a second-order or Hessian-like approximation around $b_\ell$2: $b_\ell$3 plays the role of a first-order local sensitivity term, while $b_\ell$4 captures joint sensitivity of NLL to simultaneous bit reductions in adjacent layers. Unlike classical Hessian-based methods, ILA-AMP uses empirical finite-difference approximations in function space rather than Hessians with respect to parameters, thereby incorporating quantization, KOTMS transforms, and non-Gaussian heavy-tailed activation behavior in a single NLL-driven surrogate.

3. Dynamic programming and the adjacent-interaction assumption

Because the surrogate contains only unary terms and adjacent pairwise terms, the optimization problem is a discrete chain factor graph and is solvable exactly by dynamic programming. The state is

$b_\ell$5

defined as the minimal surrogate cost for layers $b_\ell$6 when the cumulative bit sum is $b_\ell$7 and layer $b_\ell$8 uses bit-width $b_\ell$9. The initialization for $\mathcal{B} = \{2,4,6,8\}.$0 is

$\mathcal{B} = \{2,4,6,8\}.$1

and for $\mathcal{B} = \{2,4,6,8\}.$2,

$\mathcal{B} = \{2,4,6,8\}.$3

Backtracking through stored predecessors recovers the optimal bit allocation (Zhao et al., 11 Jun 2026).

The complexity is

$\mathcal{B} = \{2,4,6,8\}.$4

in time and

$\mathcal{B} = \{2,4,6,8\}.$5

in memory, reducible with rolling buffers. With $\mathcal{B} = \{2,4,6,8\}.$6, $\mathcal{B} = \{2,4,6,8\}.$7 typically in the range of Transformer block counts such as 32–80, and $\mathcal{B} = \{2,4,6,8\}.$8 for average-bit constraints such as $\mathcal{B} = \{2,4,6,8\}.$9, the dynamic program is tractable; the dominant overhead is the NLL evaluations needed to construct $\mathbf{b}=(b_1,\ldots,b_L)$0 and $\mathbf{b}=(b_1,\ldots,b_L)$1.

The restriction to adjacent interactions is not only computational. Appendix C argues that under mild stability assumptions on Jacobian products, non-adjacent pairwise interactions decay roughly like $\mathbf{b}=(b_1,\ldots,b_L)$2, so adjacent interactions dominate. The paper also reports that moving from first-order to second-order adjacent modeling substantially improves perplexity, while moving from second-order to third-order triplets costs hundreds of extra minutes of calibration with almost no benefit or even slight degradation. This establishes adjacent second order as the operative compromise between fidelity and search complexity.

4. Dependence on KOTMS and heterogeneous quantization gains

KOTMS applies a shared orthogonal transform to large matrices, such as Q/K/V or MLP weights, by learning

$\mathbf{b}=(b_1,\ldots,b_L)$3

with a Cayley parameterization and a Tri-modal GMM shaping loss on weights. The same transform is applied to activations during inference via $\mathbf{b}=(b_1,\ldots,b_L)$4, preserving function while statistically suppressing activation outliers. In activation space, the transform preserves the $\mathbf{b}=(b_1,\ldots,b_L)$5 norm, distributes energy more evenly across coordinates, and reduces the peak-to-RMS ratio

$\mathbf{b}=(b_1,\ldots,b_L)$6

to $\mathbf{b}=(b_1,\ldots,b_L)$7, which improves uniform quantization (Zhao et al., 11 Jun 2026).

However, KOTMS is optimized on weights, so its activation benefits are indirect and heterogeneous. Appendix F shows cross-layer heterogeneity in post-KOTMS activation distributions; for example, in Qwen3-8B, layer 24 exhibits much tighter quantile bands, whereas layer 4 still has high-quantile spikes. ILA-AMP is designed precisely around this disparity. All unary costs $\mathbf{b}=(b_1,\ldots,b_L)$8 and interaction terms $\mathbf{b}=(b_1,\ldots,b_L)$9 are computed after KOTMS and E2M-ATQ have already been applied, using the actual post-KOTMS activations.

This yields a direct operational interpretation. A layer that benefits strongly from KOTMS will have small $\mathcal{D}$0 and small $\mathcal{D}$1, so the dynamic program can push it to 2–4 bits. A layer that remains weak after rotation will exhibit larger NLL increases under bit reduction, so it receives 6–8 bits. ILA-AMP therefore does not assign bits from a generic statistic such as variance or magnitude alone; it jointly optimizes for the layer-wise disparity of activation quantization gains induced by the shared orthogonal transform and for the propagation of the resulting errors.

A plausible implication is that ILA-AMP should be understood less as a standalone mixed-precision heuristic than as the mechanism that converts KOTMS-induced statistical reshaping into a stable end-to-end W1.58A4 regime.

5. Empirical behavior and ablation evidence

The main ablation on LLaMA2-13B and Qwen3-14B isolates the role of ILA-AMP under low-bit activations. With 4-bit activations and E2M-ATQ plus KOTMS but without ILA-AMP, the degradation is severe: LLaMA2-13B reaches C4 PPL 25.03 and MMLU 27.52, while Qwen3-14B reaches C4 PPL 31.78 and MMLU 47.60. Adding ILA-AMP improves LLaMA2-13B to C4 PPL 10.07 and MMLU 38.17, and Qwen3-14B to C4 PPL 21.00 and MMLU 60.82. The reported gains are 2.5× better C4 PPL and +10.65 MMLU points for LLaMA2-13B, and +13.22 MMLU points for Qwen3-14B, establishing that ILA-AMP is necessary to make low-bit activations work on top of ternary weights in TWLA (Zhao et al., 11 Jun 2026).

The interaction-order ablation further supports the adjacent second-order design. On LLaMA2-7B, LLaMA3-8B, and Qwen3-8B, using WikiText2 perplexity, second-order ILA-AMP significantly improves over first-order allocation based only on $\mathcal{D}$2. Third-order triplets raise calibration time drastically—for example, by +300 minutes on Qwen3-8B—without perplexity benefit and sometimes with slight degradation.

A metric ablation at average A4 compares LIM, Z-Score (ZD), activation norm, NLL first-order only, and ILA-AMP. WikiText2 perplexities are reported as follows: for LLaMA2-7B, LIM 17.49, NLL 11.92, ILA-AMP 8.31; for LLaMA3-8B, LIM 23.21, NLL 15.03, ILA-AMP 12.83; for Qwen3-8B, LIM 33.17, NLL 21.79, ILA-AMP 16.15. These numbers show that explicit inter-layer modeling provides substantial gains beyond NLL-only first-order sensitivity.

The learned bit patterns also match the surrogate’s causal intuition. Under an average A4 budget, most layers remain at 4 bits, a few critical early layers are assigned 6 or 8 bits, and some late layers are reduced to 2 bits. The stated interpretation is that early-layer quantization errors propagate through the whole stack, whereas later layers have shorter propagation paths and can tolerate more aggressive quantization.

Collectively, these experiments support three propositions. First, W1.58A4 PTQ is viable in TWLA only when ILA-AMP is present. Second, adjacent second-order interaction modeling is empirically sufficient and more efficient than higher-order extensions. Third, the resulting mixed-precision pattern is structured by propagation depth rather than by layer-local statistics alone.

6. Implementation details, limitations, and research context

For implementation, TWLA reuses the same small calibration set employed by the preceding components, exemplified by 128 sequences of length 2048 from WikiText2. The paper reports that perplexity varies by less than 1% across calibration seeds and sizes, and that about 100 samples are sufficient. The prescribed stage order is: first apply KOTMS and fold the learned rotations into the model; second apply E2M-ATQ to the rotated weights; third, with the quantized model fixed, enumerate the NLL configurations needed to build $\mathcal{D}$3 and $\mathcal{D}$4, and then solve the dynamic program. For $\mathcal{D}$5, unary costs scale as $\mathcal{D}$6 and adjacent pairwise costs as $\mathcal{D}$7, which amounts to a few hundred forward passes for typical layer counts (Zhao et al., 11 Jun 2026).

The method assumes a layered feed-forward stack such as Transformer blocks so that the chain structure is valid. Its stated failure modes are also tied to that surrogate. If the model is extremely unstable, for example because weight quantization is improper, the surrogate may be less accurate. If the validation NLL is noisy because the evaluation set is too small, the unary and pairwise costs can become noisy. For very deep models or extremely tight bit budgets, third-order or non-adjacent effects might matter more, although the paper reports that adjacent second order is sufficient up to 70B/32B scales and A4.

Within the broader mixed-precision literature, ILA-AMP occupies a distinct position. HAWQ selects layer precision from Hessian information and provides a deterministic mixed-precision schedule, but is primarily weight-focused and uses classical second-order parameter-space sensitivity (Dong et al., 2019). “Augmenting Hessians with Inter-Layer Dependencies for Mixed-Precision Post-Training Quantization” explicitly adds empirical inter-layer dependencies to Hessian-based PTQ and quantizes both weights and activations, but does so through an augmented-Hessian ranking and bisection search rather than the NLL-based adjacent second-order chain surrogate used in ILA-AMP (Schaefer et al., 2023). SNIP addresses adaptive mixed precision during LLM pretraining, jointly choosing activation, weight, and gradient precision via an ILP driven by forward loss divergence and backward weight divergence (Pan et al., 1 Feb 2026). TASA argues that perplexity-only sensitivity can mis-rank layers for reasoning tasks and jointly optimizes calibration-data composition and mixed-precision bit allocation (Wang et al., 1 Jul 2026).

This suggests that ILA-AMP is best viewed as a PTQ-specific activation scheduler whose novelty lies in combining three elements within TWLA: post-KOTMS activation heterogeneity, empirical NLL finite differences rather than parameter Hessians, and exact dynamic programming over a chain-structured second-order surrogate. In that role, it is the mechanism that makes ternary weights and low-bit activations coexist without uniform-precision collapse.