---
title: 'TWLA: Ternary Weight & Low-Bit Activation for LLMs'
url: https://www.emergentmind.com/topics/twla
type: topic
---

# TWLA: Ternary Weight & Low-Bit Activation for LLMs

Searching arXiv for the TWLA paper and closely related PTQ baselines.
Searching arXiv for TWLA (exact title).
TWLA denotes a post-training quantization framework for large language models introduced in "TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization" [2606.13054]. It targets the difficult regime of ternary weights plus low-bit activations, with the headline setting \( \mathrm{W1.58A4} \), meaning about 1.58-bit effective weight storage and 4-bit activations. The framework addresses the claim that prior ternary post-training quantization methods reduce weight storage but often keep activations in high precision, which fundamentally limits end-to-end inference acceleration. TWLA therefore treats weight ternarization, activation outlier suppression, and layer-wise activation bit allocation as a coupled problem rather than as isolated optimizations [2606.13054].

## 1. Problem setting and deployment objective

The motivating premise is that large language models exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. TWLA is aimed at two bottlenecks. The first is weight memory footprint: very large models require substantial storage, whereas ternarization reduces weight storage and enables much cheaper arithmetic. The second is activation memory and compute: prior ternary PTQ methods usually keep activations in FP16 or otherwise high precision, and the paper argues that this prevents true end-to-end speedup [2606.13054].

The framework is organized around two structural difficulties. One is a weight-distribution mismatch. Pretrained LLM weights are described as approximately unimodal or Gaussian-like around zero, whereas ternary quantization prefers three attractors such as \(\{-1,0,+1\}\) or, in asymmetric row-wise form, \(\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}\). Direct hard projection from a unimodal distribution onto a ternary codebook therefore induces substantial error. The second difficulty is activation statistics: activations are described as heavy-tailed and dominated by outliers, so under low-bit quantization a few extreme values can determine the scale and degrade most of the tensor, especially at 4-bit precision [2606.13054].

This framing places TWLA in a specific deployment regime. It is not only a compression method for offline storage. A plausible implication is that its design is centered on inference systems in which memory bandwidth, arithmetic format, and activation movement jointly determine throughput.

## 2. Three-module architecture and quantization pipeline

TWLA consists of three components: Euclidean-to-Manifold Asymmetric Ternary Quantizer (E2M-ATQ), Kronecker Orthogonal Tri-Modal Shaping (KOTMS), and Inter-Layer Aware Activation Mixed Precision (ILA-AMP). Their roles are distinct but interdependent: E2M-ATQ minimizes layer-output error under weight ternarization, KOTMS reshapes weights into ternary-friendly tri-modal distributions while statistically suppressing activation outliers, and ILA-AMP allocates activation precision across layers under a global bit budget while modeling adjacent-layer interaction costs [2606.13054].

| Component | Stated purpose | Core mechanism |
|---|---|---|
| E2M-ATQ | Minimize layer-output error under weight ternarization | Two-stage optimization from Euclidean initialization to manifold relocation |
| KOTMS | Reshape weights and suppress activation outliers | Kronecker-structured orthogonal rotation |
| ILA-AMP | Allocate activation bit-widths under a global budget | Adjacent-layer second-order interaction costs with exact dynamic programming |

The pipeline applies KOTMS first, using a layer-wise orthogonal rotation
\[
\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.
\]
The inverse rotation is then folded into the activation path. E2M-ATQ ternarizes the rotated weights using a Euclidean warm start followed by calibration-aware manifold relocation. ILA-AMP finally assigns activation bits layer by layer, with candidate precisions such as \(2/4/6/8\), to satisfy a total budget while preventing degradation cascades caused by a few weak layers [2606.13054].

The target quantization setting emphasized by the paper is \( \mathrm{W1.58A4} \). The paper also evaluates \( \mathrm{A6} \) and \( \mathrm{A16} \), where \( \mathrm{A16} \) keeps activations in high precision and \( \mathrm{A4} \) is the hardest setting emphasized in the study [2606.13054].

## 3. E2M-ATQ and asymmetric ternary quantization

E2M-ATQ uses an asymmetric ternary parameterization,
\[
\bar{\mathbf W} = \boldsymbol{\mu}\mathbf 1^\top + \mathrm{diag}(\boldsymbol{\alpha})\mathbf T,\qquad \mathbf T\in\{-1,0,1\}^{n\times m},
\]
where \(\boldsymbol{\mu}\) is a row-wise shift, \(\boldsymbol{\alpha}\) is a row-wise scale, and \(\mathbf T\) is the ternary codebook. The row-wise shift is important because pretrained LLM weights are described as often biased and not centered at zero [2606.13054].

The method optimizes two objectives. The first is a Euclidean weight-domain objective,
\[
L_1(\boldsymbol{\mu}, \boldsymbol{\alpha}, \mathbf{T}) =
\left\| \mathbf{W} - \boldsymbol{\mu}\mathbf{1}^\top - \mathrm{diag}(\boldsymbol{\alpha})\,\mathbf{T} \right\|_F^2.
\]
The second is a calibration-aware layer-output objective,
\[
L_2(\boldsymbol{\mu},\boldsymbol{\alpha};\mathbf{T}) =
\left\| \left( \mathbf{W} - \boldsymbol{\mu}\mathbf{1}^\top - \mathrm{diag}(\boldsymbol{\alpha})\,\mathbf{T} \right)\mathbf{X} \right\|_F^2,
\]
where \(\mathbf X\) denotes calibration activations. In equivalent trace form,
\[
L_2(\boldsymbol{\mu},\boldsymbol{\alpha};\mathbf{T}) =
\mathrm{Tr}\!\left(\mathbf{E}\mathbf{S}\mathbf{E}^\top\right),\qquad
\mathbf{E} = \mathbf{W} - \boldsymbol{\mu}\mathbf{1}^\top - \mathrm{diag}(\boldsymbol{\alpha})\,\mathbf{T},
\]
with
\[
\mathbf{S}=\sum_b \mathbf{X}_b^\top \mathbf{X}_b
\]
as the activation second moment [2606.13054].

The optimization has two stages. Stage I is Euclidean initialization, whose goal is to obtain a stable ternary pattern \(\mathbf T^{(0)}\) in Frobenius geometry. The paper alternates updates of \(\boldsymbol{\mu}\) by residual mean correction, \(\boldsymbol{\alpha}\) by least squares, and \(\mathbf T\) by ternary thresholding. The thresholding rule is
\[
T_{ij}= \begin{cases}
+1, & W_{ij}>\Delta_i,\\
0,  & |W_{ij}|\le \Delta_i,\\
-1, & W_{ij}<-\Delta_i,
\end{cases}
\qquad
\Delta_i \approx \frac{0.75}{m}\sum_{j=1}^{m}|W_{ij}|.
\]
The row-wise scale update is
\[
\alpha_i =
\frac{\sum_{j=1}^{m}T_{ij}W_{ij}}{\sum_{j=1}^{m}|T_{ij}|}.
\]
The paper states that this coordinate-descent-style update is monotonically non-increasing [2606.13054].

Stage II is manifold relocation. Once \(\mathbf T\) is frozen as \(\mathbf T^{(0)}\), the method optimizes \(\boldsymbol{\mu}\) and \(\boldsymbol{\alpha}\) under the calibration metric induced by \(\mathbf S\). For each row \(i\), \((\alpha_i,\mu_i)\) solve a \(2\times2\) linear system, and the paper gives closed-form expressions for \(\alpha_i^\ast\) and \(\mu_i^\ast\). The important conceptual point is that the discrete ternary structure is held fixed while the continuous parameters are relocated in the geometry induced by observed activations [2606.13054].

## 4. KOTMS and tri-modal shaping by orthogonal rotation

KOTMS addresses the mismatch between pretrained weight distributions and ternary codebooks. Its central mechanism is a Kronecker-structured orthogonal transform
\[
\mathbf R = \mathbf R_1 \otimes \mathbf R_2,\qquad
\mathbf R_1\in\mathcal O(n_1),\ \mathbf R_2\in\mathcal O(n_2),\ n_1n_2=m,
\]
with inverse
\[
\mathbf R^{-1}=\mathbf R^\top=\mathbf R_1^\top\otimes \mathbf R_2^\top.
\]
A dense orthogonal matrix is described as too expensive for LLMs, and the Kronecker restriction is therefore part of the efficiency design [2606.13054].

The shaping objective uses a smooth tri-modal Gaussian mixture surrogate over transformed weights \( \mathbf z_i = \mathbf w_i \mathbf R \), with per-row center
\[
c_i=\frac{1}{m}\sum_{j=1}^{m}|z_{ij}|
\]
and spread
\[
\sigma_i=\mathrm{std}(\{z_{ij}\}_{j=1}^m).
\]
The loss encourages the transformed entries to lie near three attractors \(\{-c_i,0,+c_i\}\), and a regularizer controls the mass of the zero mode to match a target sparsity ratio [2606.13054].

The same orthogonal transform is also used to suppress activation outliers, because its inverse is folded into the activation path. Orthogonal transforms preserve \(\ell_2\) energy,
\[
\|\mathbf{x}\mathbf R\|_2 = \|\mathbf{x}\|_2,
\]
but can reduce peak coordinate magnitude. The paper gives the high-probability bound
\[
\|\mathbf{x}\mathbf R\|_\infty
\le
\|\mathbf{x}\|_2\sqrt{\frac{2\log(2m/\delta)}{m-2}},
\]
and interprets this as reducing the peak-to-RMS ratio, thereby shrinking the dynamic range that low-bit activation quantizers must cover [2606.13054].

KOTMS is therefore not only a weight-side preprocessing step. A plausible implication is that it functions as a shared statistical conditioning transform across weights and activations. The paper also emphasizes, however, that activation gains from KOTMS are heterogeneous across layers, which motivates a non-uniform activation-precision policy rather than a single global bit-width [2606.13054].

## 5. ILA-AMP and adjacent-layer interaction modeling

ILA-AMP is the activation-allocation component of TWLA. It assigns bit-widths \(b_\ell\in\{2,4,6,8\}\) across layers under a total budget, but differs from layer-wise mixed-precision heuristics by explicitly modeling adjacent-layer second-order interaction costs. The rationale is that quantizing one layer changes its output distribution, which becomes the next layer’s input, so the error is not purely additive layer by layer [2606.13054].

The optimization problem is
\[
\min_{\{b_\ell\}}
\sum_{\ell=1}^L C_\ell(b_\ell) + \sum_{\ell=2}^L K_{\ell-1,\ell}(b_{\ell-1}, b_\ell),
\qquad
\text{s.t. } \sum_{\ell=1}^L b_\ell \le B.
\]
Here, \(C_\ell(b)\) is the unary cost of quantizing only layer \(\ell\) to \(b\), and \(K_{\ell-1,\ell}(b', b)\) is the adjacent pairwise interaction cost. The paper defines both quantities through validation negative log-likelihood differences relative to the high-precision activation baseline, so the pairwise term measures degradation beyond what unary layer costs would predict [2606.13054].

This construction serves two purposes. First, it captures the layer-wise disparity in activation quantization gains induced by KOTMS. Second, it aims to prevent cascades triggered by a few weak layers. The paper states that second-order adjacent interaction gives the best cost-benefit tradeoff, whereas third-order interactions add substantial calibration cost but little or no gain. Because the objective is chain-structured, the paper uses dynamic programming to solve the assignment exactly [2606.13054].

The broader significance of ILA-AMP is methodological. TWLA treats activation quantization as an inter-layer dependency problem rather than a set of independent local decisions. This suggests a view of post-training quantization in which calibration statistics must be propagated structurally through the network rather than measured only per layer.

## 6. Empirical evaluation, ablations, and scope

The evaluation covers LLaMA-2 \(7\mathrm{B}, 13\mathrm{B}, 70\mathrm{B}\), LLaMA-3 \(8\mathrm{B}\), Qwen3 \(8\mathrm{B}, 14\mathrm{B}, 32\mathrm{B}\), and Qwen3-32B-Instruct. Tasks include WikiText2 perplexity, C4 perplexity, zero-shot evaluation on ARC-Challenge, ARC-Easy, HellaSwag, LAMBADA-openai, LAMBADA-standard, PIQA, and WinoGrande, and reasoning or coding evaluation on MMLU, GSM8K, and HumanEval [2606.13054].

Under the headline \( \mathrm{W1.58A4} \) setting, the paper reports the following average zero-shot accuracies.

| Model | Avg zero-shot accuracy |
|---|---:|
| LLaMA2-7B | 58.00 |
| LLaMA2-13B | 64.30 |
| LLaMA2-70B | 71.10 |
| LLaMA3-8B | 55.23 |
| Qwen3-8B | 50.42 |
| Qwen3-14B | 62.00 |
| Qwen3-32B | 65.25 |

The comparative results described in the paper emphasize that TWLA is consistently far above prior methods at \( \mathrm{A4} \). Examples given include LLaMA2-70B at \( \mathrm{A4} \), where QuaRot obtains 53.64, ResQ 56.43, and TWLA 71.10; Qwen3-32B at \( \mathrm{A4} \), where PB-LLM obtains 26.36, GPTQ 24.85, and TWLA 65.25; and LLaMA3-8B at \( \mathrm{A4} \), where PT\(^2\)-LLM obtains 25.81 and TWLA 55.23. The paper states that TWLA is generally the best or near-best across all tested models and bit settings, and especially strong at \( \mathrm{A4} \), where most baselines collapse [2606.13054].

Efficiency claims are also reported.

| Metric | Reported value |
|---|---|
| Weight storage savings | about 80%+ |
| LLaMA2-13B throughput | 86.35 tokens/s |
| FP16 throughput | 23.70 tokens/s |
| QuaRot throughput | 66.42 tokens/s |
| Speedup over FP16 | 3.64× |
| LLaMA2-7B quantization time | about 82 minutes |
| SliM-LLM quantization time | 158 minutes |

The ablation study supports the division of labor among the three modules. E2M-ATQ alone improves over the no-module baseline; KOTMS alone also helps, especially in reshaping distributions; E2M-ATQ plus KOTMS is better than either alone; and adding ILA-AMP is described as crucial under low-bit activations, especially \( \mathrm{A4} \). Additional analyses report that WikiText2 and C4 are comparable as calibration corpora, PTB is worse, and small calibration subsets are sufficient, with results stable across random seeds and sample sizes [2606.13054].

Several limitations are explicit or implied. Activation-quantizability gain from KOTMS is heterogeneous across layers. Dense or higher-order interaction modeling becomes too expensive. Calibration with ill-conditioned second moments may need stabilization. A uniform activation bit-width is described as suboptimal, which is why ILA-AMP is required [2606.13054]. Within the associated arXiv materials, TWLA refers specifically to this quantization framework; nearby references to Telecom World Models and Target Wake Time are separate concepts, and those papers explicitly state that TWLA is not their defined acronym [2604.06882] [1804.07717].

Source: https://www.emergentmind.com/topics/twla