TWLA: Ternary Weight & Low-Bit Activation for LLMs

Updated 6 July 2026

TWLA is a post-training quantization framework for LLMs that employs ternary weights and low-bit activations to reduce memory footprint and boost inference speed.
It integrates three key modules—E2M-ATQ, KOTMS, and ILA-AMP—to jointly optimize weight ternarization, activation outlier suppression, and layer-wise precision allocation.
Empirical evaluations demonstrate up to 80% weight storage savings and significant speedups over FP16 across models like LLaMA2 and Qwen3 under challenging low-bit settings.

Searching arXiv for the TWLA paper and closely related PTQ baselines. Searching arXiv for TWLA (exact title). TWLA denotes a post-training quantization framework for LLMs introduced in "TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization" (Zhao et al., 11 Jun 2026). It targets the difficult regime of ternary weights plus low-bit activations, with the headline setting $\mathrm{W1.58A4}$ , meaning about 1.58-bit effective weight storage and 4-bit activations. The framework addresses the claim that prior ternary post-training quantization methods reduce weight storage but often keep activations in high precision, which fundamentally limits end-to-end inference acceleration. TWLA therefore treats weight ternarization, activation outlier suppression, and layer-wise activation bit allocation as a coupled problem rather than as isolated optimizations (Zhao et al., 11 Jun 2026).

1. Problem setting and deployment objective

The motivating premise is that LLMs exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. TWLA is aimed at two bottlenecks. The first is weight memory footprint: very large models require substantial storage, whereas ternarization reduces weight storage and enables much cheaper arithmetic. The second is activation memory and compute: prior ternary PTQ methods usually keep activations in FP16 or otherwise high precision, and the paper argues that this prevents true end-to-end speedup (Zhao et al., 11 Jun 2026).

The framework is organized around two structural difficulties. One is a weight-distribution mismatch. Pretrained LLM weights are described as approximately unimodal or Gaussian-like around zero, whereas ternary quantization prefers three attractors such as $\{-1,0,+1\}$ or, in asymmetric row-wise form, $\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}$ . Direct hard projection from a unimodal distribution onto a ternary codebook therefore induces substantial error. The second difficulty is activation statistics: activations are described as heavy-tailed and dominated by outliers, so under low-bit quantization a few extreme values can determine the scale and degrade most of the tensor, especially at 4-bit precision (Zhao et al., 11 Jun 2026).

This framing places TWLA in a specific deployment regime. It is not only a compression method for offline storage. A plausible implication is that its design is centered on inference systems in which memory bandwidth, arithmetic format, and activation movement jointly determine throughput.

2. Three-module architecture and quantization pipeline

TWLA consists of three components: Euclidean-to-Manifold Asymmetric Ternary Quantizer (E2M-ATQ), Kronecker Orthogonal Tri-Modal Shaping (KOTMS), and Inter-Layer Aware Activation Mixed Precision (ILA-AMP). Their roles are distinct but interdependent: E2M-ATQ minimizes layer-output error under weight ternarization, KOTMS reshapes weights into ternary-friendly tri-modal distributions while statistically suppressing activation outliers, and ILA-AMP allocates activation precision across layers under a global bit budget while modeling adjacent-layer interaction costs (Zhao et al., 11 Jun 2026).

Component	Stated purpose	Core mechanism
E2M-ATQ	Minimize layer-output error under weight ternarization	Two-stage optimization from Euclidean initialization to manifold relocation
KOTMS	Reshape weights and suppress activation outliers	Kronecker-structured orthogonal rotation
ILA-AMP	Allocate activation bit-widths under a global budget	Adjacent-layer second-order interaction costs with exact dynamic programming

The pipeline applies KOTMS first, using a layer-wise orthogonal rotation

$\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.$

The inverse rotation is then folded into the activation path. E2M-ATQ ternarizes the rotated weights using a Euclidean warm start followed by calibration-aware manifold relocation. ILA-AMP finally assigns activation bits layer by layer, with candidate precisions such as $2/4/6/8$, to satisfy a total budget while preventing degradation cascades caused by a few weak layers (Zhao et al., 11 Jun 2026).

The target quantization setting emphasized by the paper is $\mathrm{W1.58A4}$ . The paper also evaluates $\mathrm{A6}$ and $\mathrm{A16}$ , where $\mathrm{A16}$ keeps activations in high precision and $\mathrm{A4}$ is the hardest setting emphasized in the study (Zhao et al., 11 Jun 2026).

3. E2M-ATQ and asymmetric ternary quantization

E2M-ATQ uses an asymmetric ternary parameterization,

$\{-1,0,+1\}$ 0

where $\{-1,0,+1\}$ 1 is a row-wise shift, $\{-1,0,+1\}$ 2 is a row-wise scale, and $\{-1,0,+1\}$ 3 is the ternary codebook. The row-wise shift is important because pretrained LLM weights are described as often biased and not centered at zero (Zhao et al., 11 Jun 2026).

The method optimizes two objectives. The first is a Euclidean weight-domain objective,

$\{-1,0,+1\}$ 4

The second is a calibration-aware layer-output objective,

$\{-1,0,+1\}$ 5

where $\{-1,0,+1\}$ 6 denotes calibration activations. In equivalent trace form,

$\{-1,0,+1\}$ 7

with

$\{-1,0,+1\}$ 8

as the activation second moment (Zhao et al., 11 Jun 2026).

The optimization has two stages. Stage I is Euclidean initialization, whose goal is to obtain a stable ternary pattern $\{-1,0,+1\}$ 9 in Frobenius geometry. The paper alternates updates of $\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}$ 0 by residual mean correction, $\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}$ 1 by least squares, and $\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}$ 2 by ternary thresholding. The thresholding rule is

$\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}$ 3

The row-wise scale update is

$\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}$ 4

The paper states that this coordinate-descent-style update is monotonically non-increasing (Zhao et al., 11 Jun 2026).

Stage II is manifold relocation. Once $\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}$ 5 is frozen as $\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}$ 6, the method optimizes $\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}$ 7 and $\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}$ 8 under the calibration metric induced by $\{-\alpha_i+\mu_i,\ \mu_i,\ +\alpha_i+\mu_i\}$ 9. For each row $\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.$ 0, $\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.$ 1 solve a $\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.$ 2 linear system, and the paper gives closed-form expressions for $\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.$ 3 and $\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.$ 4. The important conceptual point is that the discrete ternary structure is held fixed while the continuous parameters are relocated in the geometry induced by observed activations (Zhao et al., 11 Jun 2026).

KOTMS addresses the mismatch between pretrained weight distributions and ternary codebooks. Its central mechanism is a Kronecker-structured orthogonal transform

$\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.$ 5

with inverse

$\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.$ 6

A dense orthogonal matrix is described as too expensive for LLMs, and the Kronecker restriction is therefore part of the efficiency design (Zhao et al., 11 Jun 2026).

The shaping objective uses a smooth tri-modal Gaussian mixture surrogate over transformed weights $\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.$ 7, with per-row center

$\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.$ 8

and spread

$\mathbf W_\ell \leftarrow \mathbf W_\ell \mathbf R_\ell,\quad \mathbf R_\ell=\mathbf R_{1,\ell}\otimes \mathbf R_{2,\ell}.$ 9

The loss encourages the transformed entries to lie near three attractors $2/4/6/8$0, and a regularizer controls the mass of the zero mode to match a target sparsity ratio (Zhao et al., 11 Jun 2026).

The same orthogonal transform is also used to suppress activation outliers, because its inverse is folded into the activation path. Orthogonal transforms preserve $2/4/6/8$1 energy,

$2/4/6/8$2

but can reduce peak coordinate magnitude. The paper gives the high-probability bound

$2/4/6/8$3

and interprets this as reducing the peak-to-RMS ratio, thereby shrinking the dynamic range that low-bit activation quantizers must cover (Zhao et al., 11 Jun 2026).

KOTMS is therefore not only a weight-side preprocessing step. A plausible implication is that it functions as a shared statistical conditioning transform across weights and activations. The paper also emphasizes, however, that activation gains from KOTMS are heterogeneous across layers, which motivates a non-uniform activation-precision policy rather than a single global bit-width (Zhao et al., 11 Jun 2026).

5. ILA-AMP and adjacent-layer interaction modeling

ILA-AMP is the activation-allocation component of TWLA. It assigns bit-widths $2/4/6/8$4 across layers under a total budget, but differs from layer-wise mixed-precision heuristics by explicitly modeling adjacent-layer second-order interaction costs. The rationale is that quantizing one layer changes its output distribution, which becomes the next layer’s input, so the error is not purely additive layer by layer (Zhao et al., 11 Jun 2026).

The optimization problem is

$2/4/6/8$5

Here, $2/4/6/8$6 is the unary cost of quantizing only layer $2/4/6/8$7 to $2/4/6/8$8, and $2/4/6/8$9 is the adjacent pairwise interaction cost. The paper defines both quantities through validation negative log-likelihood differences relative to the high-precision activation baseline, so the pairwise term measures degradation beyond what unary layer costs would predict (Zhao et al., 11 Jun 2026).

This construction serves two purposes. First, it captures the layer-wise disparity in activation quantization gains induced by KOTMS. Second, it aims to prevent cascades triggered by a few weak layers. The paper states that second-order adjacent interaction gives the best cost-benefit tradeoff, whereas third-order interactions add substantial calibration cost but little or no gain. Because the objective is chain-structured, the paper uses dynamic programming to solve the assignment exactly (Zhao et al., 11 Jun 2026).

The broader significance of ILA-AMP is methodological. TWLA treats activation quantization as an inter-layer dependency problem rather than a set of independent local decisions. This suggests a view of post-training quantization in which calibration statistics must be propagated structurally through the network rather than measured only per layer.

6. Empirical evaluation, ablations, and scope

The evaluation covers LLaMA-2 $\mathrm{W1.58A4}$ 0, LLaMA-3 $\mathrm{W1.58A4}$ 1, Qwen3 $\mathrm{W1.58A4}$ 2, and Qwen3-32B-Instruct. Tasks include WikiText2 perplexity, C4 perplexity, zero-shot evaluation on ARC-Challenge, ARC-Easy, HellaSwag, LAMBADA-openai, LAMBADA-standard, PIQA, and WinoGrande, and reasoning or coding evaluation on MMLU, GSM8K, and HumanEval (Zhao et al., 11 Jun 2026).

Under the headline $\mathrm{W1.58A4}$ 3 setting, the paper reports the following average zero-shot accuracies.

Model	Avg zero-shot accuracy
LLaMA2-7B	58.00
LLaMA2-13B	64.30
LLaMA2-70B	71.10
LLaMA3-8B	55.23
Qwen3-8B	50.42
Qwen3-14B	62.00
Qwen3-32B	65.25

The comparative results described in the paper emphasize that TWLA is consistently far above prior methods at $\mathrm{W1.58A4}$ 4. Examples given include LLaMA2-70B at $\mathrm{W1.58A4}$ 5, where QuaRot obtains 53.64, ResQ 56.43, and TWLA 71.10; Qwen3-32B at $\mathrm{W1.58A4}$ 6, where PB-LLM obtains 26.36, GPTQ 24.85, and TWLA 65.25; and LLaMA3-8B at $\mathrm{W1.58A4}$ 7, where PT $\mathrm{W1.58A4}$ 8-LLM obtains 25.81 and TWLA 55.23. The paper states that TWLA is generally the best or near-best across all tested models and bit settings, and especially strong at $\mathrm{W1.58A4}$ 9, where most baselines collapse (Zhao et al., 11 Jun 2026).

Efficiency claims are also reported.

Metric	Reported value
Weight storage savings	about 80%+
LLaMA2-13B throughput	86.35 tokens/s
FP16 throughput	23.70 tokens/s
QuaRot throughput	66.42 tokens/s
Speedup over FP16	3.64×
LLaMA2-7B quantization time	about 82 minutes
SliM-LLM quantization time	158 minutes

The ablation study supports the division of labor among the three modules. E2M-ATQ alone improves over the no-module baseline; KOTMS alone also helps, especially in reshaping distributions; E2M-ATQ plus KOTMS is better than either alone; and adding ILA-AMP is described as crucial under low-bit activations, especially $\mathrm{A6}$ 0. Additional analyses report that WikiText2 and C4 are comparable as calibration corpora, PTB is worse, and small calibration subsets are sufficient, with results stable across random seeds and sample sizes (Zhao et al., 11 Jun 2026).

Several limitations are explicit or implied. Activation-quantizability gain from KOTMS is heterogeneous across layers. Dense or higher-order interaction modeling becomes too expensive. Calibration with ill-conditioned second moments may need stabilization. A uniform activation bit-width is described as suboptimal, which is why ILA-AMP is required (Zhao et al., 11 Jun 2026). Within the associated arXiv materials, TWLA refers specifically to this quantization framework; nearby references to Telecom World Models and Target Wake Time are separate concepts, and those papers explicitly state that TWLA is not their defined acronym (Zou et al., 8 Apr 2026, Nurchis et al., 2018).