TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization

Abstract: LLMs exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a promising compression technique, offering significant reductions in model size and inference complexity. However, existing methods struggle with heavy-tailed activation distributions and therefore keep activations in high precision, fundamentally limiting end-to-end inference acceleration. To overcome this limitation, we propose TWLA, a post-training quantization (PTQ) framework that achieves 1.58-bit weight compression and 4-bit activation quantization while maintaining high accuracy. TWLA comprises three components: (1) Euclidean-to-Manifold Asymmetric Ternary Quantizer (E2M-ATQ) minimizes layer-output error under weight ternarization via a two-stage optimization from Euclidean initialization to manifold relocation; (2) Kronecker Orthogonal Tri-Modal Shaping (KOTMS) applies a Kronecker-structured orthogonal rotation to reshape weights into ternary-friendly tri-modal distributions, while the shared rotation statistically suppresses activation outliers; and (3) Inter-Layer Aware Activation Mixed Precision (ILA-AMP) explicitly introduces adjacent-layer second-order interaction costs in bit allocation and jointly optimizes for the layer-wise disparity of activation quantization gains induced by the shared orthogonal transform, preventing cascades triggered by a few weak layers. Extensive experiments demonstrate that TWLA maintains high accuracy under W1.58A4, while delivering significant inference acceleration. The code is available at https://github.com/Kishon-zzx/TWLA.

• The paper introduces TWLA, a retraining-free PTQ framework that achieves ternary weights (1.58-bit) and low-bit (4-bit) activations for LLMs.
• It integrates E2M-ATQ, KOTMS, and ILA-AMP to stabilize quantization, reshape weight distributions, and adapt layer-wise bit allocation.
• TWLA delivers up to 90% FP16 accuracy and over 80% memory reduction, outperforming existing methods on LLaMA and Qwen models.

TWLA: Post-Training Quantization with Ternary Weights and Low-Bit Activations for LLMs

Motivation and Problem Statement

The exponential scaling of LLMs has substantially increased inference memory and compute requirements, impeding deployment on resource-constrained platforms. Traditional quantization solutions, especially ternarization ($\{-1, 0, +1\}$ weights), can aggressively reduce model size and computation, but existing PTQ approaches fail to address activation outliers, requiring full-precision activations and limiting practical speedup. The paper "TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization" (2606.13054) introduces TWLA, a retraining-free PTQ framework that achieves 1.58-bit weights and 4-bit activations, systematically targeting two core quantization bottlenecks: (1) weight-codebook misalignment and (2) heavy-tailed activation distributions.

TWLA Methodology and Components

TWLA unifies three interdependent strategies, enabling efficient end-to-end LLM inference:

• E2M-ATQ: The Euclidean-to-Manifold Asymmetric Ternary Quantizer initializes layer-wise ternary patterns via Frobenius reconstruction, followed by metric-aware shift/scale relocation using calibration statistics. This dual-stage procedure stabilizes ternarization and minimizes forward output error induced by discrete stratification.
• KOTMS: Kronecker Orthogonal Tri-Modal Shaping leverages a lightweight, orthogonal Kronecker rotation to reshape weight distributions toward a ternary-aligned tri-modal form. The shared rotation is strictly invertible and yields substantial activation outlier suppression via statistical mixing, dramatically reducing scale inflation in downstream activations.
• ILA-AMP: Inter-Layer Aware Activation Mixed Precision exploits second-order adjacent-layer interaction modeling, allocating activation bitwidths under a chain-structured quadratic surrogate objective. This approach precludes sensitivity cascades arising from poorly quantized bottleneck layers.

  Figure 1: TWLA workflow: E2M-ATQ calibrates ternary weights, KOTMS reshapes distributions and suppresses activation outliers, and ILA-AMP allocates activation bits via dynamic programming.

  

  Figure 2: KOTMS transforms unimodal weights and heavy-tailed activations into tri-modal, quantization-friendly distributions, reducing outlier impact.

Quantization Robustness and Numerical Results

TWLA's numeric outcomes demonstrate consistent superiority over SOTA ternarization and sub-2-bit baselines in both accuracy and perplexity across LLaMA and Qwen model families. Robustness persists at 4-bit activations—where competing methods collapse—delivering up to 90% of FP16 performance even on instruction-tuned LLMs, accompanied by dramatically reduced storage and compute cost.

Figure 3: LLaMA-family performance: TWLA retains accuracy under weight-only and weight-activation quantization, while competitors degrade at 4-bit precision.

TWLA's ternarization achieves average weight precision of 1.58 bits, maintaining high zero-shot accuracy on multi-task benchmarks and minimizing perplexity explosion with aggressive activation quantization. On LLaMA2-70B, accuracy increases from 53.64 (QuaRot) to 71.10 for TWLA under A4, with memory reduction exceeding 80%.

Distributional Analysis and Activation Quantizability

KOTMS's effect is quantified in multiple visualization studies. Pre-KOTMS, weights and activations exhibit unimodal and heavy-tailed profiles, respectively. Post-KOTMS, weights are tri-modal and activations are tightly bounded, improving quantization sensitivity and enabling stable deployment.

Figure 4: Qwen3-8B Layer 12 weights: transformation from Gaussian-like to tri-modal via KOTMS.

Figure 5: Qwen3-8B Layer 12 activations: reduction of outlier dominance after KOTMS.

Cross-layer analysis reveals heterogeneity: some layers benefit substantially more than others from orthogonal mixing, validating ILA-AMP’s adaptive layerwise bit allocation.

Figure 6: Qwen3-8B cross-layer activation quantile plots: KOTMS induces substantial outlier suppression; layer-wise effects are non-uniform.

Figure 7: ILA-AMP allocates per-layer activation bits under a 4-bit budget, adapting to quantization difficulty.

Efficiency, Calibration, and Ablation

TWLA provides efficient calibration: quantization time is competitive versus SliM-LLM and QuaRot, yet achieves significantly lower perplexity. Kronecker-structured orthogonal transforms are leveraged for function-preserving weight and activation shaping with minimal overhead relative to dense rotations. Modular ablation indicates strong synergy between the three modules (E2M-ATQ, KOTMS, ILA-AMP), with accuracy loss if any are omitted.

Figure 8: (a) Overhead–performance trade-off for KOTMS; (b) throughput and memory comparisons; (c) quantization time and perplexity across PTQ methods.

Figure 9: Calibration sample impact on perplexity: TWLA performance is robust to sample count and seed.

Figure 10: Calibration cost/quantization quality trade-off under increasing interaction order; 2nd-order modeling achieves best cost-effectiveness.

Practical and Theoretical Implications

TWLA enables direct deployment of LLMs on low-memory platforms, maximizing energy and latency gains. Theoretically, TWLA expands the PTQ quantization boundary by tightly integrating geometric distribution shaping, metric-aware ternarization, and layer-adaptive mixed-precision quantization. The approach generalizes to broader transformer quantization contexts, and further research will likely explore structured orthogonality, interaction modeling, and unified PTQ calibration for higher-order ternary/binarized settings.

Conclusion

TWLA advances PTQ by executing joint ternary weight and low-bit activation quantization with high accuracy and strong efficiency. Strategic distribution shaping and layer-aware allocation address quantization bottlenecks, making extreme compression regimes viable for LLMs. TWLA thus establishes a scalable foundation for cost-effective, competitive LLM deployment on commodity and mobile hardware.