KOTMS: Kronecker Orthogonal Tri-Modal Shaping

• KOTMS is a geometric module in TWLA that learns an orthogonal rotation to reshape pretrained weight distributions into a structured tri-modal form aligned with ternary quantization anchors.
• It employs a Kronecker-structured rotation that reduces storage and compute overhead while statistically suppressing activation outliers through inverse rotation equivalence.
• Empirical ablations show that KOTMS, when combined with E2M-ATQ and ILA-AMP, significantly improves low-bit quantization performance of large language models.

to=arxiv_search 大发游戏 ＿俺去也کرد് code ＿天天_json {"query":"(Zhao et al., 11 Jun 2026) TWLA Kronecker Orthogonal Tri-Modal Shaping (Wu et al., 10 Jun 2025) KrossFuse", "max_results": 5, "sort_by": "submittedDate"} to=arxiv_search code 北京赛车前json {"query":"(Zhao et al., 11 Jun 2026) TWLA Kronecker Orthogonal Tri-Modal Shaping (Wu et al., 10 Jun 2025) KrossFuse","max_results":5,"sort_by":"submittedDate"} to=search_arxiv 大发快三的 code 天天彩票与你同行_json {"query":"(Zhao et al., 11 Jun 2026)", "max_results": 3} to=search_arxiv code പ്പോൾjson {"query":"(Zhao et al., 11 Jun 2026)","max_results":3} Kronecker Orthogonal Tri-Modal Shaping (KOTMS) is the rotation-and-distribution-shaping component of TWLA, a post-training quantization framework for LLMs that targets ternary weights, denoted $\mathrm{W1.58}$, together with low-bit activations, especially $\mathrm{A4}$. KOTMS does not itself perform ternarization. Instead, it learns a structured orthogonal coordinate transform under which pretrained weight matrices become more compatible with a ternary codebook, while the same transform, by orthogonal equivalence, statistically suppresses activation outliers. In this sense, KOTMS is a geometric preprocessing mechanism that links weight-side codebook mismatch to activation-side dynamic-range stabilization (Zhao et al., 11 Jun 2026).

1. Position within TWLA and the meaning of “tri-modal”

Within TWLA, KOTMS is one of three modules. The framework comprises E2M-ATQ, KOTMS, and ILA-AMP, and their roles are explicitly differentiated rather than merged into a single optimization stage (Zhao et al., 11 Jun 2026).

Module	Role	Main object
E2M-ATQ	Optimizes asymmetric ternary quantization parameters for weights	$(\mu,\alpha,T)$
KOTMS	Learns a structured orthogonal rotation that reshapes weights and indirectly smooths activations	$\mathbf{R}=\mathbf{R}_1\otimes\mathbf{R}_2$
ILA-AMP	Assigns mixed activation precision across layers	Layerwise activation bits

The motivation for KOTMS is a paired empirical observation about pretrained LLMs. Weights are often approximately unimodal Gaussian-like per channel and concentrated near zero, whereas activations are often heavy-tailed with extreme outliers. This combination is problematic under the aggressive $\mathrm{W1.58A4}$ target. E2M-ATQ can improve continuous ternary parameters for a fixed ternary pattern, but, as the paper states, “such parameter relocation does not change the geometric mismatch between pretrained weights and the ternary codebook” (Zhao et al., 11 Jun 2026).

The phrase “tri-modal shaping” refers to the intended organization of transformed weight values around three attraction regions, namely

$\{-c_i,0,+c_i\}.$

Accordingly, “tri-modal” does not denote three data modalities. It denotes a three-mode distributional structure aligned with ternary quantization. The purpose is to make the final hard projection less sensitive to perturbations: values clustered near negative, zero, and positive anchors are better matched to a ternary codebook than values drawn from a near-unimodal bell shape.

This also clarifies KOTMS’s scope. It is neither a standalone quantizer nor a generic rotation method. Its role is to change the coordinate system in which ternarization is carried out so that the ternary projection becomes lower-error and the same orthogonal mixing statistically suppresses activation outliers.

2. Formal definition and shaping objective

For a weight matrix $\mathbf{W}\in\mathbb{R}^{n\times m}$, KOTMS rotates the $i$-th row $\mathbf{w}_i$ by a learnable orthogonal transform $\mathbf{R}\in\mathbb{R}^{m\times m}$: $\mathrm{A4}$0 Here $\mathrm{A4}$1 is the number of output rows or channels, $\mathrm{A4}$2 is the rotated hidden dimension, and $\mathrm{A4}$3 is the $\mathrm{A4}$4-th entry in the rotated row (Zhao et al., 11 Jun 2026).

The central shaping loss is a symmetric three-component Gaussian mixture. For each row $\mathrm{A4}$5, the anchors are $\mathrm{A4}$6, and the paper defines

$\mathrm{A4}$7

with

$\mathrm{A4}$8

and

$\mathrm{A4}$9

This objective makes entries near $(\mu,\alpha,T)$0, $(\mu,\alpha,T)$1, and $(\mu,\alpha,T)$2 high-likelihood under a row-wise Gaussian mixture. In the appendix, the paper gives a small-variance interpretation: $(\mu,\alpha,T)$3 That derivation formalizes the shaping loss as a differentiable soft projection onto a ternary-compatible set.

KOTMS also includes explicit zero-mode control. Defining posterior responsibilities

$(\mu,\alpha,T)$4

with

$(\mu,\alpha,T)$5

the average zero responsibility for row $(\mu,\alpha,T)$6 is

$(\mu,\alpha,T)$7

The regularizer is then

$(\mu,\alpha,T)$8

and the full shaping objective is

$(\mu,\alpha,T)$9

where $\mathbf{R}=\mathbf{R}_1\otimes\mathbf{R}_2$0 is the target zero-mode mass and $\mathbf{R}=\mathbf{R}_1\otimes\mathbf{R}_2$1 is the regularization strength.

This formulation shows that KOTMS is not merely encouraging “three bumps.” It is explicitly regulating how much mass should occupy the center mode so that the shaped distribution remains compatible with ternary sparsity.

3. Kronecker-structured orthogonality

A full dense orthogonal transform at LLM hidden sizes would be impractical. KOTMS therefore restricts the rotation to a Kronecker form: $\mathbf{R}=\mathbf{R}_1\otimes\mathbf{R}_2$2 Since both factors are orthogonal,

$\mathbf{R}=\mathbf{R}_1\otimes\mathbf{R}_2$3

This exact invertibility is what permits the transform to be folded into adjacent layers while preserving the full-precision mapping (Zhao et al., 11 Jun 2026).

The practical application identity is also explicit. If a row vector $\mathbf{R}=\mathbf{R}_1\otimes\mathbf{R}_2$4 is reshaped as

$\mathbf{R}=\mathbf{R}_1\otimes\mathbf{R}_2$5

then

$\mathbf{R}=\mathbf{R}_1\otimes\mathbf{R}_2$6

Hence the method applies the large rotation through two smaller matrix multiplies rather than materializing an $\mathbf{R}=\mathbf{R}_1\otimes\mathbf{R}_2$7 dense matrix.

Orthogonality is preserved during optimization through Cayley parameterization. For each factor,

$\mathbf{R}=\mathbf{R}_1\otimes\mathbf{R}_2$8

and

$\mathbf{R}=\mathbf{R}_1\otimes\mathbf{R}_2$9

This guarantees

$\mathrm{W1.58A4}$0

throughout optimization, avoiding explicit projection steps.

The computational rationale for the Kronecker structure is quantified. Dense storage is

$\mathrm{W1.58A4}$1

whereas Kronecker storage is

$\mathrm{W1.58A4}$2

When $\mathrm{W1.58A4}$3, this is approximately $\mathrm{W1.58A4}$4, nearly linear rather than quadratic. The appendix gives the example $\mathrm{W1.58A4}$5, $\mathrm{W1.58A4}$6: $\mathrm{W1.58A4}$7 versus

$\mathrm{W1.58A4}$8

a storage reduction factor of

$\mathrm{W1.58A4}$9

The same scaling appears in compute. Dense application costs

$\{-c_i,0,+c_i\}.$0

while Kronecker application costs

$\{-c_i,0,+c_i\}.$1

For $\{-c_i,0,+c_i\}.$2, $\{-c_i,0,+c_i\}.$3, this becomes

$\{-c_i,0,+c_i\}.$4

which the appendix describes as roughly $\{-c_i,0,+c_i\}.$5 fewer operations than dense multiplication.

4. Effects on weight geometry and activation statistics

On the weight side, KOTMS reshapes the coordinate system so that each rotated row becomes more compatible with the anchors $\{-c_i,0,+c_i\}.$6, $\{-c_i,0,+c_i\}.$7, and $\{-c_i,0,+c_i\}.$8 (Zhao et al., 11 Jun 2026). The paper characterizes the before-and-after contrast qualitatively: before KOTMS, rows are Gaussian-like and concentrated around zero; after KOTMS, they exhibit a more structured multi-cluster or symmetric tri-modal form. The appendix visualizes this on Layer 12 of Qwen3-8B, where the post-rotation distribution shows reduced tails and fewer outliers.

On the activation side, the same orthogonal transform is shared through inverse rotation equivalence. The paper states that this shared mixing disperses concentrated activation directions, reduces anisotropy, shrinks activation outliers statistically, and stabilizes dynamic range. This is presented as essential for $\{-c_i,0,+c_i\}.$9 quantization, because low-bit activation quantizers are highly sensitive to a few extreme values dominating the scale.

The appendix formalizes the activation argument using the RMS and peak-to-RMS ratio of an activation vector $\mathbf{W}\in\mathbb{R}^{n\times m}$0: $\mathbf{W}\in\mathbb{R}^{n\times m}$1 For orthogonal $\mathbf{W}\in\mathbb{R}^{n\times m}$2,

$\mathbf{W}\in\mathbb{R}^{n\times m}$3

Under random-mixing concentration arguments, the appendix gives

$\mathbf{W}\in\mathbb{R}^{n\times m}$4

and therefore

$\mathbf{W}\in\mathbb{R}^{n\times m}$5

These bounds justify the claim that orthogonal mixing can reduce activation spikiness from worst-case $\mathbf{W}\in\mathbb{R}^{n\times m}$6 behavior to logarithmic dependence. The paper is explicit, however, that this activation improvement is statistical rather than uniformly guaranteed for every layer. That caveat is central to the design of ILA-AMP.

The appendix also reports cross-layer heterogeneity in activation effects. For Q-projection input activations in Layers 4, 12, 24, and 36 of Qwen3-8B, KOTMS improves quantizability unevenly. Some layers, such as Layer 24, exhibit much tighter quantile bands after rotation; weaker-benefit layers, such as Layer 4, retain more residual high-quantile spikes. This observation is not incidental: it is the stated reason a uniform activation precision is suboptimal.

5. Optimization procedure, pipeline placement, and empirical evidence

KOTMS is learned during calibration rather than pretraining or end-to-end finetuning. The implementation details specify 128 calibration samples drawn from WikiText2, each with sequence length 2048. KOTMS parameters are optimized for 100 iterations with fixed learning rate 0.01 (Zhao et al., 11 Jun 2026).

The practical optimization loop is sequential. For each layer, $\mathbf{W}\in\mathbb{R}^{n\times m}$7 and $\mathbf{W}\in\mathbb{R}^{n\times m}$8 are initialized, for example to zeros; skew-symmetric generators are formed as

$\mathbf{W}\in\mathbb{R}^{n\times m}$9

orthogonal factors are built through Cayley transforms; the layer rotation is assembled as

$i$0

row-wise $i$1 and $i$2 are computed, with

$i$3

posterior responsibilities are evaluated; and gradient descent updates $i$4 using

$i$5

The paper also states how the Kronecker dimensions are chosen: start from $i$6, search downward for the nearest divisor, and return $i$7 with $i$8 and $i$9. This balancing is used to minimize overhead.

KOTMS is not jointly optimized with the rest of TWLA. The full pipeline is sequential: first KOTMS learns and folds layerwise orthogonal rotations; second E2M-ATQ ternarizes the rotated weights; third ILA-AMP allocates activation bits. This ordering is central to the paper’s interpretation of KOTMS as geometric preprocessing rather than a quantizer.

The modular ablations show that KOTMS is complementary rather than standalone. Without activation quantization ($\mathbf{w}_i$0), on LLaMA2-13B: baseline gives C4 PPL $\mathbf{w}_i$1, MMLU $\mathbf{w}_i$2; E2M-ATQ only gives C4 $\mathbf{w}_i$3, MMLU $\mathbf{w}_i$4; KOTMS only gives C4 $\mathbf{w}_i$5, MMLU $\mathbf{w}_i$6; and E2M-ATQ + KOTMS gives C4 $\mathbf{w}_i$7, MMLU $\mathbf{w}_i$8. On Qwen3-14B: baseline gives C4 $\mathbf{w}_i$9, MMLU $\mathbf{R}\in\mathbb{R}^{m\times m}$0; E2M-ATQ only gives C4 $\mathbf{R}\in\mathbb{R}^{m\times m}$1, MMLU $\mathbf{R}\in\mathbb{R}^{m\times m}$2; KOTMS only gives C4 $\mathbf{R}\in\mathbb{R}^{m\times m}$3, MMLU $\mathbf{R}\in\mathbb{R}^{m\times m}$4; and E2M-ATQ + KOTMS gives C4 $\mathbf{R}\in\mathbb{R}^{m\times m}$5, MMLU $\mathbf{R}\in\mathbb{R}^{m\times m}$6. These ablations show that KOTMS alone does not replace ternary quantization, but materially improves results when paired with E2M-ATQ.

Under $\mathbf{R}\in\mathbb{R}^{m\times m}$7, the same pattern appears but with stronger dependence on KOTMS. For LLaMA2-13B: E2M-ATQ only, without KOTMS or ILA-AMP, gives C4 $\mathbf{R}\in\mathbb{R}^{m\times m}$8, MMLU $\mathbf{R}\in\mathbb{R}^{m\times m}$9; E2M-ATQ + KOTMS, without ILA-AMP, gives C4 $\mathrm{A4}$00, MMLU $\mathrm{A4}$01; E2M-ATQ + ILA-AMP, without KOTMS, gives C4 $\mathrm{A4}$02, MMLU $\mathrm{A4}$03; and full TWLA gives C4 $\mathrm{A4}$04, MMLU $\mathrm{A4}$05. For Qwen3-14B: E2M-ATQ only gives C4 $\mathrm{A4}$06, MMLU $\mathrm{A4}$07; E2M-ATQ + KOTMS only gives C4 $\mathrm{A4}$08, MMLU $\mathrm{A4}$09; E2M-ATQ + ILA-AMP only gives C4 $\mathrm{A4}$10, MMLU $\mathrm{A4}$11; and full TWLA gives C4 $\mathrm{A4}$12, MMLU $\mathrm{A4}$13. The paper’s interpretation is that KOTMS is necessary for low-bit activation viability but still insufficient without ILA-AMP.

The main benchmark results support the same conclusion indirectly. On LLaMA2-70B, W1.58A4, TWLA reaches 71.10 average zero-shot accuracy versus 56.43 for ResQ and 53.64 for QuaRot. On Qwen3-32B, W1.58A4, TWLA reaches 65.25, while GPTQ and PB-LLM collapse near the mid-20s. Since KOTMS is the only TWLA component explicitly designed to reshape weights into ternary-friendly form and suppress activation outliers, these results reinforce its central role.

The overhead study further shows that balanced Kronecker factors are operationally important. On the Qwen3 family, making $\mathrm{A4}$14 and $\mathrm{A4}$15 more balanced reduces orthogonal-transform overhead from roughly 1 bit to <0.01 bit, while average accuracy drops by <2\%. This is why balanced factorizations are used in all experiments.

6. Scope, misconceptions, and relation to adjacent Kronecker methods

Several common misunderstandings can be resolved directly from the formulation. First, KOTMS is not ternarization itself. The ternary projection is handled by E2M-ATQ; KOTMS changes the coordinate system in which that projection is performed. Second, “tri-modal” refers to a three-mode weight distribution centered on $\mathrm{A4}$16, not to three data modalities. Third, KOTMS is not sufficient by itself for robust $\mathrm{A4}$17 PTQ: the paper’s own ablations show that it underperforms the full combination with E2M-ATQ and ILA-AMP (Zhao et al., 11 Jun 2026).

The paper also states clear limitations. Its activation benefit is indirect because the learning objective is defined purely in the weight domain, so the gains are heterogeneous across layers. It does not provide a theorem guaranteeing that every learned rotation will produce a tri-modal structure for every layer. Nor does it discuss catastrophic failure cases in detail beyond the observed dependence on E2M-ATQ and ILA-AMP.

A useful boundary can be drawn by comparing KOTMS to the multimodal embedding method “Fusing Cross-modal and Uni-modal Representations: A Kronecker Product Approach” (Wu et al., 10 Jun 2025). That work also uses Kronecker products for geometry shaping, but in a different regime: it fuses pretrained cross-modal and uni-modal embeddings so that same-modality similarities become multiplicatively sharpened while cross-modal alignment is largely preserved. Its core mechanism is a product-kernel construction, optionally approximated by random projection. By contrast, KOTMS uses a Kronecker-structured orthogonal rotation in LLM post-training quantization, with a tri-modal Gaussian-mixture shaping objective and activation-side effects arising through inverse rotation equivalence. The comparison shows that Kronecker algebra supports multiple kinds of geometric intervention, but only KOTMS combines orthogonality, ternary-codebook alignment, and activation-outlier suppression in the specific setting of $\mathrm{A4}$18 PTQ.

A plausible implication is that KOTMS belongs to a broader family of analytic geometry-shaping techniques in which the coordinate system is redesigned before downstream compression or retrieval. However, the validated scope in the paper remains narrow and specific: KOTMS is presented and evaluated as a PTQ module within TWLA for ternary-weight, low-bit-activation quantization of LLMs, not as a general-purpose rotation framework.