---
title: LinearlyCompressedGPT Compression Techniques
url: https://www.emergentmind.com/topics/linearlycompressedgpt
type: topic
---

# LinearlyCompressedGPT Compression Techniques

LinearlyCompressedGPT refers to a broad family of techniques for reducing the computational and memory footprint of GPT-style (decoder-only Transformer) language models through purely linear structural modifications, factorization, or quantization, usually without full retraining. Unlike distillation, pruning, or non-linear rewiring, these methods compress the large matrix multiplications and embeddings at the core of Transformer models using quantization, structured matrix approximations, hierarchical blockwise dimension reductions, or sparse and grouped computation. This enables deployment to resource-constrained environments while maintaining acceptable loss in performance. LinearlyCompressedGPT has been realized through a variety of frameworks; major lines of work include blockwise quantization, Kronecker and tensor-train (TT) decomposition, hierarchical dynamic grouping, and progressive depthwise linear projection.

## 1. Blockwise Quantization: The BCT Approach

Blockwise Compression of Transformers (BCT) implements LinearlyCompressedGPT through *blockwise shift quantization* on all linear matrices and bias vectors, without retraining [2304.01483]. The method partitions each matrix into discrete B×B blocks and applies independent scale quantization within each. This sharply reduces quantization-induced distribution shift compared to per-layer schemes, eliminating retraining requirements.

**Core Quantization Process:**  
Let $x\in\mathbb{R}^{B\times B}$, $k$ denote bit-width, $I_k=[-2^{k-1},2^{k-1}-1]$.
- Compute the block’s shift:  
  $\text{shift}(x) = \lfloor \log_2\,(\max_{i,j}|x_{i,j}|\,/\,2^{k-1}) \rfloor$
- Quantize:  
  $x_c = \text{clip}\left(\text{round}(x \cdot 2^{-\text{shift}}), -2^{k-1},2^{k-1}-1\right)$
- Decompress:  
  $Q^{-1}(x_c;\text{shift}) = x_c \cdot 2^{\text{shift}}$

At inference, GEMMs are computed in low-bit, exponent-aligned blocks.

**Error Bound and Theoretical Guarantees:**  
Elementwise quantization error is bounded by half a quantization bin; error per layer is $O(2^{-k})$. Residuals do not induce out-of-distribution behavior globally, and empirical boxplots demonstrate per-block error does not propagate destructively.

**Empirical Results:**
- BERT-base (as a stand-in for GPT): $4$-bit weights plus $8$-bit activations yield $7.988\times$ model size reduction; $<1\%$ accuracy loss (e.g., $-0.8\%$ GLUE SST-2). Pure $8$-bit quantization yields $4\times$ reduction with near-zero loss.
- fp8 (8-bit float) BCT achieves $4\times$ size reduction with $<0.01\%$ accuracy loss.

**Block/Bit Parameterization and Trade-offs:**  
Block size $B=32\ldots128$ is typical; $k=4$ for aggressive shrinkage, $k=8$ for zero-loss compression. Larger $B$ reduces meta-data but coarsens the quantization; smallest $k$ and largest $B$ that maintain acceptable perplexity are recommended.

## 2. Hierarchical Dynamic Grouping and Attention

Hierarchical models, exemplified by GPTHF, fundamentally restructure the Transformer, compressing token sequences into fixed-size sentence embeddings and operating subsequent transformations at the sentence level [2503.11426].

**Architecture:**
- A word-level Transformer encoder processes tokens within a sentence using block-local self-attention.
- Sentence-level representations are pooled, forming $e_i =$ Pooling$_{s_i}$(wlt_encoder(...)) for each sentence.
- A second Transformer “body” operates causally over sentence embeddings.

**Dynamic Sparse Attention:**
- During encoding, attention is masked to intra-sentence blocks.
- At the sentence level, embeddings attend to all prior sentences.

**Inference and Caching Optimization:**  
By caching finished sentence embeddings, GPTHF reuses computation, yielding per-token complexity that scales as $O(L_s^2+S^2)$ (where $L_s$ is sentences), rather than $O(N^2)$ tokenwise.

**Empirical Trade-offs:**  
GPTHF achieves up to $10\times$ reduction in FLOPs and $3\times$ speedup on certain tasks, at the expense of $\sim 5$-point perplexity penalty. Sentence-splitting is critical; generation quality can be affected by sentence-boundary prediction [2503.11426].

## 3. Structured Linear Factorizations: Kronecker, TT, and Orthogonal Transformations

Several schemes target the replacement of dense matrices with mathematically structured, low-parametric forms:

### 3.1 Kronecker Products

Krony-PT and KnGPT2 both compress transformer and embedding matrices via Kronecker factorizations [2412.12351, 2110.08152]. For a weight $W \in \mathbb{R}^{m\times n}$, choose $m=m_1 m_2,\, n=n_1 n_2$, then approximate $W \approx A \otimes B$, with storage dropping to $m_1 n_1 + m_2 n_2$ from $mn$.

- Krony-PT: Either single or multi-factor, leveraging Van Loan SVD initialization or pruning-based methods. Compression of the GPT-2 FFN from $3072\times768$ by factors $4$ and $1$ gives effective models of $81$M versus original $124$M—perplexity outperforms distillation baselines [2412.12351].
- KnGPT2: Compresses half of all linear layers and embedding via rank-1 factorizations, recovers performance with only minimal pretraining (intermediate-layer KD) [2110.08152].

### 3.2 Tensor-Train (TT) Decomposition

TTD reshapes large matrices into high-order tensors and factors them into sequential “cores” [2501.19135, 2307.00526]. Given $W\in\mathbb{R}^{M\times N}$, tensorize $M,N$ into $d$-tuples, then represent $W$ as multiplication through a chain of $d$ TT-cores. Compression ratio is:

$$
\text{CR} = \frac{MN}{\sum_{k=1}^d (m_k n_k) r_{k-1} r_k}
$$

- TTD achieves layer-level compression up to $1000\times$ and whole-network $1.6$–$1.94\times$, with minimal loss: e.g., $+2.62$ PPL, $-4.21$ C-EVal for ChatGLM3-6B, LLaMA2-7B [2501.19135].

TT is particularly effective for the embedding layer (e.g., experimentally, $2\times$–$3.3\times$ compression with negligible loss) [2307.00526].

### 3.3 Orthogonal Transforms and Structured Projections

ProcrustesGPT rotates weights via orthogonal $Q$ to maximize compressibility under structured families such as Kronecker-sum or permutation-sparse matrices [2506.02818]. Layerwise alternating minimization is performed:

- Step A: Project weights into chosen structured class given $Q$.
- Step B: Solve a weighted Orthogonal Procrustes Problem to update $Q$.

Without fine-tuning, $14$–$25\%$ weight compression is attainable with consistently lower perplexity than other fine-tuning-free baselines [2506.02818].

## 4. Architectural Linear Projection Variants

A distinct technique modifies the GPT stack by inserting linear dimensionality reductions between groups of layers. In the LinearGPT architecture (lc-gpt), after every two blocks, the hidden dimension is linearly halved, with intermediate linear layers $W^{(i)}\in\mathbb{R}^{D_{i+1}\times D_i}$ [2404.14462].

**Structural Recursion:**
$$
\mathbf{x}^{(i+1)} = W^{(i+1)}\, \mathrm{Block}_{D_i}(\mathrm{Block}_{D_i} (\mathbf{x}^{(i)}))
$$

This reduces total parameter count by $36\%$ and speeds up training by $19\%$ with no measurable loss in task performance on code-completion objectives.

## 5. Vocabulary and Output Layer Compression

High memory and compute cost in the output head can be dominated by the vocabulary projection. A two-level grouping approach partitions the vocabulary using BPE merges, then applies shared per-group linear transformations with per-group scale and shift [2411.06371].

- For $|v|$-way softmax, introduce $G=\sqrt{|v|}$ groups and $S=\sqrt{|v|}$ tokens per group. The softmax is decomposed as:

$$
p_\text{vocab}(g\cdot S + t) = p_\text{group}[g]\cdot p_{\text{token|g}}[t]
$$

- Reduces activation memory up to $3.4\times$ and speeds up throughput by up to $3\times$, with negligible drop in human-rated TinyStories metrics.

## 6. Hardware Mapping and Inference Acceleration

TTD-compressed models mapped to hardware such as FPGA via Group Vector Systolic Array (GVSA) architectures deliver further acceleration [2501.19135]. Execution of TT-sharded matrix multiplies is serviced by parallel vector PEs, with pipelined partial sum reordering. ChatGLM3-6B and LLaMA2-7B deployed in this format achieved $1.45\times$-$1.57\times$ first-token delay reductions and throughput exceeding optimized GPU baselines.

## 7. Trade-Offs, Limitations, and Selection Guidelines

- **Compression vs. Accuracy:** Aggressive quantization ($k=4$) or deep low-rank factorization provides compression up to $8\times$, typically at $<1\%$–$5\%$ loss in perplexity or task accuracy, depending on the scheme [2304.01483, 2503.11426, 2412.12351].
- **Block/Rank Choices:** Empirical sweep of block size, bit-width, Kronecker rank, or TT-rank is essential—default recommendations include block size $B=64$, $k=4$ or $8$, Kronecker rank $r=2$–$4$, TT ranks chosen to keep per-layer error within $+0.5\%$ PPL.
- **No-Retrainability:** Methods such as BCT and ProcrustesGPT can be applied directly to a pretrained model with calibration data only, avoiding expensive retraining loops [2304.01483, 2506.02818].
- **Applicability:** Structure-based methods (Kronecker, TT) are amenable both to encoder and decoder (GPT) architectures and can be combined with other compression regimes (pruning, quantization).
- **Limitations:** Sentence-compression and hierarchical methods can induce sentence boundary and generation quality artifacts, especially on small models without auxiliary modeling [2503.11426].

## References

- BCT: Blockwise Compression of Transformer-based Models without Retraining [2304.01483]
- GPTHF: Text Compression for Efficient Language Generation [2503.11426]
- ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations [2506.02818]
- Krony-PT: GPT2 compressed with Kronecker Products [2412.12351]
- KnGPT2: Kronecker Decomposition for GPT Compression [2110.08152]
- TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition [2307.00526]
- LLM Vocabulary Compression for Low-Compute Environments [2411.06371]
- LC-GPT: Towards smaller, faster decoder-only transformers [2404.14462]
- TTD on FPGA: A Tensor-Train Decomposition based Compression of LLMs on Group Vector Systolic Accelerator [2501.19135]

LinearlyCompressedGPT frameworks thus offer a flexible design space, ranging from quantization to low-rank tensorization, for shrinking GPT-family models while retaining their essential generative capacity.

Source: https://www.emergentmind.com/topics/linearlycompressedgpt