---
title: 'TensorGPT: Efficient Tensor-Train Compression for LLMs'
url: https://www.emergentmind.com/topics/tensorgpt
type: topic
---

# TensorGPT: Efficient Tensor-Train Compression for LLMs

TensorGPT refers to a class of compression approaches for large language models (LLMs), particularly those within the GPT family, based on tensor-train decomposition (TTD) and matrix product state (MPS) representations. TensorGPT targets drastic reductions in model memory and storage requirements—making transformer architectures feasible for deployment on edge or resource-constrained devices—by exploiting the inherent low-rank structure of both token embeddings and weight matrices. It also includes schemes for end-to-end training and inference with tensorized layers. The fundamental innovation lies in replacing high-dimensional parameter matrices with a sequence of low-dimensional "cores," yielding significant parameter compression while preserving language modeling capacity [2307.00526, 2306.02697].

## 1. Challenges in Large Language Models and Token Embeddings

State-of-the-art LLMs such as GPT-2 allocate a substantial portion of total parameters to the token embedding matrix. For instance, GPT-2's embedding table $W \in \mathbb{R}^{V \times d}$ (with $V = 50\,257$, $d = 768$) constitutes approximately 31% of its overall 1.5 B parameters. The resulting tens of millions of parameters in $W$ not only increase storage burdens but also preclude on-device deployment, such as on a Raspberry Pi, which is restricted by limited RAM/flash capacity. The situation is aggravated if the vocabulary is dynamic, as naïvely recomputing a full SVD for every new token insertion is computationally infeasible [2307.00526].

These bottlenecks also affect the fully connected layers in the Transformer’s feed-forward blocks, where each linear projection contains millions of parameters—complicating training, storage, and inference, especially for non-datacenter applications [2306.02697].

## 2. Tensor-Train Decomposition and Matrix Product States

Tensor-Train Decomposition (TTD), introduced by Oseledets (2011), expresses high-dimensional tensors $\mathcal{X} \in \mathbb{R}^{I_1 \times \cdots \times I_N}$ as products of $N$ cores $\{G^{(n)}\}$, each of size $R_{n-1} \times I_n \times R_n$, with $R_0 = R_N = 1$. The representation is:
$$
\mathcal{X}_{i_1,\dots,i_N} \approx \sum_{r_0,\dots,r_N} G^{(1)}_{r_0, i_1, r_1} G^{(2)}_{r_1, i_2, r_2} \cdots G^{(N)}_{r_{N-1}, i_N, r_N}
$$
In physics, this format is known as the Matrix Product State (MPS). TT-SVD, a sequential SVD procedure, facilitates automated extraction of these cores (see Alg. 1 in [2307.00526]). By choosing the mode sizes $\{I_n\}$ and number of cores $N$ to approximate the original dimension and by truncating SVDs to lower TT-ranks, substantial parameter compression is achieved.

For weight matrices $W \in \mathbb{R}^{D_{in} \times D_{out}}$, a closely related Tensor Train Matrix (TTM) factorization views $W$ as a $2M$-way tensor and factorizes it into $M$ matrix-valued TT cores, reducing storage from $D_{in}\cdot D_{out}$ to a sum scaling as $O(R^2 M \max\{I_m, J_m\})$, where $R$ is the uniform TT-rank and $I_m$, $J_m$ are mode sizes for input/output [2306.02697].

## 3. Methodology and Implementation in TensorGPT

### Token Embedding Compression (MPS Approach)

Given the embedding matrix $W \in \mathbb{R}^{V \times d}$, TensorGPT compresses each token embedding $x_v \in \mathbb{R}^d$ as follows:

1. **Padding & Tensorization**: Pad $d$ to a product of small mode sizes, ideally a power of two. For GPT-2, $d=768$ is padded to $d'=1024 = 2^{10}$, giving $N=10$, $I_n=2$.
2. **Per-token MPS/TT-SVD**: Reshape $x_v$ to an order-$N$ tensor $\mathcal{X}^{(v)} \in \mathbb{R}^{2 \times \cdots \times 2}$ and perform TT-SVD, producing cores $G^{(v, n)}$ for $n=1,\ldots,N$.
3. **Storage**: Retain the set of TT cores across all tokens, vastly reducing memory versus storing the dense $x_v$ [2307.00526].

This per-token compression is training-free, requires no extra data, and leverages a scan over TT-rank choices $R_n$ (e.g., $R_n \in \{1,2,4,8\}$) to trade off reconstruction accuracy and storage.

### Tensor Train Matrix for Projections

For fully-connected layers, the Tensor Train Matrix (TTM) representation proceeds as follows:

- **Reshape**: $W \in \mathbb{R}^{D_{in} \times D_{out}}$ is viewed as a $2M$-way tensor, dimensions $I_1,\ldots,I_M= D_{in}$; $J_1,\ldots,J_M= D_{out}$.
- **TTM Factorization**: $W_{i_1, j_1, ..., i_M, j_M} = \sum G^{(1)}_{r_0, i_1, j_1, r_1} \cdots G^{(M)}_{r_{M-1}, i_M, j_M, r_M}$, where each $G^{(m)}$ is a rank-$R_m$ four-way core.
- **Forward and Backward Propagation**: Efficient einsum-based forward contraction and custom memory-aware backward passes are implemented. The optimal (Einsum forward + Full-Matrix backward) reduces per-layer gradient memory from $\sim$1 GB to $\sim$192 MB [2306.02697].

## 4. Parameter Reduction and Empirical Performance

### Embedding Layer Compression

Theoretical compression ratio is:
$$
\eta = \frac{d}{\sum_{n=1}^N R_{n-1} I_n R_n}
$$
For typical choices ($I_n = 2$, $R_n=4$ for $n=2\dots9$), this yields $\eta \approx 3.31\times$, i.e., a 3.31-fold reduction. Aggressive choices ($R_n=1$) can achieve up to $38.4\times$ compression [2307.00526]. For GPT-2, this method theoretically results in $46.89\%$ fewer total model parameters, with a compression ratio $39.38\times - 65.64\times$ for the embedding layers.

| Method                      | TT-ranks                  | η (×) | Generation Loss |
|-----------------------------|---------------------------|-------|-----------------|
| Original                    | –                         | 1.00  | 13.71           |
| Direct TT on $W$ (2-core)   | [1,32,1]                  | 23.64 | 10.14           |
| MPS on each $x_v$           | [1,2,4,\ldots,4,2,1]      | 3.31  | 9.01            |
| MPS on each $x_v$ (max η)   | [1,1,...,1]               | 38.40 | 20.77           |

A compression of $3.31\times$ yields a lower generation loss (9.01) than the original (13.71), indicating slight regularization from the low-rank approximation. Maximal compression reduces performance but maintains core semantic fidelity [2307.00526].

### TTM for Feed-Forward Layers

By replacing GPT-2’s dense MLP matrices with TTM layers, the following parameter reductions and effects on perplexity are observed [2306.02697]:

| Model      | #Params (M) | % of Baseline | Perplexity |
|------------|------------|---------------|------------|
| GPT-2 Small| 124.4      | 100%          | 17.55      |
| TTM-16     | 68.1       | 54%           | 21.33      |
| TTM-64     | 83.6       | 67%           | 18.08      |

A 40% size reduction leads to a perplexity increase of only ≈3.2, with downstream GLUE and summarization performance matching or slightly underperforming standard GPT-2 and surpassing SVD-compressed and distilled baselines.

## 5. Computational Complexity and Hardware Implications

The offline TT-SVD compression for embeddings has $O(dR^2_{max})$ per token cost. Online reconstruction requires $O(N R^2 I)$ per token, and storage is reduced to $O(V N R^2 I)$, which is significantly less than the $O(Vd)$ for the original matrices. For hardware-limited environments, such as Raspberry Pi, this enables models to fit entirely in memory or flash [2307.00526].

In the TTM approach for projections, TTM layers' forward passes have complexity $O(B M \max\{D_{in}, D_{out}\} \max_k\{I_k, J_k\} R^2)$; custom backpropagation routines ensure memory usage is comparable to dense layers [2306.02697].

## 6. Insights, Limitations, and Broader Applicability

A key empirical finding is that GPT embeddings admit a strong low-rank structure. Certain tensor modes exhibit lower mean absolute errors after reconstruction, suggesting interpretable subspace significance. While aggressive compression ($\eta > 20\times$) degrades generation quality, moderate compression ($3{-}10\times$) can preserve or even improve downstream task performance—potentially via regularization effects [2307.00526].

Practical implementation guidelines recommend padding embedding dimensions to powers of two, initializing with low TT-ranks, and compressing new tokens in dynamic vocabularies without affecting existing TT cores.

Current limitations include the need to integrate position/mask embeddings into the MPS format and to validate full end-to-end inference speedup in hardware. Ongoing extensions involve generalizing TT-based compression to other transformer blocks, including BERT, GPT-3, and OPT, by replacing dense projections with TTM layers, and designing TT-based transformer blocks for entire model compression and acceleration [2307.00526, 2306.02697].

## 7. Conclusion and Future Directions

TensorGPT demonstrates that large language models, especially those from the GPT family, have substantial low-rank redundancy in both embeddings and dense projections. By leveraging tensor-train decomposition and matrix product state representations, up to 38.4× parameter compression is achievable without additional fine-tuning, and with little or no loss in performance for moderate compression settings. The approach is broadly applicable across transformer-based architectures and is particularly advantageous for on-device applications constrained by memory and storage. A plausible implication is that further efficiency gains are attainable by integrating positional/mask encodings into the tensorized formats and by implementing TT-based end-to-end transformer blocks, paving the way for highly compressed and efficient LLM deployments [2307.00526, 2306.02697].

Source: https://www.emergentmind.com/topics/tensorgpt