---
title: 'TSV-Compress: Efficient Model Compression'
url: https://www.emergentmind.com/topics/tsv-compress-tsv-c
type: topic
---

# TSV-Compress: Efficient Model Compression

TSV-Compress (TSV-C) is a model compression technique designed to reduce the storage and computational requirements of per-task fine-tuned neural network weights, while preserving accuracy. TSV-C leverages the observed low-rank structure of "layer-task matrices," compressing them to approximately 10% of their original size with minimal accuracy degradation, typically retaining at least 99% of original task performance. TSV-C is integral to pipelines such as model merging, where per-task parameter changes need to be efficiently represented and combined [2412.00081].

## 1. Definition and Mathematical Formulation

For a pretrained model backbone and a given downstream task $\tau$, let $\theta_{pre}$ be the pretrained weights and $\theta_{ft}(\tau)$ the fine-tuned weights. At a specific layer $t$ (where the weights are naturally a matrix, such as those in fully-connected or convolutional layers), the layer-task matrix is defined as
$$
M_\tau^t \equiv \Delta_\tau^t = \theta_{ft}(\tau)^t - \theta_{pre}^t,
$$
where $M_\tau^t \in \mathbb{R}^{m \times n}$. For layers not natively represented as matrices, TSV-C defaults to ordinary Task Arithmetic without compression [2412.00081].

## 2. Algorithmic Procedure and SVD-Based Compression

The core of TSV-Compress is the use of truncated singular value decomposition (SVD) to approximate each $M_\tau^t$:
$$
M_\tau^t = U_\tau^t \Sigma_\tau^t (V_\tau^t)^T,
$$
with $U_\tau^t \in \mathbb{R}^{m \times k}$, $\Sigma_\tau^t \in \mathbb{R}^{k \times k}$, $V_\tau^t \in \mathbb{R}^{n \times k}$, $k = \min(m, n)$. To produce a compressed approximation, TSV-C forms a rank-$r$ truncated version:
$$
\hat M_\tau^t = U_\tau^t[:, 1:r] \;\Sigma_\tau^t[1:r, 1:r] \; V_\tau^t[:, 1:r]^T,
$$
where $r$ is the minimal value such that
$$
\frac{\sum_{i=1}^r \sigma_i}{\sum_{i=1}^k \sigma_i} \geq 0.99,
$$
ensuring at least 99% Frobenius norm (energy) retention. Simultaneously, TSV-C enforces $r \leq \lceil 0.10\, k \rceil$, so at most 10% of singular components are preserved, capping storage usage at 10% of the original per-layer parameters [2412.00081].

## 3. Implementation Details

The TSV-Compress procedure is as follows (summarized in pseudocode):

```python
Input:
    – Pretrained weights θ_pre
    – Fine-tuned weights θ_ft(τ) for each task τ
    – Target energy α = 0.99, max fraction f = 0.10
Output:
    – Compressed task differences { ˆΔ_τᵗ }

for each task τ do
    for each layer t with weight matrix M ← θ_ft(τ)ᵗ − θ_preᵗ do
        [m,n] ← dimensions of M
        k ← min(m,n)
        [U, Σ, Vᵀ] ← SVD(M)
        s ← diag(Σ); total ← sum(s); csum ← 0; r_thresh ← 0
        for i in 1..k do
            csum ← csum + s[i]
            if csum / total ≥ α then
                r_thresh ← i
                break
        r_max ← ceil(f * k)
        r ← min(r_thresh, r_max)
        U_r ← U[:, 1:r]; Σ_r ← Σ[1:r,1:r]; V_r ← V[:, 1:r]
        store (U_r, Σ_r, V_r) as ˆΔ_τᵗ
    end
end
```

At inference, the compressed $\hat\Delta_\tau^t$ is reconstructed for each layer as
$$
\hat\Delta_\tau^t = U_r \Sigma_r V_r^T,
$$
and the modified layer weights are
$$
\theta_{MT}^t = \theta_{pre}^t + \alpha_{merge} \cdot \hat\Delta_\tau^t,
$$
with $\alpha_{merge} = 1.0$ by default [2412.00081].

Storage per layer is reduced from $m \times n$ parameters to $r(m + n + 1)$, which, given $r \leq 0.1 \min(m, n)$, guarantees storage cost is at most $10\%$ of the original for each eligible layer.

## 4. Empirical Compression Performance

Empirical results on the ViT-B-32 architecture are summarized as follows:

| Method                    | 8 tasks      | 14 tasks     | 20 tasks     |
|---------------------------|------------- |------------- |------------- |
| Finetuned (100%)          | 92.83 (100)  | 90.88 (100)  | 91.37 (100)  |
| TALL-Mask + TIES          | 93.13 (100.4)| 90.92 (100)  | 91.11 (99.7) |
| TSV-C (Ours)              | 92.62 (99.7) | 90.29 (99.3) | 90.64 (99.1) |

Subscripts represent normalized accuracy (percentage of original). For all scenarios, TSV-C uses approximately 10% of per-task parameter storage, retaining at least 99% of original accuracy [2412.00081].

## 5. Computational Complexity and Practical Considerations

- **SVD Complexity:** The standard per-layer SVD operation has complexity $\mathcal{O}(m n k)$, but can be accelerated to $\mathcal{O}(m n r)$ with randomized SVD methods, which are suitable due to the small retained rank $r$.
- **Storage:** For a weight matrix of size $m \times n$, storing decomposed forms at rank $r$ requires
$$
m r + r + r n = r(m + n + 1)
$$
parameters per layer, maximizing at $0.1\, m n$ for $r = 0.1 \cdot \min(m, n)$.
- **Implementation Tips:**
  - Batch SVD computations across layers or tasks using GPU libraries, such as `torch.linalg.svd`.
  - Employ randomized or truncated SVD to manage compute cost for large matrices.
  - Store singular factors contiguously as three tensors per layer for efficient reconstruction.
  - Preallocate and reuse workspace buffers to minimize GPU memory fragmentation.
  - For multi-task scenarios with a shared backbone, cache the SVD of the pretrained weights to reduce computation [2412.00081].

## 6. Integration and Use in Model Merging

TSV-Compress is designed for seamless integration into model merging pipelines, where low-rank, compressed task-specific deltas can be reconstructed on demand and combined with the pretrained backbone. This is particularly valuable in settings where many tasks share a backbone, as only the compressed singular factors (Task Singular Vectors) for each delta need to be stored and transmitted. The existence of low-rank layer-task matrices also supports improved model merging methods, such as TSV-Merge, which leverage singular vector interactions to reduce task interference [2412.00081].

## 7. Comparative Context and Applications

TSV-Compress offers a compact and accurate alternative to existing compression and masking schemes such as TALL-Mask + TIES, maintaining competitive or superior accuracy with an explicit mathematical guarantee on energy retention and storage footprint. Its applicability is circumscribed to layers naturally represented by matrices (e.g., fully-connected, convolutional structures), with ordinary task arithmetic fallback otherwise. The demonstrated results substantiate its utility for scalable multi-task adaptation, federated settings, and efficient model deployment [2412.00081].

Source: https://www.emergentmind.com/topics/tsv-compress-tsv-c