Pareto-Guided SVD Compression for LLMs and VLMs

• PGSVD is a zero-shot, activation-informed low-rank compression framework that leverages layer activations to guide a Pareto-optimal trade-off between model size and loss change.
• It formulates compression as a bi-objective optimization using a uniform tolerance to determine heterogeneous per-layer ranks based on activation errors.
• The method employs an alternating least-squares factorization that optimizes activation-weighted SVD, reducing parameters and FLOPs while maintaining model accuracy.

Searching arXiv for the specified paper and closely related work on PGSVD and activation-aware low-rank compression. Pareto-Guided Singular Value Decomposition (PGSVD) is a zero-shot, activation-informed low-rank compression framework for LLMs and vision–LLMs (VLMs). It combines a loss-sensitivity upper bound that ties layer-wise activation-based compression to network-level loss, a bi-objective formulation of compression as a trade-off between model size and loss change, and a Pareto-guided rule that selects heterogeneous ranks through a single uniform tolerance. In the formulation introduced in "Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM" (Solgi et al., 7 Oct 2025), the resulting pipeline uses sampled activations to guide factor optimization and an alternating least-squares (ALS) solver to improve stability and accuracy over truncation-based SVD.

1. Definition and problem setting

PGSVD addresses the deployment problem posed by foundation models that deliver strong performance but impose severe memory and compute costs during inference. The framework targets zero-shot compression—without retraining—in order to reduce both memory and FLOPs while preserving model quality. Its basic mechanism is low-rank factorization, in which a weight matrix $W$ is replaced by factors $A B$, with $A \in \mathbb{R}^{N\times r}$ and $B \in \mathbb{R}^{r\times M}$. This reduces parameters from $N M$ to $r(N+M)$ and lowers per-forward FLOPs from $O(NM)$ to $O(r(N+M))$ (Solgi et al., 7 Oct 2025).

The framework is motivated by a limitation of direct weight-space compression. Many pre-trained layers do not exhibit low-rank weights, so direct truncation of $W$ can degrade accuracy substantially. By contrast, activations $X$ often exhibit stronger low-rank structure. PGSVD therefore treats activation-informed distortion, rather than weight-space distortion alone, as the relevant compression target. The conventional truncated SVD objective is

$A B$0

whereas the activation-aware objective is

$A B$1

The distinction is consequential. The activation-informed distortion measures how the compressed layer perturbs the layer outputs under observed inputs. This suggests that PGSVD is not merely a matrix approximation procedure, but a deployment-oriented compression scheme in which the optimization target is aligned with downstream model behavior.

2. Loss sensitivity and surrogate compression theory

The theoretical core of PGSVD is a loss-sensitivity upper bound linking activation-based layer compression errors to the network-level change in loss. Consider a feedforward layer $A B$2 with pre-activations $A B$3, an elementwise nonlinearity $A B$4, and batch activations

$A B$5

Let the compressed weight be $A B$6, and assume $A B$7 has bounded slope $A B$8. For a differentiable scalar loss $A B$9, the stated theorem gives

$A \in \mathbb{R}^{N\times r}$0

where $A \in \mathbb{R}^{N\times r}$1, $A \in \mathbb{R}^{N\times r}$2, and

$A \in \mathbb{R}^{N\times r}$3

The proof sketch is explicitly first-order: a first-order perturbation analysis bounds the propagated activation perturbation by layer-dependent Jacobian norms and the compression-induced distortion $A \in \mathbb{R}^{N\times r}$4, and a first-order Taylor bound converts activation perturbation into loss change via $A \in \mathbb{R}^{N\times r}$5. The theoretical implication stated for the method is that, for fixed data and model, the coefficients are constants, so reducing $A \in \mathbb{R}^{N\times r}$6 provably reduces a surrogate upper bound on the loss change.

PGSVD then formulates compression as a bi-objective optimization. For a rank-$A \in \mathbb{R}^{N\times r}$7 approximation $A \in \mathbb{R}^{N\times r}$8, define the relative weight-space error

$A \in \mathbb{R}^{N\times r}$9

where $B \in \mathbb{R}^{r\times M}$0 is a tolerance. Let $B \in \mathbb{R}^{r\times M}$1 collect ranks and let

$B \in \mathbb{R}^{r\times M}$2

denote the network parameter count. The bi-objective problem is

$B \in \mathbb{R}^{r\times M}$3

Using $B \in \mathbb{R}^{r\times M}$4 and the loss-sensitivity theorem yields the scalar surrogate

$B \in \mathbb{R}^{r\times M}$5

with

$B \in \mathbb{R}^{r\times M}$6

Under a compression budget $B \in \mathbb{R}^{r\times M}$7, the surrogate rank-allocation program becomes

$B \in \mathbb{R}^{r\times M}$8

This formulation places PGSVD within a class of compression methods that optimize a tractable surrogate rather than the exact post-compression loss. A common misconception is that the framework’s theory directly optimizes the true loss; the stated result is instead an upper-bound argument, and the paper explicitly notes that the loss bound is first-order and conservative.

3. Pareto-guided rank allocation via a uniform tolerance

A distinctive aspect of PGSVD is its conversion of rank allocation into tolerance allocation. For any $B \in \mathbb{R}^{r\times M}$9 and $N M$0, the framework defines a unique minimal rank $N M$1 such that

$N M$2

with parameter count $N M$3. The associated map is

$N M$4

The paper states that the rank-allocation problem and the $N M$5-allocation problem

$N M$6

have the same optimal value (Solgi et al., 7 Oct 2025).

The central Pareto-guided claim is then made under a robust surrogate in which $N M$7 for all $N M$8, together with bounded, nonincreasing convex envelopes for each layer’s parameter function $N M$9. Under those assumptions, there exists a uniform solution $r(N+M)$0, and every uniform tolerance $r(N+M)$1 corresponds to a point on the surrogate Pareto frontier of the bi-objective problem. The per-layer ranks are nevertheless heterogeneous because each layer’s spectrum determines

$r(N+M)$2

This directly addresses another common misunderstanding: a single tolerance does not imply a single rank. The tolerance is uniform, but the ranks are heterogeneous because layers have different SVD profiles $r(N+M)$3. In the PGSVD formulation, the “single-knob” control variable is the tolerance $r(N+M)$4, while the rank pattern is induced by layer-specific spectra. For VLMs, the same idea is extended to separate tolerances $r(N+M)$5 and $r(N+M)$6 for the vision and text towers, respectively.

The budget-to-tolerance mapping is also explicit. Given a compression budget $r(N+M)$7, one solves for $r(N+M)$8 using the upper envelope $r(N+M)$9 via

$O(NM)$0

The layer-wise ranks then follow by the minimal-rank condition $O(NM)$1. This suggests that PGSVD replaces ad hoc layer-wise rank search with a structured Pareto-guided rule, although the theoretical guarantee is stated only for the homogeneous-sensitivity surrogate and bounded-profile assumptions.

4. Algorithmic pipeline and ALS factorization

PGSVD is presented as a zero-shot pipeline with four stages: activation sampling, tolerance selection, rank allocation, and activation-weighted ALS factorization. The procedure requires sampled activations but no retraining. The step-by-step pipeline is as follows.

First, representative inputs are run through the model to collect per-layer activations $O(NM)$2, stacked by columns. The empirical activation covariance is then formed as

$O(NM)$3

Second, a single uniform tolerance $O(NM)$4 is chosen, or per-modality tolerances $O(NM)$5 in VLMs. If a parameter or FLOP budget $O(NM)$6 is given, the mapping uses the convex envelope relation $O(NM)$7.

Third, each layer receives the minimal rank

$O(NM)$8

where $O(NM)$9 is the weight-space relative truncation error.

Fourth, low-rank factors are initialized by SVD on $O(r(N+M))$0. With

$O(r(N+M))$1

the initialization is

$O(r(N+M))$2

These factors are then refined by ALS to minimize

$O(r(N+M))$3

The ALS updates use the empirical activation covariance $O(r(N+M))$4: $O(r(N+M))$5 Only $O(r(N+M))$6 pseudo-inverses are required. The distinction from plain truncated SVD is explicit: truncated SVD optimizes $O(r(N+M))$7, whereas ALS optimizes $O(r(N+M))$8, thereby weighting errors by the empirical activation covariance $O(r(N+M))$9. The paper further states that this avoids the numerical fragility, including Cholesky failures, of whitening-based methods and is more efficient than EVD-based implementations.

The algorithmic outline takes as input weights $W$0, activation covariances $W$1, tolerance $W$2 or $W$3, and ALS iterations $W$4. For each layer, it computes $W$5, initializes $W$6 by rank-$W$7 SVD, performs $W$8 to $W$9 ALS updates, and returns the factorized model $X$0. The paper emphasizes that activation sampling affects factor optimization but not the rank allocation when using uniform $X$1 (Solgi et al., 7 Oct 2025).

5. Scope, computational characteristics, and implementation practice

The reported compression scope covers all linear layers in self-attention—query, key, value, and output projections—and all linear projections in MLP blocks. In CLIP-like VLMs, both the image encoder, such as ViT, and the text encoder transformers are compressed, typically with separate tolerances per modality.

The parameter and FLOP accounting are straightforward. For layer $X$2, parameters drop from $X$3 to $X$4, so the total parameter count is

$X$5

For inference on a single vector $X$6, FLOPs reduce from $X$7 to $X$8 through

$X$9

The main hyperparameter is the uniform tolerance $A B$00, or $A B$01 in multimodal settings. It may be chosen via a target budget $A B$02 using $A B$03, or by sweeping $A B$04 to trace the surrogate Pareto front. Activation sampling requires only a small batch of representative data per modality to estimate $A B$05. The reported implementation guidance is to use full precision for inference during activation collection and double precision during factorization for numerical stability. The recommended ALS setting is $A B$06, with empirical improvements plateauing between 5–10 iterations; even a single iteration yields acceptable perplexity. Stopping may be based on fixed $A B$07 or on stabilization of $A B$08. No regularization is required in the reported experiments.

These implementation choices place PGSVD between purely analytic compression and post-training adaptation. It is zero-shot and deployment-oriented, yet still activation-informed. A plausible implication is that its operational cost is concentrated in one pass for activation collection and a relatively small number of ALS refinements, rather than in any fine-tuning loop.

6. Empirical behavior, comparative position, and limitations

The reported evaluation covers LLMs—LLaMA-2-7B, LLaMA-2-13B, and Mistral-7B—and VLMs based on CLIP (Solgi et al., 7 Oct 2025). For LLMs, metrics include perplexity on WikiText-2 and zero-shot accuracy on ARC-E, CSQA, LAMBADA, PIQA, and WinoGrande, with additional tasks ARC-C, BoolQ, HellaSwag, MathQA, MMLU, and RACE. On LLaMA-2-7B at 20% compression, PGSVD reports PPL $A B$09 versus SVD-LLM $A B$10 and SVD-ALS $A B$11, with average accuracy $A B$12 versus $A B$13 and $A B$14. At 40% compression, it reports PPL $A B$15 versus $A B$16 and $A B$17, with average accuracy $A B$18 versus $A B$19 and $A B$20. On LLaMA-2-13B, the reported values are PPL $A B$21 and average accuracy $A B$22 at 20%, and PPL $A B$23 with average accuracy $A B$24 at 40%, outperforming both SVD-LLM and SVD-ALS. On Mistral-7B, the reported values are PPL $A B$25 and average accuracy $A B$26 at 20%, and PPL $A B$27 with average accuracy $A B$28 at 40%.

Throughput is reported to be similar to uniform-rank SVD-ALS under naïve Python implementations, with substantial gains over base models across sequence lengths. The ALS ablation states that compression times are lower than EVD-based SVD variants and that perplexity improvements plateau after approximately 5–10 ALS iterations.

For VLMs, the evaluation uses zero-shot Top-1 and Top-5 over Caltech101, Food101, OxfordPets, StanfordCars, EuroSAT, and DTD. At 20% compression, PGSVD reports average Top-1/Top-5 $A B$29, compared with SVD-ALS $A B$30 and naive SVD $A B$31. At 40% compression, the reported values are $A B$32, compared with $A B$33 for SVD-ALS and $A B$34 for SVD. The paper further reports that modality-aware tolerances $A B$35 and $A B$36 yield better trade-offs than a single global $A B$37, acknowledging cross-modality disparities.

The comparison with pruning baselines is restricted in the paper to WikiText-2 perplexity. For LLaMA-2-7B at 10%/30%/50% compression, PGSVD yields $A B$38, outperforming LLM-Pruner and ShortGPT, and is competitive with SliceGPT at lower ratios. For LLaMA-2-13B, it reports $A B$39, consistently better than LLM-Pruner and ShortGPT and competitive with SliceGPT at high compression. On reasoning accuracy at 20% compression for LLaMA-2-7B, PGSVD is reported to maintain balanced performance, with PIQA $A B$40 and ARC-E $A B$41, while SliceGPT drops sharply.

In comparative positioning, PGSVD is distinguished from truncated SVD/PCA, LoRA, and prior activation-aware methods such as Weighted SVD, ASVD, and SVD-LLM. The stated differences are threefold: a network loss bound linking $A B$42 to $A B$43; a bi-objective formulation with a Pareto-guided uniform tolerance that induces heterogeneous ranks with theoretical guarantees; and an ALS solver that avoids the numerical fragility of whitening-based methods and is more efficient than EVD-based implementations.

The paper also states the principal limitations. The Pareto guidance assumes homogeneous sensitivity and bounded, nonincreasing convex envelopes for layer SVD profiles. The loss bound is first-order and conservative, so extreme compressions may exceed its validity. The uniform $A B$44 rule is surrogate-optimal, whereas data-driven per-layer $A B$45 choices could further improve accuracy at the cost of robustness and complexity. Future directions explicitly identified include learning tolerance allocations from data, combining PGSVD with quantization and lightweight fine-tuning, refining the loss bounds through tighter higher-order analysis, and extending the approach to larger and more diverse VLM architectures.