LoRAct: Low-Rank Activation Compression

• Low-Rank Activation Compression (LoRAct) is a set of methods that approximates full activation tensors with low-rank factorization to cut memory and compute costs.
• The approach uses techniques like truncated SVD, random projection, and sampling-based orthogonal decomposition to efficiently compress activations during training or post-training.
• Empirical results show significant reduction in activation storage and FLOPs across models such as Transformers, ViTs, and CNNs, with minimal performance degradation.

Low-Rank Activation Compression (LoRAct) denotes a class of methods that exploit the empirical low-rank structure of neural activations to reduce memory, compute, or model size by replacing full activation tensors or activation-induced mappings with low-rank surrogates. In training and fine-tuning, LoRAct typically compresses intermediate activations saved for backpropagation; in post-training compression, it selects low-rank factors that minimize activation reconstruction or activation-induced output distortion rather than parameter distortion; and in continual or recursive settings it reuses fixed or dynamically tracked activation subspaces over time (Shi et al., 27 Sep 2025, Shamshoum et al., 2024, Apolinario et al., 25 Sep 2025, Çakar et al., 19 Apr 2026). The common premise is that activations, pre-activations, or attention-value outputs often exhibit rapid spectral decay even when weight matrices do not, so preserving the dominant activation subspace can retain task performance while substantially reducing activation storage, parameter count, or FLOPs (Chowdhury et al., 4 Jul 2025, Martínez, 29 Sep 2025).

1. Motivation and empirical basis

A central motivation for LoRAct is that activation storage is often the dominant memory term during training. For a Transformer or MLP-based model processing batch size $B$ and context length $L$, an activation tensor of size $B\times L\times d$ can be flattened to $A\in\mathbb R^{d\times n}$ with $n=B\cdot L$, and storing $A$ for backpropagation costs $O(d\cdot n)$ memory per layer. As $B$ or $L$ grows, the activation memory $B\cdot L\cdot d$ scales linearly and can dominate gradient and optimizer-state memory, quickly exceeding GPU budgets such as $L$0 GB (Shi et al., 27 Sep 2025).

The empirical justification is spectral. In the fine-tuning setting, singular-value plots show a rapidly decaying “long tail,” and to retain $L$1 of $L$2 energy one often needs only $L$3–$L$4 of the singular values even at large $L$5 or $L$6. This motivates approximating $L$7 as approximately rank-$L$8 with $L$9 (Shi et al., 27 Sep 2025). A related observation in LLM compression is that weight matrices are often not low-rank, whereas the activations they induce are markedly more compressible. One report explicitly contrasts standard weight factorization with the claim that “weights” of LLMs are often “not” low-rank while activations exhibit strong low-rank structure, motivating activation reconstruction rather than weight reconstruction (Chowdhury et al., 4 Jul 2025).

The same pattern appears outside standard feed-forward fine-tuning. In recursive architectures, storing activations across $B\times L\times d$0 unrolled steps has memory cost $B\times L\times d$1, yet the row-covariance of these activations is highly low-rank. LASER attributes this to weight-sharing, which concentrates iterative computation along a small number of dominant eigendirections, and reports that activations occupy an effectively linear, low-dimensional subspace during recursive unrolling (Çakar et al., 19 Apr 2026). This suggests that low-rank activation structure is not confined to a single architecture family, but recurs across LLMs, ViTs, CNNs, and weight-tied recursive models.

2. Mathematical formulations and optimization targets

The canonical LoRAct formulation approximates an activation matrix $B\times L\times d$2 by a rank-$B\times L\times d$3 factorization,

$B\times L\times d$4

with $B\times L\times d$5, $B\times L\times d$6, and $B\times L\times d$7. By Eckart–Young, the optimal rank-$B\times L\times d$8 approximation minimizes

$B\times L\times d$9

and in practice this is often written as

$A\in\mathbb R^{d\times n}$0

In online fine-tuning, this factorization reduces stored activation memory from $A\in\mathbb R^{d\times n}$1 to $A\in\mathbb R^{d\times n}$2, with compression ratio

$A\in\mathbb R^{d\times n}$3

The backward pass then reconstructs $A\in\mathbb R^{d\times n}$4 and computes gradients using $A\in\mathbb R^{d\times n}$5 in place of $A\in\mathbb R^{d\times n}$6 (Shi et al., 27 Sep 2025).

A second formulation compresses weights through activation-aware objectives. BALF begins from the observation that standard low-rank factorization minimizes parameter distortion,

$A\in\mathbb R^{d\times n}$7

but instead targets expected layer-output distortion,

$A\in\mathbb R^{d\times n}$8

Using activation whitening, BALF forms $A\in\mathbb R^{d\times n}$9, performs SVD in the whitened space, and shows that the empirical activation distortion at layer $n=B\cdot L$0 when truncating to rank $n=B\cdot L$1 is

$n=B\cdot L$2

This makes the optimization target explicitly activation-aware rather than parameter-aware (Martínez, 29 Sep 2025).

A third formulation introduces importance weighting over activation dimensions. IMPACT optimizes

$n=B\cdot L$3

then upper-bounds it by a weighted reconstruction objective

$n=B\cdot L$4

where the scaling vector $n=B\cdot L$5 is derived from the mean squared gradient $n=B\cdot L$6. The closed-form solution is given by the top $n=B\cdot L$7 eigenvectors of

$n=B\cdot L$8

so the retained subspace reflects both activation covariance and gradient sensitivity (Chowdhury et al., 4 Jul 2025).

Attention-specific formulations instantiate the same principle at a finer granularity. FLAT-LLM performs head-wise PCA on the attention value output $n=B\cdot L$9, truncates to $A$0, and absorbs the PCA basis into the value and output projections: $A$1 The induced reconstruction error is

$A$2

so rank selection directly controls activation-space distortion (Tian et al., 29 May 2025).

3. Online activation compression for training and fine-tuning

The most direct LoRAct training pipeline compresses activations online during the forward pass. In “Memory-Efficient Fine-Tuning via Low-Rank Activation Compression,” a layer computes its full-precision output, extracts the activation $A$3, immediately decomposes $A$4, discards $A$5, and stores only $A$6. On the backward pass it reconstructs $A$7 and computes gradients at $A$8. The reported memory reduction is therefore achieved without calibration data, because decomposition is performed online, per batch (Shi et al., 27 Sep 2025).

That work couples the online scheme with a sampling-based orthogonal decomposition algorithm. Instead of truncated SVD with cost $A$9 or RSVD with cost $O(d\cdot n)$0, it uniformly samples $O(d\cdot n)$1 rows of $O(d\cdot n)$2, alternates QR-based power iterations, and sets $O(d\cdot n)$3, $O(d\cdot n)$4. The reported cost is $O(d\cdot n)$5 with no Gaussian random matrix generation, and the paper states that the method offers improved computational efficiency and a tighter error bound compared to the widely used RSVD (Shi et al., 27 Sep 2025).

CompAct compresses activations differently. It flattens $O(d\cdot n)$6 to $O(d\cdot n)$7, samples a random projection $O(d\cdot n)$8 with entries $O(d\cdot n)$9, and stores

$B$0

plus the random seed rather than the full activation. It then computes compressed weight gradients $B$1 and maintains optimizer states in the $B$2 subspace rather than the $B$3 space. The method is designed so that the compute graph never stores the full activation tensor (Shamshoum et al., 2024).

LANCE replaces repeated per-step decompositions with a one-shot higher-order SVD on a calibration set. For a tensor activation $B$4, it computes mode-wise SVDs once, stores orthonormal factor matrices $B$5, and subsequently compresses activations to a core tensor $B$6 during runtime. The same fixed subspaces are then reused across minibatches, reducing both memory and computational overhead relative to iterative low-rank methods (Apolinario et al., 25 Sep 2025).

LASER extends the training-time perspective to recursive models. It stores stepwise coefficients $B$7 under a shared basis $B$8, updates the basis by one step of matrix-free power iteration,

$B$9

and uses a fidelity threshold to trigger either subspace expansion or hard reset. The method is explicitly dynamic: the basis is tracked during training rather than fixed once and for all (Çakar et al., 19 Apr 2026).

4. Calibration-based and fine-tuning-free compression

A major branch of LoRAct is post-training or fine-tuning-free model compression, where calibration data are used to discover activation subspaces and then absorb them into the model parameters. BALF exemplifies this design. Its pipeline collects second-moment matrices $L$0 from a small calibration set, forms whitening matrices by eigendecomposition, performs SVD in the whitened space, allocates ranks under a user-specified FLOPs or parameter budget via a Lagrangian-relaxed knapsack, and finally replaces each layer with two sequential low-rank sub-layers. The method is explicitly described as fine-tuning-free and inference-only (Martínez, 29 Sep 2025).

FLAT-LLM also uses calibration data, but at head granularity within multi-head attention. It computes covariance matrices $L$1 for each head, performs PCA, truncates to rank $L$2, and absorbs the basis into $L$3 and $L$4. Rank allocation is nonuniform across decoder blocks: importance scores are computed from cosine angles between input and output hidden states, a global budget $L$5 is imposed, and a greedy redistribution yields per-decoder ratios $L$6 with $L$7 (Tian et al., 29 May 2025).

IMPACT adds gradient sensitivity to calibration. During profiling it logs mean activations $L$8, activation covariance $L$9, and mean squared gradients $B\cdot L\cdot d$0 for each linear layer. Compression then forms

$B\cdot L\cdot d$1

where $B\cdot L\cdot d$2 is derived from $B\cdot L\cdot d$3, diagonalizes $B\cdot L\cdot d$4, selects rank by an energy threshold on $B\cdot L\cdot d$5, and replaces each original layer $B\cdot L\cdot d$6 with a two-layer factorization $B\cdot L\cdot d$7. This explicitly departs from uniform activation reconstruction by prioritizing activation dimensions with larger loss sensitivity (Chowdhury et al., 4 Jul 2025).

PGSVD and Swift-SVD further systematize activation-aware rank selection. PGSVD formulates low-rank compression as a bi-objective problem over parameter count and loss change, chooses ranks by a single tolerance $B\cdot L\cdot d$8, and refines factors via alternating least squares minimizing $B\cdot L\cdot d$9. Swift-SVD incrementally aggregates covariance of output activations, performs a single eigenvalue decomposition of $L$00, and yields the closed-form compressed weight

$L$01

Both methods are training-free, but differ in how they connect activation statistics to rank allocation and refinement (Solgi et al., 7 Oct 2025, Qi et al., 2 Apr 2026).

5. Theory, error bounds, and rank allocation

The theory of LoRAct spans approximation error, loss sensitivity, recovery guarantees, and training dynamics. For the online fine-tuning variant of LoRAct, the sampling-based orthogonal decomposition admits a deterministic spectral-error bound,

$L$02

and under uniform row sampling of size $L$03, the expected error satisfies

$L$04

The paper states that this matches or improves the classical RSVD bound without Gaussian-matrix constants (Shi et al., 27 Sep 2025).

PGSVD supplies a network-level loss bound. For perturbed weights $L$05, the paper bounds the scalar loss change by

$L$06

It then scalarizes the compression problem into minimizing $L$07 under a parameter budget and proves that, under homogeneous sensitivity and common convex envelopes, a uniform error tolerance $L$08 is optimal for the surrogate $L$09-allocation problem. As $L$10 varies, the induced rank vector traces surrogate Pareto-optimal size–error pairs (Solgi et al., 7 Oct 2025).

The training-time theory in “Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm” distinguishes sharply between linear and nonlinear operators. If activation compression is unbiased, then for a linear operator the compressed gradients remain unbiased,

$L$11

and the input gradient is exact: $L$12 For nonlinear operators, second-order terms induce bias, and upstream gradient errors can amplify through Jacobian products. The same work gives a model-level gradient variance bound and states that, under the standard $L$13-smoothness assumption, applying activation compression to all linear operators does not change the convergence rate (Wei et al., 2 May 2026).

A complementary post-training theory is given in “Theoretical Guarantees for Low-Rank Compression of Deep Neural Networks.” There the pre-activation matrix is $L$14, and compression seeks $L$15 with $L$16 such that $L$17. Three recovery theorems are stated: a strong low-rank theorem with sub-Gaussian noise, a weak approximate-low-rank theorem based on nuclear-norm and $L$18-norm constraints, and a nonlinear recovery result for ReLU observations. In all cases, the mean squared error

$L$19

vanishes as dimensions grow under the respective assumptions (Zhang et al., 4 Feb 2025).

6. Reported outcomes, trade-offs, and limitations

The empirical literature reports gains in several distinct regimes: activation-memory reduction during fine-tuning, training-time peak-memory reduction, fine-tuning-free structural compression, continual learning, and recursive computation. The reported outcomes are heterogeneous because the settings differ, but they are broadly consistent with the premise that activation low-rankness is exploitable in practice.

Approach	Reported outcome	Source
LoRAct fine-tuning	LLaMa2-7B, Alpaca & FLAN-v2: LoRA $L$20, $L$21 GB total, $L$22 GB act; LoRAct $L$23 $L$24, $L$25 GB total, $L$26 GB act	(Shi et al., 27 Sep 2025)
LoRAct language modeling	WikiText-2: LoRA PPL $L$27, $L$28 GB total, $L$29 GB act; LoRAct $L$30 PPL $L$31, $L$32 GB total, $L$33 GB act	(Shi et al., 27 Sep 2025)
LoRAct vision fine-tuning	ViT-B/16 on CIFAR-100: LoRA $L$34, $L$35 GB total, $L$36 GB act; LoRAct $L$37 $L$38, $L$39 GB total, $L$40 GB act	(Shi et al., 27 Sep 2025)
CompAct training	LLaMA-350M pretraining, $L$41: peak $L$42 GB $L$43 GB; PPL $L$44. RoBERTa-base fine-tuning: $L$45 GB $L$46 GB	(Shamshoum et al., 2024)
BALF inference compression	ResNeXt-101 on ImageNet: $L$47 FLOPs, $L$48 params, $L$49 top-1 pp; ViT-B/16: $L$50 FLOPs, $L$51 params, $L$52 top-1 pp	(Martínez, 29 Sep 2025)
FLAT-LLM	On Llama-2 7B with $L$53–$L$54 compression, $L$55–$L$56 speedups over dense, and $L$57–$L$58 faster than SliceGPT and SVD-LLM	(Tian et al., 29 May 2025)
IMPACT	Up to $L$59 greater model size reduction with accuracy comparable to state-of-the-art baselines; up to $L$60 faster and $L$61 less GPU memory compared to AFM	(Chowdhury et al., 4 Jul 2025)
LANCE and LASER	LANCE reports up to $L$62 activation-storage reduction; LASER reports $L$63 net activation-memory reduction	(Apolinario et al., 25 Sep 2025, Çakar et al., 19 Apr 2026)

Within the online fine-tuning setting, the most explicit trade-off is rank. The LoRAct paper states that smaller $L$64 yields more memory saving but eventually degrades performance; in vision tasks one needs $L$65 for acceptable accuracy, whereas in language tasks even $L$66 can match or slightly outperform (Shi et al., 27 Sep 2025). CompAct similarly reports a continuum: $L$67 gives $L$68–$L$69 memory saving for almost no effect on final accuracy, $L$70 gives $L$71 memory saving with $L$72 PPL drop, and $L$73 gives $L$74 memory saving with moderate perplexity degradation (Shamshoum et al., 2024).

Several recurrent misconceptions are explicitly addressed by the literature. First, low-rank compression of weights and low-rank compression of activations are not equivalent. Multiple papers argue that activations are more compressible than weights, and BALF, FLAT-LLM, IMPACT, PGSVD, and Swift-SVD all define their objectives in activation-aware terms rather than pure parameter distortion (Martínez, 29 Sep 2025, Tian et al., 29 May 2025, Chowdhury et al., 4 Jul 2025, Solgi et al., 7 Oct 2025, Qi et al., 2 Apr 2026). Second, minimizing uniform activation reconstruction error is not always sufficient: IMPACT shows that activation dimensions contribute unequally to model performance and therefore weights reconstruction by gradient sensitivity (Chowdhury et al., 4 Jul 2025). Third, compression is not uniformly safe across operator classes. The LLM training analysis reports that compressing only linear operators maintains stable convergence, whereas compressing nonlinear operators such as SiLU, RMSNorm, and Softmax can collapse or lose $L$75 accuracy (Wei et al., 2 May 2026).

The practical limitations are equally method-specific. Some methods require calibration data: BALF uses a small calibration set such as $L$76 K–$L$77 K images, FLAT-LLM uses $L$78 samples, and IMPACT recommends $L$79–$L$80 in-domain samples (Martínez, 29 Sep 2025, Tian et al., 29 May 2025, Chowdhury et al., 4 Jul 2025). Other methods avoid this requirement: the online LoRAct fine-tuning method states that no calibration data is needed because decomposition is done online per batch (Shi et al., 27 Sep 2025). LANCE notes that repeated low-rank decompositions introduce computational overhead and addresses this with one-shot HOSVD; LASER notes that subspaces drift and therefore adds fidelity-triggered reset mechanisms (Apolinario et al., 25 Sep 2025, Çakar et al., 19 Apr 2026). These differences indicate that “LoRAct” is best understood not as a single algorithm, but as a technical family organized around a shared empirical fact: neural activations are often substantially lower-rank than the ambient spaces in which they are represented.