Activation- and Influence-Aware Ranks (AIR)

• The paper introduces a novel integration of backward influence with activation-aware SVD to guide low-rank approximations in transformer compression.
• It employs a closed-form alternating least squares sweep with a monotone descent guarantee to optimize model performance while reducing parameters.
• Empirical results on LLaMA-7B demonstrate over 18% improvement in perplexity at 60% parameter retention, showing AIR's advantage over traditional methods.

Activation- and Influence-Aware Ranks (AIR) is an SVD-based LLM compression framework that guides each weight matrix’s low-rank approximation with a backward-signal influence metric while starting from the activation-aware optimum of SVD-LLM(W) (Harder et al., 18 Jun 2026). It addresses post-training compression of pretrained transformer weight matrices by combining forward activation statistics with an element-wise influence matrix, then applying a single closed-form alternating least squares (ALS) sweep under a monotone-descent guarantee. AIR is layer-local, calibration-based, and intended to preserve function rather than merely minimize weight reconstruction error; in the reported experiments it improves perplexity over SVD-LLM(W) by more than $18\%$ at $60\%$ parameter retention and composes orthogonally with end-to-end methods such as LoRA (Harder et al., 18 Jun 2026).

1. Compression setting and activation-aware foundation

AIR operates in the standard post-training low-rank compression setting for a pretrained transformer weight matrix

$\mathbf{W}\in\mathbb{R}^{m\times n},$

which is replaced by a lower-rank approximation so that inference uses fewer parameters, fewer FLOPs, and less memory. The low-rank factorization is implemented as

$\mathbf{W}\approx \mathbf{U}_k\mathbf{V}_k^\top,$

or, in singular-value form,

$$\mathbf{W}_k=\mathbf{U}_k\boldsymbol{\Sigma}_k\mathbf{V}_k^\top,$$

with inference computed as

$$\mathbf{y}=\mathbf{U}_k(\mathbf{V}_k^\top\mathbf{x}).$$

This reduces multiply-accumulate cost from $mn$ to $k(m+n)$ (Harder et al., 18 Jun 2026).

The framework begins from the observation that plain SVD is function-agnostic: it minimizes weight reconstruction error, not preservation of the model’s input-output mapping. AIR therefore inherits the activation-aware construction of SVD-LLM(W). Given calibration activations $\mathbf{X}$, the layer-local objective is to preserve $\mathbf{W}\mathbf{X}$ rather than $60\%$0 itself. SVD-LLM(W) profiles the layer by constructing

$60\%$1

which yields the activation-aware objective

$60\%$2

This objective has the truncated-SVD optimum of $60\%$3, so AIR begins from the exact activation-aware solution rather than replacing it with a different criterion (Harder et al., 18 Jun 2026).

This design places AIR within the activation-aware branch of low-rank compression, but only as a starting point. The paper’s central critique is that activation-aware methods remain forward-local: they preserve directions that are strong or common in calibration activations, yet are agnostic to whether those preserved activations influence the final prediction (Harder et al., 18 Jun 2026).

2. Influence-aware objective and backward signal integration

AIR augments activation-aware reconstruction with an element-wise influence matrix

$60\%$4

whose entries estimate how much reconstruction error at each weight element matters to model behavior. The resulting objective is

$60\%$5

or equivalently

$60\%$6

Here $60\%$7 controls the strength of influence reweighting. At $60\%$8, AIR reduces exactly to the activation-aware objective of SVD-LLM(W) (Harder et al., 18 Jun 2026).

The additive anchor $60\%$9 is essential. AIR does not replace activation-aware compression by pure influence weighting; it perturbs the activation-aware metric so that low-influence elements are not discarded outright. In the paper’s formulation, AIR redistributes reconstruction error away from the most influential weight elements while retaining the forward activation structure supplied by SVD-LLM(W) (Harder et al., 18 Jun 2026).

The influence term is motivated through a Taylor expansion of task loss around the pretrained weight matrix,

$\mathbf{W}\in\mathbb{R}^{m\times n},$0

which AIR uses as a sensitivity probe rather than an optimization target. The paper explicitly allows several backward influence signals: Weight$\mathbf{W}\in\mathbb{R}^{m\times n},$1Gradient, diagonal Fisher, and LRP-$\mathbf{W}\in\mathbb{R}^{m\times n},$2. Its default is AttnLRP with $\mathbf{W}\in\mathbb{R}^{m\times n},$3, accumulated over calibration data by magnitude,

$\mathbf{W}\in\mathbb{R}^{m\times n},$4

and then normalized to unit mean per layer (Harder et al., 18 Jun 2026).

This suggests that AIR’s notion of “influence” is not architectural or topological; it is a local, element-wise backward sensitivity signal that modulates where approximation error is allowed to reside.

3. Closed-form ALS sweep and monotone descent

AIR solves the weighted low-rank problem by initializing at the activation-aware optimum,

$\mathbf{W}\in\mathbb{R}^{m\times n},$5

then performing a single rank-wise ALS sweep in profiled space. Writing

$\mathbf{W}\in\mathbb{R}^{m\times n},$6

the sweep isolates each rank-1 component through the residual

$\mathbf{W}\in\mathbb{R}^{m\times n},$7

AIR then updates the $\mathbf{W}\in\mathbb{R}^{m\times n},$8-th factor while holding the others fixed. The paper gives a closed-form update for $\mathbf{W}\in\mathbb{R}^{m\times n},$9,

$\mathbf{W}\approx \mathbf{U}_k\mathbf{V}_k^\top,$0

followed by a closed-form update for the unnormalized left factor,

$\mathbf{W}\approx \mathbf{U}_k\mathbf{V}_k^\top,$1

and normalization

$\mathbf{W}\approx \mathbf{U}_k\mathbf{V}_k^\top,$2

The sweep proceeds in backward rank order,

$\mathbf{W}\approx \mathbf{U}_k\mathbf{V}_k^\top,$3

because the paper reports that reverse order better protects dominant low-index singular components (Harder et al., 18 Jun 2026).

The main algorithmic guarantee is monotone descent: starting from $\mathbf{W}\approx \mathbf{U}_k\mathbf{V}_k^\top,$4 and applying the closed-form coordinate updates, $\mathbf{W}\approx \mathbf{U}_k\mathbf{V}_k^\top,$5 is non-increasing at every step. The guarantee is coordinate-descent in nature rather than a global optimality theorem for the full weighted low-rank problem. The paper is explicit that the full objective remains non-convex, but each subproblem is a strictly convex quadratic in the coordinate being updated (Harder et al., 18 Jun 2026).

After refinement, AIR maps factors back to native weight space by absorbing singular values symmetrically,

$\mathbf{W}\approx \mathbf{U}_k\mathbf{V}_k^\top,$6

so that inference uses

$\mathbf{W}\approx \mathbf{U}_k\mathbf{V}_k^\top,$7

Empirically, one sweep is reported as best: on LLaMA-7B at $\mathbf{W}\approx \mathbf{U}_k\mathbf{V}_k^\top,$8 parameter rate, one sweep yields WikiText-2 perplexity $\mathbf{W}\approx \mathbf{U}_k\mathbf{V}_k^\top,$9, whereas two and ten sweeps both yield $$\mathbf{W}_k=\mathbf{U}_k\boldsymbol{\Sigma}_k\mathbf{V}_k^\top,$$0, even though the surrogate objective continues to decrease (Harder et al., 18 Jun 2026). The paper interprets this as mild overfitting to the calibration proxy after the first correction sweep.

4. Empirical behavior, efficiency, and deployment characteristics

The main reported gains are on LLaMA-7B without finetuning, where AIR consistently improves over SVD-LLM(W) across compression rates (Harder et al., 18 Jun 2026).

Parameter retention	SVD-LLM(W) WikiText-2 PPL	AIR WikiText-2 PPL
80%	7.87	7.51
60%	13.81	11.27
40%	63.83	42.52
20%	854	472

The same trend is reported on C4, with the gap widening at stronger compression. At $$\mathbf{W}_k=\mathbf{U}_k\boldsymbol{\Sigma}_k\mathbf{V}_k^\top,$$1 parameter retention on LLaMA-7B, AIR also outperforms plain SVD, FWSVD, and ASVD by a wide margin, and the paper reports that AIR alone can exceed ACIP while AIR+LoRA performs better still (Harder et al., 18 Jun 2026).

Cross-model results at $$\mathbf{W}_k=\mathbf{U}_k\boldsymbol{\Sigma}_k\mathbf{V}_k^\top,$$2 parameter rate preserve the same ordering. AIR is reported as best on Mistral-7B, Vicuna-7B, TinyLLaMA 1.1B, LLaMA 2-7B, LLaMA 3-8B, and LLaMA 30B, with relative WikiText-2 improvements over SVD-LLM(W) ranging from $$\mathbf{W}_k=\mathbf{U}_k\boldsymbol{\Sigma}_k\mathbf{V}_k^\top,$$3 on LLaMA 30B to $$\mathbf{W}_k=\mathbf{U}_k\boldsymbol{\Sigma}_k\mathbf{V}_k^\top,$$4 on LLaMA 3-8B (Harder et al., 18 Jun 2026).

The influence-signal ablation is notable because it isolates the integration mechanism. At $$\mathbf{W}_k=\mathbf{U}_k\boldsymbol{\Sigma}_k\mathbf{V}_k^\top,$$5 parameter rate, the paper reports Ones $$\mathbf{W}_k=\mathbf{U}_k\boldsymbol{\Sigma}_k\mathbf{V}_k^\top,$$6, Weight magnitude $$\mathbf{W}_k=\mathbf{U}_k\boldsymbol{\Sigma}_k\mathbf{V}_k^\top,$$7, Weight$$\mathbf{W}_k=\mathbf{U}_k\boldsymbol{\Sigma}_k\mathbf{V}_k^\top,$$8Gradient $$\mathbf{W}_k=\mathbf{U}_k\boldsymbol{\Sigma}_k\mathbf{V}_k^\top,$$9, Fisher $$\mathbf{y}=\mathbf{U}_k(\mathbf{V}_k^\top\mathbf{x}).$$0, and LRP-$$\mathbf{y}=\mathbf{U}_k(\mathbf{V}_k^\top\mathbf{x}).$$1 $$\mathbf{y}=\mathbf{U}_k(\mathbf{V}_k^\top\mathbf{x}).$$2. This suggests that AIR’s gain is driven less by a unique influence metric than by the element-wise integration scheme, provided the signal is principled (Harder et al., 18 Jun 2026).

The reported calibration setting uses 256 WikiText-2 samples of 2048 tokens each, but AIR is also presented as data-efficient: it matches SVD-LLM(W)’s quality with about $$\mathbf{y}=\mathbf{U}_k(\mathbf{V}_k^\top\mathbf{x}).$$3 less calibration data. Runtime is modest for a post-training method: on LLaMA-7B the full AIR procedure, including profiling and influence accumulation, is reported at about 12 minutes on an A100, with about 1.5 seconds per layer for the ALS refinement itself (Harder et al., 18 Jun 2026).

The systems section argues that parameter savings do not automatically become latency savings. AIR therefore incorporates implementation changes for low-rank attention inference, including RoPE pre-application for key caching, fused low-rank value caching, and pre-allocated cache buffers. With those optimizations, the paper reports that at $$\mathbf{y}=\mathbf{U}_k(\mathbf{V}_k^\top\mathbf{x}).$$4 parameter retention the static ratios are approximately $$\mathbf{y}=\mathbf{U}_k(\mathbf{V}_k^\top\mathbf{x}).$$5 parameters, $$\mathbf{y}=\mathbf{U}_k(\mathbf{V}_k^\top\mathbf{x}).$$6 FLOPs, and $$\mathbf{y}=\mathbf{U}_k(\mathbf{V}_k^\top\mathbf{x}).$$7 memory, with peak GPU memory around $$\mathbf{y}=\mathbf{U}_k(\mathbf{V}_k^\top\mathbf{x}).$$8 of base and per-token latency around $$\mathbf{y}=\mathbf{U}_k(\mathbf{V}_k^\top\mathbf{x}).$$9 of base at the default benchmark point. In more aggressive memory-pressure regimes, latency is reported as low as $mn$0 of base at batch size 128 and sequence length 256 (Harder et al., 18 Jun 2026).

5. Relation to prior methods and broader AIR-style context

AIR is most directly positioned against four compression baselines. Plain SVD minimizes $mn$1 and is function-agnostic. SVD-LLM(W) introduces activation-awareness through profiled-space reconstruction. FWSVD brings backward information but lacks activation-awareness and uses row-wise Fisher weighting rather than AIR’s element-wise weighting. ACIP uses backward information through end-to-end optimization, whereas AIR is layer-local and closed-form (Harder et al., 18 Jun 2026).

A plausible implication is that AIR occupies the specific niche where activation-aware local optima are retained, but backward sensitivity is used to reallocate error within that optimum’s neighborhood. That niche differs from activation-only reconstruction methods that rank directions by activation statistics alone. For example, IMPACT constructs an importance-weighted activation covariance matrix and keeps top eigen-directions in activation space, combining activation structure with gradient sensitivity, but it is not built on SVD-LLM(W)-style profiled weight reconstruction or AIR’s ALS refinement (Chowdhury et al., 4 Jul 2025).

The phrase “activation- and influence-aware” also has a broader methodological history outside LLM compression. In social-network diffusion and influence maximization, intrinsic activation plus propagation was formalized as a joint ranking problem in work on nodal intrinsic activation (Sathanur et al., 2017); self-consistent ranking under rank-conditioned marginal influence appeared in IMRank (Cheng et al., 2014); analytic activation-probability-based greedy scoring under Independent Cascade was developed in AAPC/EAAPC (Adineh et al., 2020); and threshold-sensitive FLTR studied how changing activation resistance alters diffusion-based ranks (Blesa et al., 2022). AIR, in the compression sense, does not reuse those diffusion objectives, but it shares the same structural idea: ranking or preserving model components should depend jointly on activation and on downstream effect.

The acronym AIR is also overloaded. “Attention Influence for Reasoning” uses AIR for post-training data selection based on counterfactual loss divergence under disabled retrieval heads, rather than low-rank compression (Liu et al., 15 Dec 2025). This naming overlap is terminological rather than methodological.

6. Scope, assumptions, and limitations

AIR is a post-training, calibration-based, layer-local compressor. It is not data-free, since it requires forward profiling statistics and backward influence accumulation on calibration samples. It is also not globally optimized across layers: the paper uses a uniform per-layer parameter rate

$mn$2

and explicitly leaves dynamic rank allocation for future work (Harder et al., 18 Jun 2026).

The paper states several scope limitations. Scaling to $mn$3B+ models remains open. Encoder-decoder and MoE architectures are not studied. AIR’s empirical speedups are contingent on systems-level implementation details rather than guaranteed by factorization alone. More ALS sweeps can overfit the calibration proxy and slightly worsen held-out perplexity even while decreasing $mn$4. Under extreme compression, AIR still degrades substantially, although the paper reports that degradation is more graceful than with activation-only baselines (Harder et al., 18 Jun 2026).

The choice of $mn$5 also reflects a core assumption. At $mn$6, AIR collapses to SVD-LLM(W). Performance improves as $mn$7 increases toward the reported optimum near $mn$8, then worsens when influence weighting becomes too dominant. This indicates that AIR relies on a balance rather than a replacement principle: activation-aware reconstruction remains the base geometry, and influence-awareness only redistributes the residual budget (Harder et al., 18 Jun 2026).

Taken together, these constraints define AIR as a function-preserving SVD compression framework rather than a universal rank-allocation theory. Its contribution is specific and technically narrow: to begin at the activation-aware optimum, inject backward influence element-wise, and obtain a low-cost refinement with a monotone-descent guarantee that improves perplexity, reduces calibration-data requirements, and can convert parameter savings into actual FLOP, peak-memory, and latency gains (Harder et al., 18 Jun 2026).