SlimDiff: Activation-Guided Diffusion Slimming

Updated 12 July 2026

SlimDiff is an activation-informed structural compression framework that reduces both attention and feedforward dimensions in diffusion models without requiring any training or fine-tuning.
It reformulates diffusion model slimming as a spectral approximation problem by leveraging timestep-dependent activation covariances to dynamically prune functional groups such as Q–K, V–O, and FFN blocks.
Empirical results on Stable Diffusion show up to 35% acceleration and a 100M parameter reduction while maintaining near-parity in quality, even with only 500 calibration samples.

Searching arXiv for the specified paper and closely related “SlimDiff” usages. SlimDiff denotes an automated activation-informed structural compression framework for diffusion models that reduces both attention and feedforward dimensionalities while remaining entirely gradient-free and training-free. It reframes diffusion-model compression as a spectral approximation task: activation covariances collected across denoising timesteps define low-rank subspaces, these subspaces guide dynamic pruning under a fixed compression budget, and the resulting slimming is carried out over functional weight groups—query–key, value–output, and feedforward projections—rather than over isolated matrices. In the same broader name family, related works include SlimDiffSR for remote sensing super-resolution and SLIM-Diff for paired MRI image–mask synthesis, but the direct method named “SlimDiff” is the 2025 diffusion-model slimming framework introduced for Stable Diffusion-style U-Nets (Roy et al., 25 Sep 2025).

1. Problem setting and diffusion-specific motivation

Diffusion models are computationally expensive for two coupled reasons: they use large U-Nets with attention and feedforward layers, and they execute many iterative denoising steps, so compute, memory, and runtime are dominated by repeated high-dimensional operators. Existing efficiency techniques such as quantization, timestep reduction, and pruning can reduce compute, memory, or runtime, but the SlimDiff formulation argues that these methods are often bottlenecked by reliance on fine-tuning or retraining to recover quality. The stated reasons are threefold: they ignore the coupled functional structure of attention and feedforward blocks, neglect timestep-dependent activation geometry, and suffer sequential error accumulation across the denoising trajectory (Roy et al., 25 Sep 2025).

The diffusion context is made explicit through the standard forward process

$x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\qquad \epsilon\sim\mathcal{N}(0,I),$

with $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ . SlimDiff collects activations $a_{l,t}$ inside the U-Net at multiple denoising timesteps $t\in\{1,\dots,T\}$ . The central empirical premise is that activation geometry changes with timestep: activations are described as near-isotropic early and highly anisotropic late. That non-uniformity is why the method treats covariance estimation as timestep-aware rather than static. The paper further notes that, under $v$ -parameterization, inference-time U-Net activations are identical modulo affine transformations, so the activation-based compression viewpoint is not tied to a single reverse-process parameterization (Roy et al., 25 Sep 2025).

A recurring misconception is to treat diffusion-model slimming as equivalent to ordinary low-rank compression of individual matrices. SlimDiff rejects that framing. Its compression target is the denoising trajectory, not just a collection of disconnected linear layers, and its approximations are chosen to reduce tail energy under activation-aware covariance metrics so that error does not compound destructively across timesteps.

2. Activation-informed spectral formulation

For a module $l$ at timestep $t$ , SlimDiff defines activations $a_{l,t}\in\mathbb{R}^{d_l}$ and estimates an empirical covariance

$\widehat{C}_{l,t}=\frac{1}{N_t}\sum_{n=1}^{N_t} a_{l,t}^{(n)}a_{l,t}^{(n)\top}.$

Because calibration sample counts are modest, the method uses a regularized covariance $\widetilde{C}_{l,t}$ , for example by shrinkage or diagonal loading. These timestep-wise covariances are then merged into a module-level mixture

$\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 0

with whitening root $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 1. The weights $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 2 are adaptive and fidelity-aware rather than uniform (Roy et al., 25 Sep 2025).

The weighting and global allocation mechanism are anchored by the Spectral Influence Score, defined for weight matrix $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 3 and pre-activation covariance $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 4 as

$\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 5

The paper characterizes this quantity as scale/variance invariant and uses it both to set timestep mixture weights and to aggregate per-module importance. In effect, SlimDiff converts compression into a covariance-weighted spectral approximation problem: module directions that are active under the denoising trajectory receive preferential retention, whereas inactive or weakly aligned directions are pruned first.

This formulation is designed to address sequential denoising error accumulation. The claim is not merely that covariances summarize data statistics, but that they summarize which directions are active at which timesteps and therefore determine which low-rank subspaces are safe to retain under repeated application of the compressed denoiser.

3. Functional-group decompositions in attention and FFN blocks

SlimDiff compresses attention and feedforward modules through functional groups instead of through isolated matrices. In standard attention, with input features $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 6,

$\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 7

and the output is

$\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 8

The method groups query–key and value–output operators jointly because the effective computations depend on the products $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 9 and $a_{l,t}$ 0 under the input metric. In GeGLU-style FFN blocks, with up-projections $a_{l,t}$ 1, $a_{l,t}$ 2 and down-projection $a_{l,t}$ 3,

$a_{l,t}$ 4

and the elementwise gate breaks simple linear factorization, which motivates a structured approximation strategy (Roy et al., 25 Sep 2025).

The reconstruction objective is defined over the functional mapping itself:

$a_{l,t}$ 5

where $a_{l,t}$ 6 is the Q–K bilinear map, the V–O linear composite, or the FFN gated map. Closed-form solutions are obtained by aligning decompositions with covariance-weighted metrics through whitening and Nyström approximations.

For Q–K, SlimDiff defines the whitened composite

$a_{l,t}$ 7

computes its SVD $a_{l,t}$ 8, truncates to rank $a_{l,t}$ 9, and unwhitens via

$t\in\{1,\dots,T\}$ 0

The stated error bound is

$t\in\{1,\dots,T\}$ 1

For V–O, it similarly defines

$t\in\{1,\dots,T\}$ 2

performs WSVD, and sets

$t\in\{1,\dots,T\}$ 3

with exact covariance-metric error

$t\in\{1,\dots,T\}$ 4

For FFN, SlimDiff uses a Nyström approximation with a shared column selector $t\in\{1,\dots,T\}$ 5 applied to both up-projections,

$t\in\{1,\dots,T\}$ 6

and then computes the closed-form optimal down-projection

$t\in\{1,\dots,T\}$ 7

where

$t\in\{1,\dots,T\}$ 8

The reported error bound is

$t\in\{1,\dots,T\}$ 9

and column selection is performed with CPQR, which the paper describes as yielding stable low-rank subspaces for the gated FFN (Roy et al., 25 Sep 2025).

4. Global budget allocation and the training-free pipeline

SlimDiff does not compress all blocks uniformly. It formulates budgeted rank allocation as

$v$ 0

where blocks $v$ 1 have ranks $v$ 2 and costs $v$ 3. Because direct estimation of $v$ 4 for every block is expensive, the method uses spectral influence scores aggregated over timesteps to drive retention. The convex surrogate introduces sparsity variables $v$ 5 and retention fractions $v$ 6:

$v$ 7

The resulting closed-form solution is

$v$ 8

These retention fractions are then mapped to ranks through effective widths such as attention head dimensions $v$ 9, $l$ 0 or FFN width $l$ 1, followed by hardware-friendly rounding, for example to multiples of $l$ 2, with a minimum $l$ 3 for stability. The average sparsity $l$ 4 is determined by bisection to meet the exact budget $l$ 5 (Roy et al., 25 Sep 2025).

The training-free pipeline is explicit. SlimDiff first builds SlimSet, approximately $l$ 6 prompts, using semantic coreset selection with CLIP embeddings, geometric median, stratification by distinctiveness, farthest-point sampling, and deduplication. It then runs denoising for chosen timesteps to record activations at attention and FFN sites, computes timestep-wise covariances and their regularized versions, derives spectral influence scores and whitening roots, applies CPQR/Nyström to FFNs and WSVD to Q–K and V–O groups, allocates ranks globally under the budget, rewrites weights to reduced shapes, and finally generates and evaluates without any fine-tuning. Complexity is reported as practical with approximately $l$ 7 prompts and tens of timesteps; SVDs are parallelizable across heads and layers; and FFN CPQR/Nyström is approximately $l$ 8 for column subset selection with $l$ 9.

The method’s structural effect is a direct slimming of attention and feedforward dimensions. For attention, Q–K compression yields new per-head dimensions $t$ 0 and $t$ 1, and V–O compression yields new value/head dimension $t$ 2. For FFNs, the hidden width $t$ 3 is reduced to $t$ 4 through the shared selector and closed-form down-projection update.

5. Empirical evidence on Stable Diffusion

The reported experiments target Stable Diffusion v1.4 and v1.5 latent U-Nets, with attention and FFN modules compressed, and evaluate on MS-COCO 2014 val, LAION Aesthetics, ImageRewardDB, PartiPrompts, and human preference benchmarks HPS v2.1, ImageReward, and Pick-a-Pic v1. Images are generated at $t$ 5 with $t$ 6 denoising steps, CFG $t$ 7, and default latent resolution. In this setting, SlimDiff is reported to achieve generation quality on par with uncompressed models while requiring only about $t$ 8 calibration samples—over $t$ 9 fewer than prior work—and the abstract states that it achieves up to $a_{l,t}\in\mathbb{R}^{d_l}$ 0 acceleration and approximately $a_{l,t}\in\mathbb{R}^{d_l}$ 1M parameter reduction without any backpropagation (Roy et al., 25 Sep 2025).

On COCO, the paper reports that SlimDiff for Stable Diffusion v1.5 obtains FID $a_{l,t}\in\mathbb{R}^{d_l}$ 2 versus $a_{l,t}\in\mathbb{R}^{d_l}$ 3 for the uncompressed model and CLIPScore $a_{l,t}\in\mathbb{R}^{d_l}$ 4 versus $a_{l,t}\in\mathbb{R}^{d_l}$ 5; for v1.4 it reports FID $a_{l,t}\in\mathbb{R}^{d_l}$ 6 versus $a_{l,t}\in\mathbb{R}^{d_l}$ 7 and CLIPScore $a_{l,t}\in\mathbb{R}^{d_l}$ 8 versus $a_{l,t}\in\mathbb{R}^{d_l}$ 9. Human preference metrics are similarly close: HPS v2.1 $\widehat{C}_{l,t}=\frac{1}{N_t}\sum_{n=1}^{N_t} a_{l,t}^{(n)}a_{l,t}^{(n)\top}.$ 0 versus $\widehat{C}_{l,t}=\frac{1}{N_t}\sum_{n=1}^{N_t} a_{l,t}^{(n)}a_{l,t}^{(n)\top}.$ 1, ImageReward $\widehat{C}_{l,t}=\frac{1}{N_t}\sum_{n=1}^{N_t} a_{l,t}^{(n)}a_{l,t}^{(n)\top}.$ 2 versus $\widehat{C}_{l,t}=\frac{1}{N_t}\sum_{n=1}^{N_t} a_{l,t}^{(n)}a_{l,t}^{(n)\top}.$ 3, and Pick-a-Pic $\widehat{C}_{l,t}=\frac{1}{N_t}\sum_{n=1}^{N_t} a_{l,t}^{(n)}a_{l,t}^{(n)\top}.$ 4 versus $\widehat{C}_{l,t}=\frac{1}{N_t}\sum_{n=1}^{N_t} a_{l,t}^{(n)}a_{l,t}^{(n)\top}.$ 5. The paper also states that visual samples preserve semantics and style across diverse prompts.

The efficiency numbers are given at several levels. MACs per U-Net forward are $\widehat{C}_{l,t}=\frac{1}{N_t}\sum_{n=1}^{N_t} a_{l,t}^{(n)}a_{l,t}^{(n)\top}.$ 6G versus $\widehat{C}_{l,t}=\frac{1}{N_t}\sum_{n=1}^{N_t} a_{l,t}^{(n)}a_{l,t}^{(n)\top}.$ 7G; for $\widehat{C}_{l,t}=\frac{1}{N_t}\sum_{n=1}^{N_t} a_{l,t}^{(n)}a_{l,t}^{(n)\top}.$ 8-step generation they are $\widehat{C}_{l,t}=\frac{1}{N_t}\sum_{n=1}^{N_t} a_{l,t}^{(n)}a_{l,t}^{(n)\top}.$ 9T versus $\widetilde{C}_{l,t}$ 0T. Reported latency is $\widetilde{C}_{l,t}$ 1s versus $\widetilde{C}_{l,t}$ 2s on GPU and $\widetilde{C}_{l,t}$ 3s versus $\widetilde{C}_{l,t}$ 4s on CPU, with batch size $\widetilde{C}_{l,t}$ 5 and fp16/fp32. Representative compressed checkpoints are around $\widetilde{C}_{l,t}$ 6B versus $\widetilde{C}_{l,t}$ 7B parameters, while the abstract summarizes this as an approximately $\widetilde{C}_{l,t}$ 8M reduction. Compression is reported to run on a single A100 80GB, and the setup table records a training-free pipeline runtime of “4 A100 days,” contrasted with thousands of A100-days for model training.

The ablations clarify the design choices. A SlimSet size of $\widetilde{C}_{l,t}$ 9 prompts already matches quality obtained with thousands, indicating stable activation subspaces. Spectral influence weighting gives the best FID, reported as $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 00, compared with uniform or diversity-only timestep weighting. Joint compression of self-attention, cross-attention, and FFN yields the best trade-off, and the paper identifies FFN compression as the most delicate component because of gated coupling.

6. Integration, limitations, and conceptual significance

SlimDiff is presented as implementable in Diffusers and PyTorch by wrapping attention heads and FFN blocks with compressors and collecting activations using the existing sampler at a chosen set of timesteps, for example uniformly over a $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 01-step schedule. The practical guidance is specific: calibration set size $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 02 prompts; timesteps matched to the inference schedule; diagonal loading for covariance regularization; condition-number control during whitening; rank rounding to multiples of $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 03; $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 04; and a total parameter or MAC budget $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 05 regulated by the softmax temperature $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 06 (Roy et al., 25 Sep 2025).

The limits are also explicit. Extreme compression budgets, meaning very small ranks, can degrade quality, especially in FFN because of gating and in cross-attention where text alignment is crucial. The approach assumes that SlimSet covariances are representative of deployment, so large domain shifts may reduce fidelity. Architectures with very different attention or FFN forms may require adapting whitening or selection operators. Although the method is training-free, covariance estimation and SVD computation still introduce offline compute, even if the paper describes that cost as modest with approximately $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 07 prompts.

Conceptually, SlimDiff occupies a distinct position among diffusion efficiency methods. It is not a reduced-step sampler, not a quantization method, and not an inference-time token-merging scheme. The slimming is permanent, structural, and complementary to orthogonal accelerations such as token merging or timestep reduction. The paper also frames it as “the first closed-form, activation-guided structural compression of DMs that is entirely training-free,” emphasizing that its contribution is both practical and theoretical: the compression rules are closed-form, gradient-free, and derived under covariance-weighted reconstruction criteria rather than recovered by post-compression fine-tuning (Roy et al., 25 Sep 2025).

The label “SlimDiff” is not monosemous across recent diffusion literature. In addition to the training-free compression method, closely related names are used for diffusion distillation, medical synthesis, diffusion-assisted compression, discrete simplex diffusion, and caching-based acceleration. The table summarizes the principal distinctions already present in the cited record.

Name	Domain	Core mechanism
SlimDiff	General diffusion-model slimming	Activation-informed structural compression of attention and FFN
SlimDiffSR	Remote sensing image super-resolution	Single-step teacher–student distillation with uncertainty-guided timestep assignment, RS-aware pruning, FSC, DSC, and QGAM
SLIM	Image compression for machines	RoI-focused latent compression plus text-conditioned latent diffusion enhancement
SLIM-Diff	Epilepsy FLAIR MRI	Shared latent image–mask diffusion with tunable $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 08 loss
ShortListing Model (SLM)	Discrete variable generation	Streamlined simplex-based diffusion via progressive candidate pruning
Spectral Diffusion (SD)	Lightweight image synthesis	Wavelet gating and spectrum-aware distillation for slim latent diffusion
X-Slim	Diffusion inference acceleration	Training-free multi-level caching across timesteps, blocks, and tokens

SlimDiffSR turns diffusion super-resolution into a single-step, uncertainty-aware teacher distilled into a slim student and reports up to $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 09 inference acceleration and a $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 10 parameter reduction against multi-step diffusion baselines (Wang et al., 4 May 2026). SLIM targets machine vision compression, focusing bits on RoI latents and using a pretrained latent diffusion denoiser conditioned on RoI-focused captions; it reports BD-Rate reductions versus MPA of $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 11 for ResNet50 and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 12 for ConvNeXt-T (Lee et al., 20 Dec 2025). SLIM-Diff, by contrast, is a compact joint diffusion model for paired FLAIR MRI slices and lesion masks in which $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 13-prediction is reported as consistently strongest, while $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 14 improves image fidelity and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 15 better preserves lesion mask morphology (Pascual-González et al., 3 Feb 2026). The ShortListing Model, although summarized in the provided material as a “SlimDiff” approach for discrete variables, officially names the method SLM and operates on simplex centroids with progressive candidate pruning (Song et al., 24 Aug 2025). “Diffusion Probabilistic Model Made Slim” introduces Spectral Diffusion rather than a method literally named SlimDiff, using wavelet gating and spectrum-aware distillation to obtain $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 16– $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 17 computational complexity reduction relative to latent diffusion models (Yang et al., 2022). “No Cache Left Idle” likewise uses X-Slim, a caching controller rather than structural compression, and reports up to $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 18 speedup on FLUX.1-dev and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ 19 on HunyuanVideo (Wen et al., 14 Dec 2025).

Taken together, these usages show that “SlimDiff” and its cognates designate a family of efforts to reduce diffusion cost, but they do so through substantially different levers: structural low-rank compression, distillation, semantic compression, compact joint modeling, simplex-state redesign, spectral architectural modification, or cache-based inference control. Within that family, the 2025 SlimDiff method is distinguished by its activation-guided, closed-form, training-free compression of functional groups under timestep-aware covariance geometry.