Dynamic Slimming: Adaptive Model Compression

• Dynamic slimming is a family of techniques that conditionally adjusts model capacity through methods such as channel pruning, staged compression, and input-dependent routing.
• It enhances computational efficiency by aligning network resources with input complexity and runtime constraints, ensuring optimal performance under varying budgets.
• Applications span semantic segmentation, video compression, speech enhancement, and LLM fine-tuning, highlighting its versatility across diverse domains.

Searching arXiv for relevant papers on dynamic slimming and related slimmable/adaptive-compute methods. Dynamic slimming denotes a family of methods that reduce a model’s effective capacity conditionally, progressively, or jointly with other optimization targets, rather than fixing a single compact architecture in advance. Across the literature, the term covers several distinct but related mechanisms: channel-width adaptation in nested subnetworks, staged pruning-and-distillation schedules, input-dependent routing over widths or experts, context-aware control of sensing and perception budgets, and data-conditioned co-optimization of inputs and networks. The unifying theme is that capacity allocation is not static: it is selected, learned, or adjusted in response to input complexity, runtime constraints, training trajectory, or deployment context (Li et al., 2021, Li et al., 2021, Chen et al., 2023, Haberer et al., 21 May 2026).

1. Genealogy and conceptual scope

The modern lineage of dynamic slimming is closely tied to channel-based structural compression. “Network Slimming” formalized a simple channel-pruning scheme in which BatchNorm scale parameters are driven toward zero by an $L_1$ penalty, after which channels with small scaling factors are pruned and the remaining dense network is fine-tuned (Liu et al., 2017). Its objective is

$$L = \sum_{(x, y)} l(f(x, W), y) + \lambda \sum_{\gamma \in \Gamma} g(\gamma),$$

with $g(s)=|s|$, where $\Gamma$ is the set of channel scaling factors (Liu et al., 2017). Although this procedure is not dynamic at inference time, it established the width axis as a primary locus for efficient architecture adaptation.

Subsequent work split into at least three branches. One branch retained the nested-width supernetwork idea and made width selectable at inference, as in slimmable semantic segmentation, slimmable video compression, and width-adaptive ConvNeXt models (Xue et al., 2022, Liu et al., 2022, Haberer et al., 21 May 2026). A second branch introduced input-conditioned control, in which a gate or router predicts the width, subnetwork, or expert subset per sample or per context, as in Dynamic Slimmable Network, Dynamic Slimmable Denoising Network, SLIM for LLM fine-tuning, adaptive speech enhancement, and navigation systems such as NaviSlim and CADENCE (Li et al., 2021, Jiang et al., 2021, Han et al., 2024, Miccini et al., 7 Jul 2025, Johnsen et al., 2024, Johnsen et al., 8 Apr 2026). A third branch generalized slimming beyond width selection to staged or joint structural adaptation, including alternate pruning-distillation in SlimSAM, expert-width reduction in SlimMoE, and data-network co-optimization in DANCE (Chen et al., 2023, Li et al., 23 Jun 2025, Li et al., 2021).

This suggests that “dynamic slimming” is best treated as an umbrella term rather than a single algorithmic template. In some papers, it means runtime width switching inside a nested supernetwork; in others, it means progressive compression over training stages; in still others, it denotes dynamic activation of only the needed adaptation or sensing capacity (Chen et al., 2023, Han et al., 2024, Johnsen et al., 2024).

2. Core mechanisms of slimming

A common mechanism is nested channel-width selection. In Dynamic Slimmable Network (DS-Net), the active filters in a convolution are the prefix slice determined by a slimming ratio $\rho \in (0,1]$:

$$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$

The paper emphasizes that filters are stored statically and contiguously in hardware, so dynamic inference changes width without incurring indexing, weight-copying, or zero-masking overhead (Li et al., 2021). Slimmable ConvNeXt follows the same prefix principle at the block level: a subnetwork at width $p$ keeps only the first $\lfloor p \cdot C \rfloor$ channels of a block with $C$ channels, and narrower subnetworks are strict subsets of wider ones (Haberer et al., 21 May 2026).

A second mechanism is progressive structural compression. SlimSAM explicitly defines “dynamic slimming” as an alternation over two decoupled sub-structures of a ViT encoder: embedding dimensions $E=\{e_i\}_{i=1}^k$ and bottleneck dimensions $$L = \sum_{(x, y)} l(f(x, W), y) + \lambda \sum_{\gamma \in \Gamma} g(\gamma),$$0. Its four-step procedure is: embedding pruning, bottleneck aligning, bottleneck pruning, and embedding aligning (Chen et al., 2023). SlimMoE adopts a related staged view: target architecture design, one-shot pruning to an intermediate size, distillation from the original full model, and repetition over several stages until the target size is reached (Li et al., 23 Jun 2025). In both cases, compression is not a single irreversible operation.

A third mechanism is sample- or context-dependent routing. DDS-Net trains a dynamic gate that predicts which candidate subnetwork should process a noisy image, while SLIM routes tokens or samples either through LoRA experts or identity experts via a soft routing function $$L = \sum_{(x, y)} l(f(x, W), y) + \lambda \sum_{\gamma \in \Gamma} g(\gamma),$$1:

$$L = \sum_{(x, y)} l(f(x, W), y) + \lambda \sum_{\gamma \in \Gamma} g(\gamma),$$2

Here each $$L = \sum_{(x, y)} l(f(x, W), y) + \lambda \sum_{\gamma \in \Gamma} g(\gamma),$$3 is either a LoRA module or identity (Han et al., 2024). Adaptive speech enhancement similarly attaches a router subnet to a slimmable DEMUCS backbone and chooses among utilization factors $$L = \sum_{(x, y)} l(f(x, W), y) + \lambda \sum_{\gamma \in \Gamma} g(\gamma),$$4 for each input (Miccini et al., 7 Jul 2025).

A fourth mechanism extends slimming beyond model internals to input or token processing. DANCE performs complexity-adaptive downsampling, stochastic dropping, and loss weighting at the data level before progressive pruning at the network level (Li et al., 2021). DocKylin applies Adaptive Pixel Slimming and Dynamic Token Slimming so that the multimodal model sees a shorter, more informative visual sequence (Zhang et al., 2024). AdaFlow compresses the key-value sequence used in extended self-attention by keeping only the most relevant tokens for the current query frame (Zhang et al., 8 Feb 2025).

3. Dynamic slimming in training-time compression

Training-time dynamic slimming often revolves around staged pruning and distillation. SlimSAM is a canonical case. It begins from Taylor pruning, where importance is approximated by

$$L = \sum_{(x, y)} l(f(x, W), y) + \lambda \sum_{\gamma \in \Gamma} g(\gamma),$$5

but argues that this is mismatched to SAM compression because pruning uses hard-label loss change while recovery uses distillation on soft teacher embeddings (Chen et al., 2023). To align the objectives, SlimSAM introduces disturbed Taylor importance:

$$L = \sum_{(x, y)} l(f(x, W), y) + \lambda \sum_{\gamma \in \Gamma} g(\gamma),$$6

where $$L = \sum_{(x, y)} l(f(x, W), y) + \lambda \sum_{\gamma \in \Gamma} g(\gamma),$$7 is Gaussian noise with mean $$L = \sum_{(x, y)} l(f(x, W), y) + \lambda \sum_{\gamma \in \Gamma} g(\gamma),$$8 and standard deviation $$L = \sum_{(x, y)} l(f(x, W), y) + \lambda \sum_{\gamma \in \Gamma} g(\gamma),$$9 (Chen et al., 2023). Its bottleneck-aligning and embedding-aligning losses also use changing dynamic weights $g(s)=|s|$0 with $g(s)=|s|$1, so supervision shifts over time from intermediate matching to final embedding recovery (Chen et al., 2023). The reported effect is a large gain over one-step pruning: about $g(s)=|s|$2 MIoU improvement with disturbed Taylor importance and $g(s)=|s|$3 with random importance (Chen et al., 2023).

SlimMoE compresses Phi-3.5-MoE by shrinking the hidden width of every expert uniformly while keeping the number of experts fixed. The sensitivity of a parameter is defined by

$g(s)=|s|$4

and neuron scores are aggregated as

$g(s)=|s|$5

After pruning, the model is distilled from the original teacher with a top-8 logits KL objective (Li et al., 23 Jun 2025). The paper reports that one-shot pruning leads to a large performance degradation at the initial pruning step, whereas multi-stage compression is stronger: for Phi-mini-MoE, one-stage MMLU is 68.58 and multi-stage MMLU is 69.87; for Phi-tiny-MoE, one-stage is 58.40, iterative is 60.05, and multi-stage is 60.08 (Li et al., 23 Jun 2025).

SP-Net occupies an intermediate position between slimmable inference and pruning. Standard slimmable neural networks use uniform width multipliers across layers, whereas SP-Net embeds pruned, non-uniform subnetworks into a shared largest model (Kuratsu et al., 2022). Its multi-base pruning strategy trains several base models first and then prunes them to produce subnetworks for smaller widths, which can be parallelized and avoids the repeated prune–fine-tune cycles of iterative pruning (Kuratsu et al., 2022). The paper reports that, compared with S-Net at the same FLOPs on ImageNet, SP-Net improves accuracy by $g(s)=|s|$6–$g(s)=|s|$7 for ResNet-50, $g(s)=|s|$8–$g(s)=|s|$9 for VGGNet, $\Gamma$0–$\Gamma$1 for MobileNetV1, and $\Gamma$2–$\Gamma$3 for MobileNetV2 (Kuratsu et al., 2022).

4. Input-adaptive and runtime-selective slimming

Runtime-selective slimming is centered on gates and routers. DS-Net uses a double-headed dynamic gate with an attention head and a slimming head. The gate selects a candidate slimming ratio from a discrete list $\Gamma$4 via

$\Gamma$5

The paper argues that dynamic sparse patterns typically fail to achieve actual acceleration, whereas contiguous slicing is hardware-friendly. On ResNet-50 with 25% channels, measured latency is 12.2 ms for the full model, 12.4 ms for masking, 16.6 ms for indexing, 7.9 ms for slicing, and 7.2 ms for the ideal channel-scaled case (Li et al., 2021). DS-Net reports typical compute reduction of $\Gamma$6–$\Gamma$7 and 1.62x real-world acceleration over ResNet-50 and MobileNet with minimal accuracy drops on ImageNet (Li et al., 2021).

DDS-Net transfers this paradigm to restoration. It first trains a slimmable denoising supernetwork, then progressively tailors channel configurations, and finally trains a dynamic gate to map noisy inputs to optimized subnetworks (Jiang et al., 2021). The gate uses features from the first convolutional block and predicts one-hot subnetwork choices based on online easy/hard sample identification. Sample difficulty is defined by the PSNR gap between the largest and smallest subnetworks:

$\Gamma$8

The gate loss combines a cross-entropy term and a complexity penalty (Jiang et al., 2021). On SIDD, DDS-MPRNet at the 5M regime uses 5.11M FLOPs/Pixel versus 5.39M for static MPRNet, while improving PSNR from 39.68 to 39.79 (Jiang et al., 2021).

Adaptive speech enhancement extends input-conditioned slimming to audio. The slimmable DEMUCS backbone supports utilization factors $\Gamma$9, and a router network chooses the UF using a Gumbel-softmax relaxation:

$\rho \in (0,1]$0

(Miccini et al., 7 Jul 2025). The joint objective is

$\rho \in (0,1]$1

where the auxiliary terms push average utilization toward a target and discourage collapse onto a few UFs (Miccini et al., 7 Jul 2025). The headline result is that training the dynamically-slimmable model to use 10% capacity on average matches or exceeds the speech quality of a static 25% UF model while reducing MACs by 29% (Miccini et al., 7 Jul 2025).

In LLM fine-tuning, SLIM interprets dynamic slimming as conditional activation of parameter-efficient adapters. The architecture mixes LoRA experts and identity experts and uses weight-yielding with sliding clustering to improve out-of-domain routing (Han et al., 2024). The reported downstream average score is 86.63 for SLIM versus 77.39 for LoRA, 77.32 for DoRA, 85.12 for LoRAMoE, 83.06 for MixLoRA, and 83.11 for MoLA, while the average generalization score is 75.65 for SLIM versus 31.24, 29.41, 66.39, 51.08, and 47.79 for the same baselines (Han et al., 2024). This suggests that dynamic slimming need not be limited to convolutional width; it can also mean sparse, selective activation of low-rank adaptation paths.

5. Data, token, and context co-adaptation

A recurring misconception is that slimming concerns only network parameters. Several systems instead treat data, tokens, or sensing as first-class optimization targets.

DANCE explicitly co-optimizes input data and network architecture for semantic segmentation. It estimates image spatial complexity by

$\rho \in (0,1]$2

where $\rho \in (0,1]$3 and $\rho \in (0,1]$4 are Sobel-filtered gradients and $\rho \in (0,1]$5 is the number of pixels (Li et al., 2021). The indicator is reported as about 0.15% FLOPs and less than 0.5% energy relative to a DeepLabv3+ pass on a $\rho \in (0,1]$6 patch (Li et al., 2021). After mapping complexity to a normalized value $\rho \in (0,1]$7, DANCE uses: downsampling ratio $\rho \in (0,1]$8, dropping probability $\rho \in (0,1]$9, and loss weight $$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$0, with

$$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$1

It then performs progressive channel pruning during training using the Network Slimming criterion based on BatchNorm scaling factors (Li et al., 2021). On DeepLabv3+ at Cityscapes, co-optimization yields -35.75% training FLOPs, -56.57% inference FLOPs, and +3.41 mIoU, outperforming sequential data-only or network-only optimization (Li et al., 2021).

DocKylin moves slimming to the visual document pipeline. Adaptive Pixel Slimming computes a Sobel gradient map, removes low-information regions, and reduces average image resolution on DocVQA from $$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$2 to $$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$3, a reduction of 36.0%; on InfoVQA it reduces $$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$4 to $$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$5, a reduction of 18.1% (Zhang et al., 2024). Dynamic Token Slimming then clusters visual tokens into essential and nonessential groups, routes each nonessential token to its most similar essential token, and merges them by similarity-weighted aggregation (Zhang et al., 2024). In ablation on DocVQA validation, average token length drops from 2770 without slimming to 2198 with APS and to 1250 with APS + DTS, while latency drops from 1.00 to 0.61 and accuracy improves from 63.1 to 65.1 (Zhang et al., 2024).

AdaFlow similarly slims the effective sequence processed by attention. In standard extended self-attention, a query frame attends to all tokens from all frames:

$$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$6

AdaFlow computes DIFT-based cosine-similarity heatmaps between keyframes, keeps only the $$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$7 pixel positions with the highest values, and attends to slimmed sequences $$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$8 instead (Zhang et al., 8 Feb 2025). Pruning is activated if the number of keyframes exceeds 14, and the system consistently retains the token count corresponding to 14 frames (Zhang et al., 8 Feb 2025). The result is high-quality long video editing of more than 1k frames on one A800 GPU, about ten times longer than compared methods such as TokenFlow (Zhang et al., 8 Feb 2025).

6. Domain-specific instantiations

Dynamic slimming has been specialized across perception, compression, recommendation, and generative modeling.

In semantic segmentation, SlimSeg trains one network whose width can be switched at inference time across $$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$9, or $p$0 for DeepLabv3+ with MobileNetv2 (Xue et al., 2022). Its objective combines segmentation loss, boundary detection loss, and boundary-guided masked segmentation loss:

$p$1

with $p$2 and $p$3 (Xue et al., 2022). The paper reports that stepwise downward distillation improves all widths, and boundary supervision concentrates corrections near borders; about 50% of the corrected pixels are within 5 pixels of boundaries (Xue et al., 2022). On Cityscapes, Slim-SFNet-ResNet50 attains 74.4 mIoU at $p$4 with 39.4 GFLOPs and 46.2 FPS, and 78.4 mIoU at $p$5 with 607.9 GFLOPs and 9.0 FPS (Xue et al., 2022).

In video compression, SlimVC uses uniform slimming factors $p$6 across a feature autoencoder and a temporal entropy model (Liu et al., 2022). The underlying rate-distortion objective is

$p$7

with $p$8 (Liu et al., 2022). For 1080p sequences, low-rate encoding total is about 73 GFLOPs and high-rate encoding total is about 436 GFLOPs, compared with 643 GFLOPs for STEM and 3074 GFLOPs for DVC (Liu et al., 2022). The paper reports up to 20× speedup in low-rate settings (Liu et al., 2022).

In edge recommendation, DIET generates edge-specific “diets” from a frozen generic backbone using recent interactions. Each masked layer is written as

$p$9

where $\lfloor p \cdot C \rfloor$0 keeps the top-$\lfloor p \cdot C \rfloor$1 absolute values and a straight-through estimator is used for backpropagation (Fu et al., 2024). DIETING further represents repeated modules with a single layer template (Fu et al., 2024). On MovieLens-100K with SASRec, DIET improves NDCG from 0.0517 to 0.0635 and Hit from 0.1077 to 0.1319; the paper characterizes this as roughly a 22.82% NDCG improvement and 22.47% Hit improvement (Fu et al., 2024).

In diffusion models, SlimDiff compresses attention and feedforward dimensionalities without retraining, using activation covariances across denoising timesteps (Roy et al., 25 Sep 2025). Its trace-normalized Rayleigh quotient is

$\lfloor p \cdot C \rfloor$2

Module ranks are then allocated under a global budget using an entropy-regularized convex surrogate (Roy et al., 25 Sep 2025). The method reports up to 35% inference speedup, about $\lfloor p \cdot C \rfloor$3M parameter reduction, and 22% FLOP reduction while preserving quality; on Stable Diffusion v1.5, it compresses from 1.04B to 0.76B parameters and achieves FID 13.12 versus 13.07 for the uncompressed model (Roy et al., 25 Sep 2025). This suggests a broader interpretation of dynamic slimming: deployment-time compression can be statically materialized but dynamically informed by time-varying activation statistics.

7. Systems perspective, controversies, and open directions

One persistent controversy concerns what should count as “dynamic.” DS-Net and DDS-Net are dynamic in the strict runtime sense because different inputs activate different widths at test time (Li et al., 2021, Jiang et al., 2021). SlimSAM and SlimMoE are dynamic in a staged training sense because architecture and supervision evolve over multiple compression phases (Chen et al., 2023, Li et al., 23 Jun 2025). DANCE is dynamic in yet another sense: data are adaptively downsampled or dropped per image, while the network is progressively pruned to match the induced scale distribution (Li et al., 2021). A plausible implication is that the field now uses “dynamic slimming” to describe any slimming process in which capacity is adjusted conditionally or progressively rather than chosen once and left unchanged.

Another point of disagreement concerns whether theoretical sparsity yields real acceleration. DS-Net explicitly argues that dynamic sparse patterns on convolutional filters fail to achieve actual acceleration because of indexing, weight-copying, or zero-masking, and proposes contiguous slicing as the remedy (Li et al., 2021). SP-Net arrives at a similar conclusion from the opposite direction: non-uniform pruned channels degrade latency unless Slimmable Channel Sorting restores contiguous prefixes (Kuratsu et al., 2022). This suggests that dynamic slimming is as much a systems problem as an optimization problem.

A third issue is whether adaptability should be applied only to the model or to the full perception-action stack. NaviSlim jointly adapts network width and sensor power levels, with average reduced model complexity between 57–92% and sensor utilization between 61–80% relative to static neural networks designed for the most difficult scenario (Johnsen et al., 2024). CADENCE closes the loop between navigation policy and depth-network slimming factor $\lfloor p \cdot C \rfloor$4, even allowing $\lfloor p \cdot C \rfloor$5 to bypass image acquisition and network execution entirely (Johnsen et al., 8 Apr 2026). Compared with a static state-of-the-art baseline, CADENCE reports reductions of 9.67% in sensor acquisitions, 16.1% in power consumption, 74.8% in inference latency, and 75.0% in energy expenditure, with a 7.43% increase in navigation accuracy (Johnsen et al., 8 Apr 2026). These results indicate that the most consequential forms of dynamic slimming may emerge when model selection is integrated with actuation and sensing rather than treated as a standalone compression layer.

Finally, recent work points toward increasingly heterogeneous deployment regimes. Slimmable ConvNeXt motivates width-adaptive inference by variable compute availability across cloud GPUs, edge servers, phones, battery states, thermal throttling, or latency deadlines (Haberer et al., 21 May 2026). On ImageNet-1k, Slimmable ConvNeXt-T with three subnetworks reaches 80.8% top-1 at around 4.5–4.6 GMACs and 77.4% at around 1.2 GMACs, outperforming HydraViT, MatFormer, SortedNet, and DynaBERT at comparable compute (Haberer et al., 21 May 2026). This line of work suggests that dynamic slimming is becoming a general deployment abstraction: one weight set, multiple executable capacities, and optional routing that aligns active compute with instantaneous budget.

Taken together, the literature defines dynamic slimming not as a single technique but as a design principle for scalable intelligence: width, experts, tokens, data, sensing, and even calibration budgets are treated as elastic resources. The principal research challenge is no longer merely how to compress a model, but how to match capacity to the structure of inputs, tasks, devices, and trajectories as they change (Li et al., 2021, Li et al., 2021, Han et al., 2024, Haberer et al., 21 May 2026).