Dynamic Slimmable Network

• Dynamic Slimmable Networks are weight-sharing models that adjust the active subnetwork’s width based on the input to trade accuracy for efficiency.
• They employ contiguous channel slicing and dynamic gating mechanisms, ensuring dense and hardware-friendly computations.
• Training integrates in-place distillation, the sandwich rule, and gated optimization to jointly learn weights and optimal width configurations.

Dynamic Slimmable Network (DSN) denotes a family of weight-sharing neural networks in which a single super-network supports multiple subnetworks of different widths and, in the stricter sense established by DS-Net, can select the active width dynamically at test time according to the input so as to trade accuracy for efficiency without loading separate models. The lineage runs from slimmable neural networks, which made one model executable at different widths through shared parameters and Switchable Batch Normalization, to universally slimmable models with arbitrary-width execution, and then to input-dependent dynamic slimming designed for real hardware efficiency (Yu et al., 2018, Yu et al., 2019, Li et al., 2021).

1. Origins, scope, and terminology

The foundational slimmable-network formulation defines a network “executable at different widths,” where width is the number of channels in a layer. A specific width configuration is a “switch,” and typical examples are 0.25×, 0.5×, 0.75×, and 1.0×. The practical motivation is adaptive accuracy-efficiency trade-offs at runtime: a single deployed model can be reconfigured according to device capability, latency budget, battery state, or other on-device constraints (Yu et al., 2018).

Universally Slimmable Networks (US-Nets) strengthened this idea by asking whether one trained model can run at any width, not just a few fixed switch points. Their answer was affirmative, provided that training uses the sandwich rule and inplace distillation, and that batch-normalization statistics are calibrated after training rather than maintained for every possible width. In this formulation, inference can use any width in a continuous range, which makes it possible to evaluate a dense FLOPs-Accuracy spectrum from a single model (Yu et al., 2019).

DS-Net reframed the problem from static width choice to input-dependent width choice. Rather than merely allowing runtime selection from a width list, it predicts the width for each sample at test time. The central claim is that dynamic pruning methods often reduce theoretical computation without delivering wall-clock acceleration because arbitrary sparse filter subsets are hardware-unfriendly. DS-Net therefore restricts the dynamic choice to contiguous channel prefixes, so that the active subnetwork remains dense in memory and can be executed by simple tensor slicing (Li et al., 2021).

This terminological evolution matters. In some papers, DSN refers broadly to dynamic or slimmable width-adaptive inference; in others, especially DS-Net and its descendants, it denotes a specific hardware-aware regime in which the effective architecture changes with the input while weights remain stored statically and contiguously. A common misconception is that all slimmable networks are already “dynamic” in the input-conditioned sense. The literature distinguishes clearly between budget-selectable slimmable models and models whose width is chosen online by a learned controller.

2. Core architectural mechanisms

At the architectural level, DSN is built around a super-network containing the parameters of all narrower subnetworks. A width multiplier scales channel counts or hidden dimensions, and narrower subnetworks reuse prefixes or slices of the larger model’s parameters. In the original slimmable setting, all switches share convolutional and fully connected weights, while each switch has its own batch-normalization parameters and statistics through Switchable Batch Normalization (Yu et al., 2018).

DS-Net makes the slicing rule explicit. If a stage has $N$ output channels and the gate predicts a slimming ratio $\rho \in (0,1]$, only the first $\rho N$ channels are used:

$$\mathcal{Y} = \mathbf{W}[~:\rho \times N] * \mathcal{X}.$$

Because the selected channels are always the first contiguous block, the runtime subnetwork is dense rather than irregularly sparse. DS-Net couples this with a double-headed dynamic gate comprising an attention head and a slimming head. The slimming head predicts a one-hot choice over a discrete candidate list $L_\rho$, and the selected ratio is

$$\rho_n = \mathcal{A}(\theta, \mathcal{X}) \cdot L_\rho.$$

The attention head is used during supernet training to improve representation learning, while the slimming head is used for hard width selection at test time (Li et al., 2021).

DS-Net++ generalizes this from dynamic channel count alone to multi-dimensional dynamic slicing. For CNNs, the elastic dimensions include filter number, kernel size, group number or per-group filter number, dilation, padding, and layer number in residual networks. For transformers, the elastic dimensions include embedding dimension, Q-K-V dimension, number of heads, MLP ratio or hidden dimension, and depth. The paper gives explicit slicing formulations for patch embedding, Q/K/V projections, attention output projection, and MLP blocks, extending dynamic slimming from classical CNN width scaling to multi-axis architectural adaptation (Li et al., 2021).

Application-specific DSNs instantiate these mechanisms differently. In small-footprint keyword spotting, CNN slimming is done by dropping convolution kernels or channels as width decreases, while transformer slimming reduces the dimensions of Q/K/V and MLP hidden states; BatchNorm in CNNs and LayerNorm in transformers are made switchable per width (Akhtar et al., 2023). In speech separation, slimmability is applied to SepFormer-style multi-head attention and feed-forward blocks via a utilization factor $U_t$, so that only the first $\lceil U'_m H \rceil$ heads and the first $\lceil U'_m D \rceil$ hidden units are active at a frame (Elminshawi et al., 8 Jul 2025). In speech enhancement, the architecture is divided into static and dynamic parts, and a frame-wise gate controls whether the dynamic branch contributes to grouped GRUs, multi-head attention, convolutional layers, and fully connected layers (Zhao et al., 13 Oct 2025).

3. Training and optimization regimes

The basic slimmable training recipe is joint optimization over multiple widths. In each iteration, one mini-batch is loaded, gradients are cleared, each width in the predefined switch list is activated in turn, the corresponding loss is computed and backpropagated, and the shared weights are updated once after all switches have been processed. The objective is the unweighted sum of losses over the trained widths (Yu et al., 2018).

US-Nets introduced two training devices that became central in later DSN work. The sandwich rule always includes the smallest and largest widths in each iteration, together with random intermediate widths. Inplace distillation uses the largest-width output as teacher: the largest width is trained on ground-truth labels, while narrower widths are trained against the detached full-width prediction. For arbitrary-width models with batch normalization, US-Nets avoid switchable BN over infinitely many widths by computing post-training BN statistics for each desired width (Yu et al., 2019).

DS-Net separates supernet learning from gate learning. In Stage I, the slimmable supernet is trained with In-place Ensemble Bootstrapping (IEB), which uses an exponential moving average target network as a stable teacher and trains the slimmest subnet against an ensemble average of sampled teacher outputs. In Stage II, the supernet weights are frozen and the gate is trained with a loss combining classification, complexity, and Sandwich Gate Sparsification (SGS). SGS identifies easy, dependent, and hard samples online and uses this signal to prevent the gate from collapsing into a static routing policy (Li et al., 2021).

Joslim argued that shared weights alone are not enough: the width configurations themselves should also be optimized jointly with the weights. It formulates training as

$$\arg\min_{\bm{\theta},w}\mathbb{E}_{c\sim U(l,u)} L_{\mathcal{S}(\bm{\theta},w(c))},$$

where $\rho \in (0,1]$0 maps a target computational budget $\rho \in (0,1]$1 to an architecture. The method approximates this joint problem by alternating optimization, targeted sampling of budgets, local weight updates, and a temporal-sharing search procedure called MOBO-TS2. The key distinction from conventional globally scaled slimmable networks is that $\rho \in (0,1]$2 can encode layer-wise non-uniform widths rather than a single multiplier shared by all layers (Chin et al., 2020).

DSPNet carries slimmable training into self-supervised pretraining. The online encoder is slimmable, the target branch remains full-size and follows BYOL’s EMA update, and training samples the smallest subnet, the largest subnet, and $\rho \in (0,1]$3 random subnetworks according to the sandwich rule, with $\rho \in (0,1]$4 by default. All sampled online subnetworks regress to the same target representation, so the method functions as simultaneous self-supervised learning and in-place distillation across widths (Wang et al., 2022).

Task-specific studies confirm that multi-width training can be efficient. In keyword spotting, the super-network is trained by running the same batch through several widths, accumulating gradients, and updating the shared parameters once; increasing from 1 to 40 widths raises training time per step from 1.01 s to 3.72 s, which the paper describes as far less than linear scaling (Akhtar et al., 2023).

4. Runtime control, hardware efficiency, and cost objectives

A central DSN claim is that theoretical sparsity is not equivalent to deployable acceleration. DS-Net explicitly compares several ways of obtaining a 25% ResNet-50 on an RTX 2080 Ti: Full ResNet-50: 12.2 ms, Masking: 12.4 ms, Indexing: 16.6 ms, Slicing: 7.9 ms, and Ideal individual 0.25× ResNet-50: 7.2 ms. This comparison motivates the preference for dense contiguous slicing over arbitrary channel masking or indexing (Li et al., 2021).

DS-Net++ makes the same point in a broader setting. By treating dynamic inference as dynamic weight slicing rather than sparse computation, it reports 2-4x computation reduction and 1.62x real-world acceleration over MobileNet, ResNet-50, and Vision Transformer, with minimal ImageNet accuracy drops of 0.1-0.3%. The paper’s emphasis is that dense contiguous slices align better with hardware kernels than irregular sparse patterns do (Li et al., 2021).

The resource objective in DSN research is not limited to FLOPs. Joslim defines a width configuration function over budgets and can optimize either FLOPs or inference memory footprint. For memory, it uses

$\rho \in (0,1]$5

and reports that, on ImageNet with MobileNetV2, improvements of up to 1.7 percentage points in top-1 accuracy can be attained for FLOPs-based trade-offs and up to 8 percentage points for memory-footprint optimization. This directly challenges the assumption that “slimming every layer equally” is adequate for all deployment constraints (Chin et al., 2020).

Later DSN variants make the control signal more fine-grained. In speech separation, the gate predicts a utilization level $\rho \in (0,1]$6 for each frame from a discrete set such as $\rho \in (0,1]$7 or $\rho \in (0,1]$8, and the training objective adds a complexity term

$\rho \in (0,1]$9

with either signal-independent or signal-dependent weights $\rho N$0. The signal-dependent version penalizes easy segments more strongly than hard ones, biasing the system toward low utilization on silence and non-overlap while allowing high utilization on difficult regions (Elminshawi et al., 8 Jul 2025).

In speech enhancement, dynamic control is also frame-wise, but the gate is binary. During training, $\rho N$1 is produced by Gumbel-Softmax with $\rho N$2; at inference, $\rho N$3. Metric-Guided Training replaces a fixed sparsity target with a quality-dependent target activation ratio

$\rho N$4

where $\rho N$5 is the DNS-MOS P.835 OVRL score. This ties dynamic computation explicitly to estimated input quality without adding inference overhead (Zhao et al., 13 Oct 2025).

5. Representative empirical results across domains

In image classification and detection, the foundational result was that slimmable networks achieve similar, and often better, accuracy than individually trained models at the same widths. On ImageNet, the original slimmable paper reports, for example, S-MobileNet v1 errors of 28.5%, 30.5%, 35.2%, and 46.9% at 1.0×, 0.75×, 0.5×, and 0.25×, compared with 29.1%, 31.6%, 36.7%, and 50.2% for individually trained models; corresponding gains also appear in COCO Faster R-CNN, Mask R-CNN, and Keypoints R-CNN, especially at slimmer widths (Yu et al., 2018).

DS-Net extends these gains to input-dependent routing. It reports that DS-Net consistently outperforms static counterparts and state-of-the-art static and dynamic compression methods by up to 5.9%, while achieving 2-4x computation reduction and 1.62x real-world acceleration over ResNet-50 and MobileNet with minimal accuracy drops on ImageNet. For example, DS-ResNet-S reaches 74.6% top-1 at 1.2B MAdds, and DS-MBNet-S reaches 70.1% at 153M MAdds with 39 ms latency (Li et al., 2021). DS-Net++ pushes the same paradigm to ViTs; it reports DS-ViT-L++ at 5.6B MAdds and 83.0% top-1, alongside similar improvements for CNN backbones (Li et al., 2021).

In self-supervised learning, DSPNet shows that a slimmable pretrained supernet can replace multiple separately pretrained backbones. For MobileNetV3 widths $\rho N$6, smaller desired networks generally improve over individually pretrained counterparts in ImageNet linear evaluation and semi-supervised evaluation. The reported pretraining cost is about 2.11× the cost of one width-1.0 MobileNetV3 baseline, compared with 4.42× for pretraining all desired widths separately, which the paper describes as roughly half the cost saved (Wang et al., 2022).

In image denoising, DDS-Net applies DSN ideas to VDNet, DANet, and MPRNet through a three-stage pipeline of supernet training, progressive slimming, and gate learning. On SIDD at roughly 20K FLOPs/pixel, DDS-VDNet reports 39.01 PSNR / 0.953 SSIM, compared with 37.89 / 0.943 for I-VDNet; on DND at roughly 20K FLOPs/pixel, it reports 39.20 / 0.9485, compared with 38.56 / 0.9437 for the static baseline. The paper summarizes that DDS versions consistently outperform individually trained static denoising networks, with the gain largest under tight resource constraints (Jiang et al., 2021).

In small-footprint keyword spotting, slimmable CNNs and transformers are trained under an on-device budget of less than 250k parameters. Concrete model sizes include a CNN baseline of 199k parameters for in-house data and 243k for Google Speech Commands, and a transformer baseline of 120k parameters for in-house data and 67k for Google Speech Commands. Width-0.25 subnetworks can be as small as 13k–16k parameters for CNNs and 24k or 15k for transformers. On Google Speech Commands, the slimmable CNN reports 0.9047 accuracy at width 1.0, compared with 0.8915 for the scratch-trained model, while the slimmable transformer reports 0.8787, compared with 0.8623 (Akhtar et al., 2023).

In speech separation, DSN adapts computation per segment rather than per utterance. On WSJ0-2mix and WHAM!, it achieves a better performance-efficiency trade-off than individually trained static networks. The paper reports that, with 25% overlap, the DSN gives more than 50% complexity reduction without performance loss, and that learned utilization patterns are interpretable: silence tends to receive the lowest utilization, non-overlap low or intermediate utilization, and full overlap the highest utilization (Elminshawi et al., 8 Jul 2025).

In speech enhancement, the proposed DSN with Metric-Guided Training is evaluated against lightweight baselines on Voicebank+Demand. The paper reports that MGT-DSN performs on par with FTF-Net while using only 73% of its MACs/s, and only 15% of the computational load of CCFNet+. The table reports approximately 0.14M parameters, 0.22G MACs, PESQ 2.98, STOI 0.94, and SI-SDR 18.8 for MGT-DSN, versus 0.30G MACs for FTF-Net (Zhao et al., 13 Oct 2025).

The DSN idea also appears in noncanonical deployment settings. NaviSlim applies dynamic slimming and sensor adaptation to micro-drone navigation and reports average reductions of 57–92% model complexity and 61–80% sensor utilization relative to a static model sized for the hardest scenario (Johnsen et al., 2024). In split computing for edge-IoT object detection, slimmable ensemble encoders allow runtime adaptation of encoder width and bitrate; the proposed method reports 16 configs, 85.7 MB RAM, 954 ms disk load, and 1811.5 ms warmup, and claims up to 4× reduction in encoder latency and up to 2.8× reduction in memory usage relative to competing solutions (Assine et al., 2023).

6. Distinctions, limitations, and continuing directions

The first major distinction is between budget-selectable and input-selective slimmability. Early slimmable networks and US-Nets allow a runtime system to choose widths from a predefined list or a continuous range, but they do not require a learned controller that maps inputs to widths (Yu et al., 2018, Yu et al., 2019). By contrast, DS-Net, DDS-Net, speech separation DSN, speech enhancement DSN, and NaviSlim all introduce gates or policies that choose the active subnetwork online from the input or context (Li et al., 2021, Jiang et al., 2021, Elminshawi et al., 8 Jul 2025, Zhao et al., 13 Oct 2025, Johnsen et al., 2024). DSPNet occupies an intermediate position: after pretraining, the desired networks are simply slimmed from the trained online encoder, and the paper explicitly states that no special search or gating is described for downstream use (Wang et al., 2022).

A second distinction concerns normalization and weight-sharing stability. The original slimmable paper reports that naive shared-width training with shared BN collapses to around 0.1% top-1 accuracy on ImageNet, motivating Switchable Batch Normalization (Yu et al., 2018). US-Nets treat arbitrary-width BN differently, using post-statistics calibration rather than maintaining separate BN states for all possible widths (Yu et al., 2019). Later systems use task-dependent alternatives, such as GroupNorm in DS-ResNet and switchable LayerNorm in slimmable transformers for keyword spotting (Li et al., 2021, Akhtar et al., 2023).

A third issue is width-space design. Conventional slimmable networks often use a single global multiplier for all layers, but Joslim argues that this ignores unequal layer sensitivity and that width search and weight training should not be decoupled. Its formulation and results suggest that layer-wise width optimization matters especially when the target resource is memory rather than FLOPs (Chin et al., 2020). DS-Net++ makes a related point from the architectural side by allowing elasticity in multiple dimensions, not only uniform width (Li et al., 2021).

A fourth issue is the difference between theoretical and realized efficiency. DS-Net’s slicing results show that dense contiguous selection can approach the latency of individually trained narrow models, while masking and indexing may fail to accelerate at all (Li et al., 2021). At the same time, the speech separation study notes that current hardware is optimized for static computation graphs, so dynamic inference does not always translate directly into real-world speedups (Elminshawi et al., 8 Jul 2025). This suggests that DSN is simultaneously a modeling problem and a systems problem.

A plausible implication is that DSN research is moving from global, manually specified width multipliers toward layer-wise, modality-specific, and signal-dependent resource allocation. That trend is visible in joint width-and-weight optimization, multi-dimensional slicing, self-supervised slimmable pretraining, frame-wise speech processing, and context-aware compute-and-sensing control for navigation (Chin et al., 2020, Li et al., 2021, Wang et al., 2022, Zhao et al., 13 Oct 2025, Johnsen et al., 2024). Within that trajectory, the stable core idea remains unchanged: one super-network, many subnetworks, and a training procedure that makes those subnetworks simultaneously usable rather than merely extractable.