Width-Wise Subnetwork Construction

• Width-wise subnetwork construction is a technique that reduces the width dimension—such as channels, neurons, or expert branches—to derive smaller, valid subnetworks from a larger model.
• It employs structured methods like prefix-nested extraction, channel slicing, and routing-aware selection to maintain substantial parameter sharing while enabling flexible deployment choices.
• The approach enhances scalable training, adaptive inference, and resource-efficient model adaptation by addressing challenges like gradient coupling and variation in shared-weight evaluations.

Searching arXiv for the cited papers to ground the article and confirm metadata. Width-wise subnetwork construction denotes a family of methods that derive smaller deployable models by restricting the width dimension of a larger parent architecture rather than primarily reducing depth. In the cited literature, “width” is not uniform across architectures: it may mean hidden channels, neurons, parallel fully-connected branches, or expert FFN branches in MoE layers. Despite that variation, the recurring objective is structurally similar: construct subnetworks that remain valid end-to-end models, preserve substantial parameter sharing with a larger model, and expose a discrete or structured set of operating points for deployment, search, distributed training, or parameter-efficient adaptation (Li et al., 2023). The topic therefore spans nested prefix slicing, locally relaxed channel selection, expert-subset extraction, and worker- or client-specific width masks, with recurring concerns about tractability, aggregation, gradient coupling, and the fidelity of shared-weight evaluation (Su et al., 2021).

1. Conceptual scope and definitions

Width-wise subnetwork construction is distinguished from depth-wise selection by its locus of reduction. Depth-wise methods remove layers or truncate a network; width-wise methods retain the layer stack and instead select a smaller subset of within-layer computational components. The literature instantiates those components differently. In "Subnetwork-to-go" (Li et al., 2023), width is explicitly “the number of fully-connected (FC) layers in the layer,” implemented as parallel FC branches inside each dynamic-width residual RNN block. In WHALE-FL (Su et al., 2024), subnetworks are obtained by “shrinking the width of hidden channel with specific ratios.” In CafeNet (Su et al., 2021), a width candidate is a per-layer channel-count tuple $c=(c_1,\ldots,c_L)$, and the construction problem is how that width should be instantiated inside a one-shot supernet. In RISE (Zheng et al., 4 Apr 2026), the natural width dimension of an MoE model is the set of experts within each MoE layer, so a width-wise subnetwork is a subset of expert FFN branches while the full depth remains intact.

This variability of meaning is a central technical point rather than a terminological accident. A common misconception is that width-wise construction always means standard channel slimming. The cited work shows otherwise. Width may be branch count in a residual RNN (Li et al., 2023), hidden-channel count in a CNN-style federated model (Su et al., 2024), locally varying channel subsets in a NAS supernet (Su et al., 2021), neuron or channel masks in distributed subnetwork training (Singh et al., 11 Jul 2025), or expert selection in multilingual sparse MoE models (Zheng et al., 4 Apr 2026). What unifies these settings is the structural reduction of a model’s parallel capacity dimension.

A second recurring property is nesting, though it is not universal. Several methods construct subnetworks so that smaller widths are literal subsets of larger ones. "Subnetwork-to-go" uses a deterministic prefix rule over FC branches, yielding

$$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$

WHALE-FL inherits a similarly nested hierarchy,

$W^P\subset W^{P-1}\subset \cdots \subset W^1,$

with extraction written as tensor slicing $W(g,r)[:d_i,:k_i]$ (Su et al., 2024). By contrast, CafeNet explicitly relaxes strict nesting through locally free channel selection, preserving only partial overlap between nearby widths (Su et al., 2021). This suggests that width-wise construction is best viewed as a spectrum ranging from fully nested prefix sharing to partially shared local alternatives.

2. Canonical construction patterns

The simplest and most explicit pattern is the prefix-nested supernetwork. In "Subnetwork-to-go" (Li et al., 2023), the full network has maximum depth $D$ and maximum width $W$, with experiments using

$D=12,\qquad W=16.$

A width-$w$ subnetwork keeps only the first $w$ FC branches in every dynamic-width residual RNN layer. Branches $w+1,\ldots,W$ are omitted. No learned gating, arbitrary sparse masks, or post-hoc pruning are used for construction itself. The extraction rule is therefore deterministic, ordered, and nested. The paper’s formulation $$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$0 makes explicit that narrower subnetworks are literal parameter subsets of the full elastic model.

A second canonical pattern is the channel-prefix construction. WHALE-FL (Su et al., 2024) adopts HeteroFL-style width shrinkage and represents a family of discrete candidate subnetworks

$$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$1

with hidden-channel shrinkage ratio $$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$2 and size scaling

$$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$3

Its implementation clue is the slicing rule

$$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$4

where $$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$5 and $$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$6. This strongly indicates ordered selection of leading channels or dimensions. The resulting subnetworks are width-wise, ordered, nested, and parameter-sharing.

CafeNet (Su et al., 2021) treats the standard prefix rule itself as a problem. Under fixed sharing,

$$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$7

the smaller width is always fully contained in the larger, so the weight-sharing degree

$$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$8

equals $$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$9 for any pair of widths. CafeNet argues that this maximal coupling makes neighboring widths too hard to distinguish. It therefore introduces a base/free decomposition

$W^P\subset W^{P-1}\subset \cdots \subset W^1,$0

with a local zone

$W^P\subset W^{P-1}\subset \cdots \subset W^1,$1

free-channel count $W^P\subset W^{P-1}\subset \cdots \subset W^1,$2, and base size $W^P\subset W^{P-1}\subset \cdots \subset W^1,$3. The construction

$W^P\subset W^{P-1}\subset \cdots \subset W^1,$4

relaxes strict prefix inheritance while keeping the search space tractable. This is a decisive departure from the fully nested model family.

A fourth pattern appears in MoE models. RISE (Zheng et al., 4 Apr 2026) constructs a width-wise subnetwork as a subset of layer-expert pairs

$W^P\subset W^{P-1}\subset \cdots \subset W^1,$5

where each pair denotes expert $W^P\subset W^{P-1}\subset \cdots \subset W^1,$6 in MoE layer $W^P\subset W^{P-1}\subset \cdots \subset W^1,$7. The depth structure remains unchanged; only selected experts are trainable, while all unselected experts and all shared or non-expert parameters are frozen. Here width is not channel count but the expert set inside each sparse layer. This suggests that width-wise construction generalizes naturally to architectures whose principal parallelism is expert routing rather than dense hidden channels.

3. Shared-weight training and selection regimes

Width-wise construction interacts directly with training strategy because different widths often share parameters. "Subnetwork-to-go" (Li et al., 2023) uses a two-network training scheme per sample: the full model $W^P\subset W^{P-1}\subset \cdots \subset W^1,$8 and one randomly sampled subnetwork $W^P\subset W^{P-1}\subset \cdots \subset W^1,$9 are both evaluated, and the losses are summed,

$W(g,r)[:d_i,:k_i]$0

This is neither Once-for-All-style sandwich training nor progressive shrinking; it is a simpler regime that always optimizes the largest model and one random subnetwork together. Because narrower subnetworks use the first $W(g,r)[:d_i,:k_i]$1 branches, gradients from different widths accumulate on shared prefix parameters. A plausible implication is that the construction rule and the training rule are inseparable: prefix nesting alone does not determine quality unless the shared parameters are trained across the supported width range.

CafeNet makes this dependence even more explicit. Since width $W(g,r)[:d_i,:k_i]$2 corresponds to a family of local subnetworks $W(g,r)[:d_i,:k_i]$3, training uses a min-min strategy: $W(g,r)[:d_i,:k_i]$4 Only the best local realization for a sampled width is backpropagated. Search then uses a complementary max-max rule over validation accuracy under FLOPs constraints (Su et al., 2021). This means the construction is not merely a static mapping from widths to channel subsets; it is tied to an optimization protocol that decides which local instantiation represents a width during training and search.

In multilingual MoE adaptation, RISE (Zheng et al., 4 Apr 2026) separates construction from subsequent optimization. Routing statistics are collected first, experts are ranked, a budgeted subnetwork is assembled, and then only the selected experts are updated under

$W(g,r)[:d_i,:k_i]$5

The subnetwork is thus selected by routing evidence rather than learned jointly as a slimmable supernet. This suggests a second broad regime: width-wise construction can be either pretrained-supernet extraction or post hoc subnetwork identification inside a fixed parent model.

Distributed subnetwork training in SDP (Singh et al., 11 Jul 2025) uses yet another regime. Each worker receives a fixed permanent mask $W(g,r)[:d_i,:k_i]$6 satisfying the uniform-representation constraint

$W(g,r)[:d_i,:k_i]$7

Worker $W(g,r)[:d_i,:k_i]$8 trains

$W(g,r)[:d_i,:k_i]$9

and masked gradients are synchronized by

$D$0

Here width-wise construction is not for inference-time customization but for distributed memory reduction. The masks are fixed rather than resampled, and parameter overlap is controlled explicitly.

4. Dynamic aggregation, scheduling, and routing-aware construction

Several papers show that width-wise construction is not only a structural operation; it often requires a companion mechanism that interprets the active width configuration. In "Subnetwork-to-go" (Li et al., 2023), the transform-average-concatenate (TAC) module computes input-dependent and width-dependent coefficients $D$1 over the active branches, with

$D$2

The active branch outputs are then aggregated as a weighted sum. Because TAC uses mean pooling across the active branches, the pooled representation changes when the number of active branches changes. The same shared parameters therefore adapt differently to different widths. This is the paper’s main width-specific innovation: subnetworks are not merely prefixes of branches; they are prefixes plus width-sensitive reweighting.

WHALE-FL (Su et al., 2024) addresses a different problem: not how to construct a new width hierarchy, but how to schedule among predefined width levels under heterogeneous and time-varying client conditions. It combines system efficiency

$D$3

with a training-efficiency term $D$4 derived from local loss dynamics, and defines a utility

$D$5

That utility is normalized, quantized into a discrete width ladder $D$6, and clipped by the client capability ceiling $D$7. Width-wise construction is therefore the substrate on which adaptive scheduling operates. The paper explicitly states that its main contribution is not a new construction rule, but adaptive subnetwork selection over a fixed nested width hierarchy.

RISE (Zheng et al., 4 Apr 2026) offers a routing-aware analogue of this idea for MoE models. It estimates expert activation probabilities

$D$8

constructs target-language specificity scores

$D$9

and universal-expert overlap scores

$W$0

Selection is then depth-stratified: shallow and deep layers use specificity, middle layers use overlap. The resulting tripartite subnetwork reflects a measured convergence-divergence routing pattern across layers rather than a uniform width rule. This suggests that, in sparse conditional architectures, width-wise construction can be routing-informed rather than purely structural.

A related but more classical transformer variant appears in WideNet (Xue et al., 2021). There, width expansion is realized by replacing FFN layers with MoE experts, while attention and MoE parameters are shared across blocks and LayerNorm remains block-specific. The router computes

$W$1

and the MoE output is

$W$2

Each expert is therefore a width-wise subnetwork, and the effective subnetwork is token-dependent and block-dependent. This suggests a broader interpretation in which width-wise construction need not always produce a static smaller model; it may also define a conditional family of sparse within-layer subnetworks.

5. Empirical findings and comparative patterns

The cited work reports several recurring empirical patterns. In "Subnetwork-to-go" (Li et al., 2023), TAC improves extracted subnetworks across multiple width-depth settings. Table 1 includes, for example, $W$3 improving from $W$4 dB to $W$5 dB, and $W$6 improving from $W$7 dB to $W$8 dB, with only modest MAC increases. The same paper reports that extracted subnetworks outperform independently trained counterparts: $W$9 improves from $D=12,\qquad W=16.$0 dB to $D=12,\qquad W=16.$1 dB, and $D=12,\qquad W=16.$2 from $D=12,\qquad W=16.$3 dB to $D=12,\qquad W=16.$4 dB. The full network spans approximately $D=12,\qquad W=16.$5G to $D=12,\qquad W=16.$6G MACs across its $D=12,\qquad W=16.$7 width-depth subnetworks. At the same time, the authors explicitly note that for subnetworks with on-par model complexities, the deeper one is in general better than the wider one. Width therefore increases deployment granularity and can improve performance, but depth appears more parameter-efficient in that application.

CafeNet (Su et al., 2021) provides empirical support for relaxing strict prefix sharing. Increasing the local offset $D=12,\qquad W=16.$8 improves performance but increases training time; on ImageNet MobileNetV2, one epoch rises from $D=12,\qquad W=16.$9 s at $w$0 to $w$1 s at $w$2, then to $w$3 s at $w$4. The method improves width search outcomes over fixed-sharing baselines, and the abstract highlights a $w$5 improvement on EfficientNet-B0. A plausible implication is that the supernet’s usefulness as a width evaluator depends materially on how widths are embedded into shared parameters, not only on the optimization budget.

WHALE-FL (Su et al., 2024) emphasizes latency-accuracy tradeoffs rather than raw subnetwork quality. Relative to HeteroFL, it reports about $w$6 speedup on CNN@MNIST, $w$7 on ResNet18@CIFAR10, $w$8 on Transformer@WikiText2, and $w$9 on CNN@HAR. Its “Subnetwork Size Changes over Rounds” result shows that selected widths increase and decrease over training according to system efficiency and training status. This supports a view in which width-wise construction is especially valuable when it enables many compatible operating points rather than a single compressed model.

In SDP (Singh et al., 11 Jul 2025), width-wise masking can preserve performance in the moderate-overlap regime but degrades sharply at low overlap. For ResNet-18 on CIFAR-10 under cosine scheduling, neuron masking yields $w$0 at $w$1, $w$2 at $w$3, but $w$4 at $w$5. The paper concludes that block masking is more robust, attributing the difference to stronger gradient alignment and better preservation of residual pathways. Width-wise construction is therefore viable for distributed memory savings, with the paper reporting a $w$6–$w$7 reduction in memory usage without loss in performance in the favorable regime, but it is not the preferred strategy in the tested residual architectures.

RISE (Zheng et al., 4 Apr 2026) shows a different empirical profile. On TyDiQA, Qwen3-30B-A3B improves from $w$8 to $w$9 for Bengali, from $w+1,\ldots,W$0 to $w+1,\ldots,W$1 for Russian, and from $w+1,\ldots,W$2 to $w+1,\ldots,W$3 for Indonesian; Phi-3.5-MoE-Instruct improves from $w+1,\ldots,W$4 to $w+1,\ldots,W$5 for Bengali. The paper reports target-language F1 gains of up to $w+1,\ldots,W$6 with minimal cross-lingual degradation, while training only $w+1,\ldots,W$7 of Qwen’s experts or $w+1,\ldots,W$8 of Phi’s experts. A strong causal validation is that pruning the RISE-selected experts collapses Bengali performance to $w+1,\ldots,W$9 on Qwen and $$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$00 on Phi. This suggests that, in sparse MoE systems, width-wise expert subsets can correspond to functionally central pathways rather than arbitrary sparse slices.

6. Limitations, misconceptions, and related paradigms

A central limitation is that many methods support only a discrete family of widths. "Subnetwork-to-go" (Li et al., 2023) allows $$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$01 and $$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$02; WHALE-FL (Su et al., 2024) uses $$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$03 discrete levels; CafeNet (Su et al., 2021) searches a discretized width space further compressed by FLOPs-sensitive bins. Claims of “arbitrary width” in these papers therefore mean arbitrary width within a predefined supported set, not continuous scaling.

A second limitation concerns architecture specificity. The branch-based width of "Subnetwork-to-go" is not standard CNN channel width (Li et al., 2023). RISE’s width dimension is the expert bank of an MoE layer (Zheng et al., 4 Apr 2026). SDP’s width-wise masking is explicit for CNNs and linear layers but is not specified for Swin-T or generic transformers (Singh et al., 11 Jul 2025). This suggests that width-wise subnetwork construction is a general structural principle, but each concrete extraction rule remains tightly coupled to the architectural unit that defines width.

A third issue is that construction quality depends heavily on how shared weights are coupled across widths. Prefix nesting simplifies extraction and aggregation but can over-couple neighboring widths, which is precisely CafeNet’s criticism of the fixed pattern $$\text{subnet}(1)\subset \text{subnet}(2)\subset \cdots \subset \text{subnet}(W).$$04 (Su et al., 2021). Conversely, relaxing the prefix rule improves width discrimination but enlarges the search space and training overhead. The resulting tension between tractable sharing and faithful width evaluation is a recurrent design tradeoff.

Another misconception is that width-wise construction is synonymous with pruning. The cited literature includes direct extraction without pruning or finetuning (Li et al., 2023), adaptive scheduling over inherited width levels (Su et al., 2024), fixed-mask distributed training (Singh et al., 11 Jul 2025), routing-informed expert selection (Zheng et al., 4 Apr 2026), and sparse conditional MoE routing (Xue et al., 2021). Pruning is one mechanism among several, and some papers explicitly position their contribution against post-hoc pruning or distillation.

Finally, width-wise construction should be distinguished from unstructured subnetwork discovery. Methods such as Modular Risk Minimization (Zhang et al., 2021) and Subnetwork Ensembles (Whitaker, 2023) operate primarily through weight-level masking rather than selecting full neurons, channels, or expert branches. They are highly relevant to the broader study of subnetworks, but they are only partially width-wise in the strict architectural sense. This suggests a useful boundary: width-wise construction typically preserves structured within-layer units, whereas unstructured masking preserves only a sparse connection pattern.

Taken together, the literature presents width-wise subnetwork construction as a design space rather than a single method family. The dominant axes are the definition of width, the degree of nesting, the strength of parameter sharing, the presence or absence of adaptive aggregation, and the role of the resulting subnetwork family in inference, search, federated scheduling, distributed training, or parameter-efficient adaptation. Across these variants, the recurring contribution is to turn a single overparameterized model or sparse backbone into a source of multiple structurally aligned subnetworks that can be selected, trained, or adapted under differing computational or task constraints (Li et al., 2023).