---
title: Nested Sparse Networks Overview
url: https://www.emergentmind.com/topics/nested-sparse-networks-nestednet
type: topic
---

# Nested Sparse Networks Overview

Nested Sparse Networks (NestedNet) are a family of deep neural network architectures designed to embed multiple sparsity-level subnetworks within a single shared parameterization. These subnetworks are arranged such that their parameter supports are strictly nested, enabling resource-adaptive inference, multi-granularity prediction, and efficient knowledge transfer. By imposing structured sparsity constraints through learned masks, channel or layer scheduling, or recursive parameter sharing, NestedNets achieve a spectrum of accuracy-compute trade-offs and application scenarios ranging from deep compression to hierarchical classification and edge inference.

## 1. Formal Definition and Nested Sparsity Principle

A Nested Sparse Network of depth $L$ contains $L$ internal subnetworks labeled by $\ell=1,\ldots,L$, each associated with a target sparsity ratio $r_\ell$ satisfying $0 < r_1 < r_2 < \cdots < r_L = 1$. The weights retained at level $\ell$ are a subset of those at level $\ell+1$, enforcing a strictly nested structure:
\[
\mathrm{supp}(M^{(1)}) \subseteq \mathrm{supp}(M^{(2)}) \subseteq \cdots \subseteq \mathrm{supp}(M^{(L)}),
\]
where $M^{(\ell)} \in \{0,1\}^d$ are binary masks over weights $W \in \mathbb{R}^d$ [1712.03781, 2203.03324]. A subnetwork at level $\ell$ uses $W^{(\ell)} = W \odot M^{(\ell)}$, with $\odot$ the element-wise product, resulting in each higher-level subnetwork containing all the parameters of the smaller ones.

A related but structurally distinct variant is the height-$s$ NestNet, which recursively defines scalar activation functions of each neuron as a nested network of height $\le s-1$, yielding parameter-efficient yet more expressive architectures [2205.09459].

## 2. Mathematical Formulation

The joint training objective simultaneously optimizes all subnetworks on the same dataset, incorporating regularization:
\[
\min_{W} \sum_{\ell=1}^L \mathcal{L}(f(X; W \odot M^{(\ell)}), Y) + \lambda \mathcal{R}(W, \{M^{(\ell)}\})
\]
subject to the nesting constraint on masks. $\mathcal{L}$ can be cross-entropy or other prediction losses, and $\mathcal{R}$ is typically weight decay or structured sparsity penalty [1712.03781].

In weight-level mask learning, a smooth approximation $\sigma$ replaces the nondifferentiable threshold for creating binary masks:
\[
M(W) = \sigma(|W| - \tau), \quad \sigma(x) \approx \frac{\tanh(\gamma x_+)}{\tanh(\gamma x_+)+1}
\]
with nested masks built by increasing the threshold $\tau$ in steps, ensuring $\mathrm{supp}(M^{(l)}) \subseteq \mathrm{supp}(M^{(l+1)})$ [1712.03781].

For the training of nested sparse ConvNets, a gradient-masking technique is utilized. Each sparse subnetwork $W^{(k)}$ receives its own masked gradient, with the full update given by:
\[
\hat{G} = G^{(0)} + \sum_{k=1}^N M_k \odot G^{(k)}
\]
where $G^{(0)}$ is the dense gradient and $G^{(k)}$ is the gradient from subnetwork $k$ [2203.03324].

## 3. Training Algorithms and Implementation Strategies

### a) Weight, Channel, and Layer Scheduling

- **Weight-level pruning:** Apply a series of thresholds $\tau_L < \tau_{L-1} < \cdots < \tau_1$ to construct nested supports, stopping updates for pruned weights.
- **Channel and layer scheduling:** Predefine, for each level, subsets of input/output channels and layers, typically arranged so that blocks of parameters corresponding to lower-sparsity levels are reused by higher ones. Formally, channel/nesting is implemented in block form:
  \[
  W_l^{(\ell)} = \begin{bmatrix}
      W_l^{(1,1)} & \cdots & W_l^{(1,\ell)} \\
      \vdots & \ddots & \vdots \\
      W_l^{(\ell,1)} & \cdots & W_l^{(\ell,\ell)}
  \end{bmatrix}
  \]
  where $W_l^{(i,j)}$ are block parameters [1712.03781].

- **Gradient masking and prune-while-training:** During training, masks are updated and each subnetwork’s masked gradients are accumulated for parameter updates, maintaining synchronization across sparsity levels [2203.03324].

### b) Nested Compression Formats

To efficiently store and deploy NestedNets, formats such as NestedCSR concatenate representations of all $N$ sub-networks using block-CSR layouts. The total storage is typically governed by the least-sparse supported level [2203.03324].

## 4. Resource-Aware Inference and Applications

NestedNets provide anytime, resource-aware inference capabilities. At runtime, the highest subnetwork whose compute or latency fits the available budget is selected without any need for model reloading or reinitialization. Notable application scenarios include:

- **Adaptive Deep Compression:** A single NestedNet supports multiple compression ratios (e.g., 2×, 3×), often yielding negligible accuracy drops at moderate compression.
- **Joint Knowledge Distillation:** Subnetworks at varying capacities act as student models, enabling knowledge transfer among them without explicit teacher-student separation.
- **Hierarchical (Coarse-to-Fine) Classification:** Levels in the NestedNet are associated with semantic granularity; e.g., super-class vs. subclass prediction is performed using shared representations.
- **Edge Deployment for Tiny Devices:** On MCUs, NestedNets enable dynamic trade-offs between latency, accuracy, and memory footprint, while efficient sparse inference kernels ensure minimal overhead.

## 5. Expressivity and Theoretical Properties

The recursively nested architectures (“NestNets of height $s$”) demonstrate super-approximation power. For 1-Lipschitz functions $f:[0,1]^d\to\mathbb{R}$, there exists a height-$s$ NestNet $\phi$ with $\mathcal{O}(n)$ parameters achieving uniform error:
\[
\|\,\phi - f\,\|_{L^\infty([0,1]^d)} \le 7\sqrt{d} \, n^{-(s+1)/d}
\]
In contrast, standard width–depth networks optimally achieve error $\Theta(n^{-2/d})$, so higher NestNet height $s$ delivers exponentially faster approximation rates in parameter count [2205.09459]. The nesting grants sparse parameter-sharing: sub-NestNets may be reused within and across layers, resulting in tree-structured networks with deep composition—substantial expressivity without parameter explosion.

## 6. Empirical Evidence and Performance Analysis

Nested Sparse Networks have been extensively evaluated:

- **ResNet-56 on CIFAR-10 compression:** 2× structured pruning yields 91.8% vs. 92.9% (NestedNet, channel-pruned). Unstructured 3× weight pruning achieves 92.6% (independent), 92.8% (NestedNet).
- **Wide-Residual Networks, CIFAR-10/100:** Four-level NestedNet achieves nearly all the accuracy of separately trained models; consensus heads further boost accuracy.
- **Edge deployment (CIFAR-10, CIFAR-100, PASCAL VOC):** On ARM-M7, NestedCSR achieves nearly the same latency as the equivalent single-sparsity model (e.g., 1.8% faster at 70% sparsity for ResNet9, only 10.9% slower on MobileNetV1 at the same density but still 30–50% faster than dense).
- **Pareto frontier:** Across deployable points, NestedNets lie strictly above slimmable/dynamically pruned networks in the accuracy–latency plane, supporting more configurations within a fixed storage budget [2203.03324].

## 7. Extensions and Related Architectures

Variants such as the Doubly Nested Network (DNNet) embed both layer-wise and channel-wise slicing within a single convolutional architecture. Channels are organized by topological sorting, and convolutional layers are restricted by channel-causal masks, enabling any subnetwork corresponding to specified (depth, width) to be sliced out without retraining. All submodels are supervised during training via a bank of classifier heads, enabling deployment with guaranteed accuracy-resource trade-offs across a two-dimensional budget grid [1806.07568].

The recursive height dimension of NestNets [2205.09459] takes sparsity and nesting from the parameter/mask level to an architectural meta-level, further increasing representational power through sparse, shared compositional activations.

---

Nested Sparse Networks thus provide a comprehensive architectural principle for achieving multi-level parameter sharing, resource elasticity, and deployability in deep neural networks, with both theoretical and practical state-of-the-art guarantees across inference, compression, and expressivity domains [1712.03781, 2203.03324, 2205.09459, 1806.07568].

Source: https://www.emergentmind.com/topics/nested-sparse-networks-nestednet