ThriftyNets: Deep Learning with Shared Parameters

• ThriftyNets are deep learning architectures defined by maximal parameter sharing, where a single convolutional weight tensor is recursively applied to reduce model size.
• They incorporate shortcut histories and adaptive downsampling to balance expressivity with computational efficiency and parameter budget.
• Empirical results on benchmarks like CIFAR-10/100 show that ThriftyNets achieve competitive accuracy with significantly fewer parameters than traditional CNNs.

ThriftyNets denote a modern deep learning architecture class distinguished by maximal parameter sharing, achieving high classification accuracy with sharply reduced parameter counts. In the canonical ThriftyNet model, only a single convolutional layer weight tensor is defined, which is applied recursively across multiple processing steps, with additional mechanisms—nonlinearity, normalization, downsampling, and shortcut connections—ensuring sufficient expressivity. ThriftyNets depart from conventional practice, where network depth and width typically increase parameter count, instead leveraging recurrence and parameter factorization to attain competitive performance under severe parameter budget constraints (Coiffier et al., 2020). The term "ThriftyNet" also appears independently in the context of wireless system design, where it concerns resource-efficient networking by aligning physical throughput with application-level goodput, often through advanced coding and transport-layer design (Kim et al., 2012).

1. ThriftyNet Architecture and Parameter Factorization

The ThriftyNet architecture is defined by the recursive application of a single convolutional-layer weight tensor, $$\mathbf{W} \in \mathbb{R}^{f \times f \times a \times b}$$, across $T$ iterations. The input image, padded to match $f$ channels, is processed as

$$\mathbf{x}_0 = \mathrm{PAD}(\mathbf{x}) \in \mathbb{R}^{f \times H \times W}$$

and evolved via

$$\begin{aligned} \mathbf{a}_{t+1} &= \mathbf{W} \star \mathbf{x}_t, \ \mathbf{s}_{t+1} &= \mathbf{x}_t + \sigma(\mathbf{a}_{t+1}), \ \mathbf{y}_{t+1} &= \mathrm{BN}_t(\mathbf{s}_{t+1}), \ \mathbf{x}_{t+1} &= \mathcal{D}_t(\mathbf{y}_{t+1}), \end{aligned}$$

with ReLU/tanh activations, step-dependent BatchNorm (with $2f$ affine parameters per step), and optional downsampling $\mathcal{D}_t$. Parameter factorization yields a convolutional-layer budget of only $f^2ab$ (ungrouped), maintained constant across all processing steps.

2. Residual Connections and Shortcut History

Expressivity and trainability are further enhanced via shortcut histories. For history length $h$, learnable scalars $\alpha_{t,i}$ combine up to $T$0 past activations at each step: $T$1 This incurs only $T$2 additional scalar parameters. Ablations show that learning real-valued $T$3 is essential: hard-thresholding to binarized values degrades accuracy by 1–2 points.

3. Evolution of Feature Geometry and Parameter Budget

Throughout, the channel count is kept constant at $T$4; spatial resolution evolves as

$T$5

where downsampling ratios $T$6 are scheduled as needed. Downsampling can be applied regularly or irregularly, allowing explicit control over the MACs (multiply-accumulate operations) and feature map size. The full parameter count (excluding grouped convolutions) is: $T$7 Grouped/depthwise convolutions further reduce the dominant term to $T$8.

4. Training and Inference Procedures

ThriftyNet models are trained with cross-entropy loss, SGD (learning rate $T$9, momentum $f$0), no explicit weight decay, and a step-down schedule at epochs $f$1, $f$2, and $f$3 for a total of $f$4 epochs. Data augmentation via random cropping and horizontal flipping is found to deliver most of the attainable gains; more aggressive augmentation (AutoAugment, mixup, cutmix, cutout) yields marginal or negative returns, as shown in the table below:

Augmentation	Test Accuracy (%)
standard (flip + crop)	90.64
+ auto augment	91.00
+ cutout (size=8)	90.40
+ mixup	88.09
+ cutmix	88.47

Inference involves sequentially running all $f$5 recursions; resource/accuracy trade-offs can be managed by adjusting $f$6 or pooling schedules.

5. Empirical Performance and Ablation Analyses

Results on CIFAR-10/100 establish that ThriftyNet achieves parity with much larger networks at a fraction of their parameter budget. For CIFAR-10, with approximately $f$7 parameters:

• ThriftyNet ($f$8, $f$9): $$\mathbf{x}_0 = \mathrm{PAD}(\mathbf{x}) \in \mathbb{R}^{f \times H \times W}$$0\%
• Residual ThriftyNet ($$\mathbf{x}_0 = \mathrm{PAD}(\mathbf{x}) \in \mathbb{R}^{f \times H \times W}$$1, $$\mathbf{x}_0 = \mathrm{PAD}(\mathbf{x}) \in \mathbb{R}^{f \times H \times W}$$2): $$\mathbf{x}_0 = \mathrm{PAD}(\mathbf{x}) \in \mathbb{R}^{f \times H \times W}$$3\%
• ResNet-20: $$\mathbf{x}_0 = \mathrm{PAD}(\mathbf{x}) \in \mathbb{R}^{f \times H \times W}$$4\% (using $$\mathbf{x}_0 = \mathrm{PAD}(\mathbf{x}) \in \mathbb{R}^{f \times H \times W}$$5 params)

For CIFAR-100:

• Residual ThriftyNet ($$\mathbf{x}_0 = \mathrm{PAD}(\mathbf{x}) \in \mathbb{R}^{f \times H \times W}$$6, $$\mathbf{x}_0 = \mathrm{PAD}(\mathbf{x}) \in \mathbb{R}^{f \times H \times W}$$7): $$\mathbf{x}_0 = \mathrm{PAD}(\mathbf{x}) \in \mathbb{R}^{f \times H \times W}$$8\% ($$\mathbf{x}_0 = \mathrm{PAD}(\mathbf{x}) \in \mathbb{R}^{f \times H \times W}$$9 parameters)
• DenseNet-BC (L=100, K=12): $$\begin{aligned} \mathbf{a}_{t+1} &= \mathbf{W} \star \mathbf{x}_t, \ \mathbf{s}_{t+1} &= \mathbf{x}_t + \sigma(\mathbf{a}_{t+1}), \ \mathbf{y}_{t+1} &= \mathrm{BN}_t(\mathbf{s}_{t+1}), \ \mathbf{x}_{t+1} &= \mathcal{D}_t(\mathbf{y}_{t+1}), \end{aligned}$$0\% ($$\begin{aligned} \mathbf{a}_{t+1} &= \mathbf{W} \star \mathbf{x}_t, \ \mathbf{s}_{t+1} &= \mathbf{x}_t + \sigma(\mathbf{a}_{t+1}), \ \mathbf{y}_{t+1} &= \mathrm{BN}_t(\mathbf{s}_{t+1}), \ \mathbf{x}_{t+1} &= \mathcal{D}_t(\mathbf{y}_{t+1}), \end{aligned}$$1 parameters)
• Wide-ResNet-16-8: $$\begin{aligned} \mathbf{a}_{t+1} &= \mathbf{W} \star \mathbf{x}_t, \ \mathbf{s}_{t+1} &= \mathbf{x}_t + \sigma(\mathbf{a}_{t+1}), \ \mathbf{y}_{t+1} &= \mathrm{BN}_t(\mathbf{s}_{t+1}), \ \mathbf{x}_{t+1} &= \mathcal{D}_t(\mathbf{y}_{t+1}), \end{aligned}$$2\% ($$\begin{aligned} \mathbf{a}_{t+1} &= \mathbf{W} \star \mathbf{x}_t, \ \mathbf{s}_{t+1} &= \mathbf{x}_t + \sigma(\mathbf{a}_{t+1}), \ \mathbf{y}_{t+1} &= \mathrm{BN}_t(\mathbf{s}_{t+1}), \ \mathbf{x}_{t+1} &= \mathcal{D}_t(\mathbf{y}_{t+1}), \end{aligned}$$3 parameters)

Ablations show moderate sensitivity to $$\begin{aligned} \mathbf{a}_{t+1} &= \mathbf{W} \star \mathbf{x}_t, \ \mathbf{s}_{t+1} &= \mathbf{x}_t + \sigma(\mathbf{a}_{t+1}), \ \mathbf{y}_{t+1} &= \mathrm{BN}_t(\mathbf{s}_{t+1}), \ \mathbf{x}_{t+1} &= \mathcal{D}_t(\mathbf{y}_{t+1}), \end{aligned}$$4 in the $$\begin{aligned} \mathbf{a}_{t+1} &= \mathbf{W} \star \mathbf{x}_t, \ \mathbf{s}_{t+1} &= \mathbf{x}_t + \sigma(\mathbf{a}_{t+1}), \ \mathbf{y}_{t+1} &= \mathrm{BN}_t(\mathbf{s}_{t+1}), \ \mathbf{x}_{t+1} &= \mathcal{D}_t(\mathbf{y}_{t+1}), \end{aligned}$$5 range (for fixed budget), clear gains from increased downsamplings, and strong dependence of accuracy on number of filters $$\begin{aligned} \mathbf{a}_{t+1} &= \mathbf{W} \star \mathbf{x}_t, \ \mathbf{s}_{t+1} &= \mathbf{x}_t + \sigma(\mathbf{a}_{t+1}), \ \mathbf{y}_{t+1} &= \mathrm{BN}_t(\mathbf{s}_{t+1}), \ \mathbf{x}_{t+1} &= \mathcal{D}_t(\mathbf{y}_{t+1}), \end{aligned}$$6. For example, increasing $$\begin{aligned} \mathbf{a}_{t+1} &= \mathbf{W} \star \mathbf{x}_t, \ \mathbf{s}_{t+1} &= \mathbf{x}_t + \sigma(\mathbf{a}_{t+1}), \ \mathbf{y}_{t+1} &= \mathrm{BN}_t(\mathbf{s}_{t+1}), \ \mathbf{x}_{t+1} &= \mathcal{D}_t(\mathbf{y}_{t+1}), \end{aligned}$$7 from $$\begin{aligned} \mathbf{a}_{t+1} &= \mathbf{W} \star \mathbf{x}_t, \ \mathbf{s}_{t+1} &= \mathbf{x}_t + \sigma(\mathbf{a}_{t+1}), \ \mathbf{y}_{t+1} &= \mathrm{BN}_t(\mathbf{s}_{t+1}), \ \mathbf{x}_{t+1} &= \mathcal{D}_t(\mathbf{y}_{t+1}), \end{aligned}$$8 ($$\begin{aligned} \mathbf{a}_{t+1} &= \mathbf{W} \star \mathbf{x}_t, \ \mathbf{s}_{t+1} &= \mathbf{x}_t + \sigma(\mathbf{a}_{t+1}), \ \mathbf{y}_{t+1} &= \mathrm{BN}_t(\mathbf{s}_{t+1}), \ \mathbf{x}_{t+1} &= \mathcal{D}_t(\mathbf{y}_{t+1}), \end{aligned}$$9k params) to $2f$0 ($2f$1k params) on CIFAR-10 boosts accuracy from $2f$2 to $2f$3.

Iteration $2f$4	ThriftyNet Acc. (%)	Residual Acc. (%)
10	89.2	89.3
20	90.47	90.71
45	90.18	90.95

Downsamplings $2f$5	ThriftyNet Acc. (%)	Residual Acc. (%)
1	89.10	90.62
4	90.93	91.69

6. Computational and Design Trade-offs

ThriftyNets allow fine-grained adjustment between parameter count, $2f$6, and computational cost (MACs) to target specific hardware profiles. Pooling can be front-loaded to halve computational cost at a tolerable accuracy loss (4–5 points). Grouped convolutions are available to further shrink the parameter budget. The principal trade-off is between parameter reuse (T), accuracy, and computational efficiency.

7. Broader Context and Related Work

ThriftyNet's maximal parameter factorization contrasts with deepening/widening used in standard CNNs and may be viewed as an instance of recurrent convolutional networks with weight sharing. The architecture is distinct from parameter-pruning or quantization approaches, as it operates directly through strict model weight reuse. The term "ThriftyNet" is also encountered in wireless networking literature—there, a "ThriftyNet" is defined as one whose infrastructure is dimensioned by the user-perceived goodput, not peak physical throughput, and exploits coding techniques such as TCP/NC to maintain user-level performance while minimizing physical plant and energy costs. The key principle in both contexts is resource efficiency: in neural networks, via maximal parameter reuse; in wireless networks, via protocols that close the gap between throughput and goodput, minimizing required infrastructural spend (Coiffier et al., 2020, Kim et al., 2012).