Multi-Dilated Convolutions

• Multi-dilated convolutions are an extension of DenseNet connectivity, using a fixed window of previous layers to balance feature reuse and computational efficiency.
• This approach reduces parameter count and training time, achieving >90% accuracy on benchmarks with moderate values of the window size.
• It offers a flexible trade-off between full connectivity and plain convolution, making it ideal for resource-constrained applications such as mobile systems.

Multi-dilated convolutions (in the context of local or "windowed" dense connectivity) are a generalization of the connectivity pattern in standard DenseNet architectures. While the original DenseNet establishes full connectivity within each dense block by concatenating the outputs of all preceding layers to form the input for each subsequent layer, the multi-dilated or windowed approach restricts this to a fixed window of previous layers. This reduces parameter count and computational overhead while enabling efficient feature reuse and maintaining competitive predictive accuracy. The principle has been systematically developed and analyzed in the context of "Exploring Feature Reuse in DenseNet Architectures" (Hess, 2018).

1. Formal Definition and Mathematical Framework

In the classical DenseNet arrangement, for a block of $L$ convolutional layers with growth rate $k$, the transformation at layer $\ell$ can be written as:

$x_\ell = H_\ell([x_0, x_1, \ldots, x_{\ell-1}])$

Here, $x_0$ is the block input, $H_\ell(\cdot)$ denotes the BN–ReLU–Conv operation, and $[\cdot]$ is channel-wise concatenation. Multi-dilated (windowed) dense connectivity instead defines, for a window size $w$:

• $S_\ell = \{\max(0, \ell - w), \ldots, \ell - 1\}$
• $x_\ell = H_\ell([x_j]_{j \in S_\ell})$

The input to each layer is restricted to the previous $k$0 feature maps or fewer near the start of the block. When $k$1, the standard all-to-all DenseNet structure is recovered. When $k$2, it reduces to a plain stack of convolutions without skip connections. The input channel dimension evolves as:

• For $k$3, $k$4
• For $k$5, $k$6

The parameter count for each convolutional layer with $k$7 kernels:

$k$8

The total parameters in the block:

$k$9

plus those in any transition layers, which also decrease as $\ell$0 shrinks (Hess, 2018).

2. Architectural Implementation

A windowed dense block (WinDenseNet Block) is constructed by selecting the depth $\ell$1, window $\ell$2, and growth rate $\ell$3:

• The first layer receives the block input, applies BN–ReLU–Conv($\ell$4, $\ell$5).
• For $\ell$6, the last $\ell$7 outputs are concatenated, followed by BN–ReLU–Conv($\ell$8, $\ell$9) to produce $x_\ell = H_\ell([x_0, x_1, \ldots, x_{\ell-1}])$0.
• The block output is the concatenation of the last $x_\ell = H_\ell([x_0, x_1, \ldots, x_{\ell-1}])$1 feature maps.

Transition layers are identical to DenseNet but admit only the most recent $x_\ell = H_\ell([x_0, x_1, \ldots, x_{\ell-1}])$2 feature maps. Implementation leverages a rolling buffer (queue) tracking the last $x_\ell = H_\ell([x_0, x_1, \ldots, x_{\ell-1}])$3 outputs, facilitating efficient concatenation and layer-wise computation (Hess, 2018).

3. Parameter Efficiency and Empirical Results

Reducing window size $x_\ell = H_\ell([x_0, x_1, \ldots, x_{\ell-1}])$4 sharply decreases the model's parameter count and training time, with only modest accuracy loss for moderate values of $x_\ell = H_\ell([x_0, x_1, \ldots, x_{\ell-1}])$5. Table 1 (from (Hess, 2018)) illustrates this trade-off for a $x_\ell = H_\ell([x_0, x_1, \ldots, x_{\ell-1}])$6 ($x_\ell = H_\ell([x_0, x_1, \ldots, x_{\ell-1}])$7, $x_\ell = H_\ell([x_0, x_1, \ldots, x_{\ell-1}])$8) architecture on CIFAR-10:

Window $x_\ell = H_\ell([x_0, x_1, \ldots, x_{\ell-1}])$9	Parameters	Test Accuracy (%)
1	48.9K	68.15
4	205.6K	90.35
7	379.3K	91.61
10	570.2K	92.29
13 (full)	1.02M	92.65

Small windows ($x_0$0–$x_0$1) yield very compact, fast networks that underfit, whereas mid-range windows ($x_0$2–$x_0$3) achieve $x_0$490% accuracy with $x_0$5–$x_0$6% fewer parameters and training time. When the total parameter count is kept fixed by increasing growth rate $x_0$7 as $x_0$8 drops, WinDenseNet can exceed full DenseNet accuracy at intermediate $x_0$9. Conversely, for very small $H_\ell(\cdot)$0, DenseNet's full connectivity outperforms, but for $H_\ell(\cdot)$1, local-dense variants win for their parameter budget (Hess, 2018).

4. Feature Reuse Analysis and Theoretical Insights

Analysis of the normalized mean filter-weight magnitudes per source layer in WinDenseNet reveals distinct feature reuse patterns:

• For small $H_\ell(\cdot)$2 ($H_\ell(\cdot)$3), network layers emphasize earliest available feature maps, indicating a preference for long-range connections.
• At large $H_\ell(\cdot)$4 ($H_\ell(\cdot)$5), layers attend more to recent feature maps; distant connections are less influential, especially in deeper blocks.
• In the deepest block (Block 3), the average normalized attention to distant features declines as $H_\ell(\cdot)$6 grows.

This suggests that very long-range skip connections provide diminishing marginal utility in deeper portions of the network. Allocating parameters to a higher local growth rate $H_\ell(\cdot)$7 (enabled by smaller $H_\ell(\cdot)$8) and local dense connectivity leads to more efficient feature reuse than the all-to-all pattern of DenseNet (Hess, 2018).

5. Practical Recommendations and Configuration Guidelines

For deployment scenarios with strict parameter or FLOP constraints (e.g., mobile/embedded systems), it is recommended to select $H_\ell(\cdot)$9–$[\cdot]$0 and increase $[\cdot]$1 as much as model capacity allows. Full DenseNet connectivity ($[\cdot]$2) remains optimal when maximal accuracy is the sole objective and computational budget is ample; nonetheless, WinDenseNet architectures may surpass the original DenseNet at moderate budgets for the same total parameter count.

Implementation in TensorFlow or PyTorch should maintain a queue of the most recent $[\cdot]$3 feature maps for each block. For transition layers, use only outputs from the most recent $[\cdot]$4 feature maps, and optionally apply output compression as in the DenseNet transition.

6. Comparative Significance and Generalization

Multi-dilated (windowed) convolutions generalize the skip pattern in DenseNets, creating a spectrum between standard convolutional networks ($[\cdot]$5) and full DenseNets ($[\cdot]$6). This flexibility enables tailored trade-offs between memory/computational cost and predictive accuracy. Feature-reuse analysis supports that, for a substantial range of architectures and budgets, parameter-efficient local dense connectivity with appropriate growth rate is preferable to exhaustive long-range densification. A plausible implication is that other network families utilizing skip connections may likewise benefit from a systematic exploration of multi-dilated connection patterns (Hess, 2018).