Learnable DDC (DCLS-DDC): Efficient Dilated Convolution

• Learnable DDC (DCLS-DDC) is a convolution operator that parameterizes kernel tap positions as continuous, learnable offsets, enabling flexible sampling geometries.
• It employs differentiable interpolation methods like bilinear and Gaussian to enlarge the effective receptive field without increasing parameter count.
• Empirical evaluations show improved accuracy, robustness, and interpretability across computer vision, audio processing, and spiking neural network tasks.

Dilated Convolution with Learnable Spacings (DCLS), also widely referred to in the literature as DCLS-DDC, is a convolutional operator that parametrizes the spatial or temporal positions of each nonzero element ("tap") in a convolutional kernel as continuous, learnable offsets. These offsets are optimized end-to-end via backpropagation and handled by differentiable interpolation schemes, such as bilinear or Gaussian kernels, to support non-integer sampling locations. DCLS-DDC directly generalizes both standard dilated and dense convolution, enabling efficient enlargement of the effective receptive field without increasing parameter count or imposing a rigid sampling grid. The DCLS methodology has been validated across computer vision, audio, and neuromorphic computing domains, showing consistent improvements in accuracy and robustness at fixed model size, as well as enhanced alignment between model saliency maps and human visual attention.

1. Mathematical Formulation and Interpolation Mechanisms

A DCLS layer defines the convolutional kernel as a superposition of $M$ nonzero elements ("points"), each parameterized by a scalar weight $w_m$ and a continuous position $p_m = (p^x_m, p^y_m)$ (2D) or $p_m = (p^t_m)$ (1D). The core mechanism is as follows:

• The "spokes" of the kernel are assigned learnable offsets $\Delta p_m$ that are (in general) non-integer.
• At any input location, the DCLS output is computed via

$$y[\mathbf{i}] = \sum_{m=1}^M w_m \cdot I(\mathbf{i} + p_m)$$

where $I(\cdot)$ denotes sampling at fractional locations, handled by interpolation.

Interpolation choices:

• Bilinear (triangle): Fractional positions are distributed to the four closest integer grid points according to the bilinear formula:

$$I(x+\Delta x, y+\Delta y) = (1-\alpha)(1-\beta) I(x+p, y+q) + \ldots$$

for $\Delta x = p+\alpha$, $\Delta y = q+\beta$ with $w_m$0, $w_m$1.

• Gaussian: Each point spreads via a normalized Gaussian kernel of learnable (or fixed) width $w_m$2, supporting more global and smooth influence.

Gradient computation: The interpolation operation is differentiable, enabling joint optimization of weights and offsets. Explicit closed-form expressions for $w_m$3 and $w_m$4 exist for both interpolation types, ensuring compatibility with standard deep learning frameworks (Khalfaoui-Hassani et al., 2021, Khalfaoui-Hassani et al., 2023, Chamas et al., 2024).

2. Training, Initialization, and Regularization Strategies

Parameter learning: In DCLS-DDC, weights $w_m$5, positional offsets $w_m$6, and, for Gaussian DCLS, widths $w_m$7 (if learnable) are optimized via standard backpropagation, typically without any auxiliary loss terms on the offsets. The primary training objective is the task-specific loss (e.g., classification cross-entropy).

Initialization protocols:

• Positions are initialized to a uniform grid with small random perturbations or sampled from a narrow Gaussian centered in the kernel support.
• Learning rates for offsets are usually multiplied by a factor (e.g., $w_m$8) relative to weights, accelerating spatial adaptation.
• For Gaussian DCLS, width $w_m$9 may be initialized higher early in training and annealed to a lower value.

Regularization:

• Clamp offsets to the kernel support after each update to prevent drift.
• No weight decay is applied to offsets or widths.
• An optional repulsive loss (to avoid overlap of positions) and stage-wise offset sharing are sometimes deployed but yield only marginal improvements (Khalfaoui-Hassani, 2024).

3. Algorithmic Implementation and Integration

DCLS-DDC is implemented as a drop-in replacement for depthwise-separable convolutions in CNNs or related layers in spiking and audio networks. The process consists of:

• At the start of each forward pass, constructing the effective dense kernel by interpolating each tap onto the full support.
• Efficient kernel construction is achieved via vectorized routines or custom CUDA kernels, especially with grouped or synchronized offsets ($p_m = (p^x_m, p^y_m)$0) per layer or per stage.
• During backpropagation, automatic differentiation propagates gradients through both the interpolation and convolution steps.

A typical high-level pseudocode for 2D DCLS in PyTorch is:

$p_m = (p^t_m)$5

Alignment with existing deep learning frameworks is achieved via in-place substitutions in architectures such as ResNet, ConvNeXt, ConvFormer, FastViT, or as a component in temporal and spiking neural networks (Chamas et al., 2024, Khalfaoui-Hassani et al., 2023).

4. Empirical Evaluation across Vision, Audio, and Spiking Domains

Computer Vision:

• DCLS-DDC provides consistent accuracy gains for image classification on ImageNet-1k. For ConvNeXt-T, substituting DSC with DCLS-Gauss in a 23×23 support (26 taps) increases top-1 accuracy from $p_m = (p^x_m, p^y_m)$1 to $p_m = (p^x_m, p^y_m)$2 (Khalfaoui-Hassani et al., 2023, Chamas et al., 2024, Khalfaoui-Hassani, 2024). For ResNet-50, DCLS achieves top-1 accuracy of $p_m = (p^x_m, p^y_m)$3 ($p_m = (p^x_m, p^y_m)$4 over 3×3 baseline) at constant parameter count (Khalfaoui-Hassani et al., 2021).
• Robustness enhancements are observed on ImageNet-C/A/R/Sketch: clean error rates improve and mIoU rises in semantic segmentation downstream benchmarks (e.g., ADE20K: ConvNeXt-T $p_m = (p^x_m, p^y_m)$5) (Khalfaoui-Hassani, 2024).
• Marginal reduction in throughput (e.g., ConvNeXt-T: $p_m = (p^x_m, p^y_m)$6 img/s) due to interpolation overhead, but parameter count remains unchanged.

Audio and Spiking Neural Networks:

• On AudioSet, DCLS-DDC integrated in ConvNeXt-T/ConvFormer-S18/FastViT-SA24 yields absolute mean average precision improvements of $p_m = (p^x_m, p^y_m)$7 to $p_m = (p^x_m, p^y_m)$8 mAP versus DSC, with a throughput cost of $p_m = (p^x_m, p^y_m)$9 (Khalfaoui-Hassani et al., 2023).
• In temporal SNNs, DCLS-style learnable delays (1D DCLS) enable state-of-the-art classification for SHD, SSC benchmarks with models such as DCLS-delays (3L,2KC) reaching $p_m = (p^t_m)$0 accuracy with orders of magnitude fewer parameters than dense convolutional delay models (Khalfaoui-Hassani, 2024).

5. Analysis: Why Learnable Spacings Advance Convolutional Modeling

• By allowing the kernel taps to drift off a rigid grid, DCLS-DDC adaptively concentrates sampling in task-relevant regions (e.g., salient image parts, harmonic structure in audio, optimal synaptic delays).
• Visualization of learned offsets reveals data-dependent, anisotropic “attractor patterns” that expand the effective receptive field without overparameterization (Khalfaoui-Hassani et al., 2021, Khalfaoui-Hassani et al., 2023).
• Gaussian interpolation provides smoother, more stable gradient flow than triangle-based schemes, facilitating convergence and slightly improving task accuracy (statistically significant, $p_m = (p^t_m)$1 on ImageNet) (Khalfaoui-Hassani et al., 2023).
• No increase in parameter count is incurred as offsets/widths are stage-shared and occupy a negligible fraction of the model footprint.

6. Interpretability and Human Alignment

DCLS-DDC not only improves accuracy but enhances the human resemblance of model attention, as measured by the alignment between model-generated Grad-CAM heatmaps and human visual attention maps on the ClickMe dataset (Chamas et al., 2024). Replacing standard convolutions with DCLS layers increased the Spearman correlation between model and human saliency maps in seven of eight tested architectures (e.g., ConvNeXt-Base: $p_m = (p^t_m)$2, ResNet-50: $p_m = (p^t_m)$3 under Grad-CAM). The introduction of Threshold-Grad-CAM, a variant that thresholds Grad-CAM maps post-normalization, further accentuated these gains.

Summary Table of Results from (Chamas et al., 2024):

Model	Top-1	Grad-CAM	Thr-Grad-CAM
convnext_base	83.83	0.7565	0.7572
convnext_base_dcls	84.09	0.7979	0.7845
resnet50	77.84	0.6135	0.7125
resnet50_dcls	78.35	0.6252	0.7261

The improved interpretability is attributed to more flexible and data-aligned sampling patterns, offering a tangible connection between network computations and human vision strategies.

7. Limitations and Prospects

While DCLS-DDC consistently provides modest but reliable performance improvements at iso-parameters, several trade-offs and open challenges remain:

• Inference Overhead: Kernel construction and on-the-fly interpolation increase per-batch compute cost by $p_m = (p^t_m)$4 relative to DSC, especially in large or high-dimensional settings. This remains manageable for most practical deployments, especially on modern GPU hardware (Khalfaoui-Hassani et al., 2023).
• Architecture Search: All reported results reflect drop-in substitution for existing convolutions; a plausible implication is that networks architected natively for DCLS (via neural architecture search) may offer further gains (Khalfaoui-Hassani, 2024).
• Large-scale/3D Modeling: Efficient kernel construction for very large supports (e.g., in 3D or with high tap count) may require further algorithmic optimization, including sparse matrix multiplications.
• Cross-task Generalization: Although improvements are systematic across classification, segmentation, audio tagging, and SNNs, results in domains with extremely irregular data (e.g., video, point cloud) remain to be established.

Continued evolution of DCLS-DDC is anticipated toward hybrid attention–convolutional frameworks, dedicated DCLS-based architectures, and further studies into its relationship with human and animal biologic perception (Khalfaoui-Hassani, 2024, Khalfaoui-Hassani et al., 2023, Chamas et al., 2024).