CoDA-Nets: Dynamic Convolutional Alignment

• CoDA-Nets are neural architectures that integrate dynamic alignment mechanisms into convolutional operations to adaptively handle spatial and temporal distortions.
• They employ DTW and dynamic alignment units to generate input-dependent filters, yielding non-linear, task-adaptive convolutions with explicit feature attribution.
• Empirical evaluations on vision and time-series tasks demonstrate that CoDA-Nets boost accuracy and produce crisp contribution maps compared to standard CNNs.

Convolutional Dynamic Alignment Networks (CoDA-Nets) are a family of neural network architectures that introduce dynamic, input-dependent alignment mechanisms into convolutional layers. These models, developed across multiple lines of research, aim to address both the robustness of convolutional feature extraction under temporal or spatial distortions and the need for faithful, high-resolution interpretability of model predictions. At their core, CoDA-Nets utilize mechanisms such as Dynamic Time Warping (DTW) and learned dynamic weight generators to realign convolutional filters to input data, yielding non-linear, task-adaptive convolutions with explicit feature attribution capabilities (Iwana et al., 2017, Böhle et al., 2021, Shulman, 2019, Böhle et al., 2021).

1. Architectural Foundations

CoDA-Nets depart from standard convolutional neural networks (CNNs) by incorporating dynamic alignment in each convolutional layer. Rather than performing a rigid, linear inner product between a fixed filter $\mathbf{w}$ and a local input window $\mathbf{x}_j$, CoDA-Nets compute a filter—either by warping the alignment (using DTW) or by generating input-dependent weights (via Dynamic Alignment Units, DAUs)—that best matches the structure of the input.

There are two main instantiations:

• DTW-based CoDA-Net: For each receptive field, DTW is applied between $\mathbf{w}$ and $\mathbf{x}_j$ to find the warping path that minimizes alignment cost (typically $\ell_2$ distance), after which the filter and input are combined along this path (Iwana et al., 2017, Shulman, 2019).
• DAU-based CoDA-Net: Each convolutional filter's weight vector is a function of the local input patch $x$: $w(x) = g(ABx + b)$, where $A$, $B$, and $b$ are learned parameters, $\mathbf{x}_j$0 is a norm-constraining nonlinearity (e.g., L2-normalization or a "squash" function), and the output is $\mathbf{x}_j$1 (Böhle et al., 2021, Böhle et al., 2021).

A deep CoDA-Net stacks such dynamically aligned convolutional layers, optionally followed by fully-connected and softmax output layers for classification.

2. Mathematical Underpinnings

DTW-based Alignment

For filter $\mathbf{x}_j$2 and input window $\mathbf{x}_j$3, construct a recurrence:

$\mathbf{x}_j$4

$\mathbf{x}_j$5

Backtracking from $\mathbf{x}_j$6 yields the warping path $\mathbf{x}_j$7 as matched index pairs. The dynamically aligned convolution at position $\mathbf{x}_j$8 is

$\mathbf{x}_j$9

This is integrated into the convolutional layer, replacing the static inner product with a non-linear, warp-invariant operation (Iwana et al., 2017).

DAU-based Dynamic Weight Generation and Linearity

Each DAU computes

$\mathbf{w}$0

where $\mathbf{w}$1 is a normalization, e.g., $\mathbf{w}$2 or a squashed variant. Stacking DAU-based convolutional layers, the entire network is dynamically linear: for input $\mathbf{w}$3, final logit $\mathbf{w}$4. This property enables explicit decomposition of the network output into per-input-component contributions (Böhle et al., 2021, Böhle et al., 2021).

3. Interpretability and Contribution Maps

A defining property of DAU-based CoDA-Nets is the inherent linear decomposition of outputs. For each class $\mathbf{w}$5 and input $\mathbf{w}$6, contributions can be computed as $\mathbf{w}$7 and arranged spatially to form a contribution map or heatmap. Because the weights are dynamically aligned, these maps are crisp and highly class-discriminative (Böhle et al., 2021, Böhle et al., 2021).

Interpretability is assessed via:

• Pointing-game localization: Fraction of attribution correctly localized to an object (Böhle et al., 2021, Böhle et al., 2021).
• Pixel-removal experiments: Model class confidence is tracked under removal of least and most important pixels; faithful attributions manifest as stability or rapid collapse, respectively.

Empirically, CoDA-Net contribution maps outperform post-hoc attribution techniques (e.g., RISE, Grad, GradCAM, Integrated Gradients, DeepLIFT) both quantitatively and qualitatively, as measured on vision benchmarks such as CIFAR-10 and TinyImagenet (Böhle et al., 2021, Böhle et al., 2021).

4. Empirical Performance and Benchmarks

CoDA-Nets have been evaluated on a range of classification tasks:

• Time-series: DTW-based CoDA-Nets yield superior accuracy under temporal deformations (e.g., Unipen, UCI Spoken Arabic Digit, Activities of Daily Life). Example: On Unipen 1a/1b/1c, CoDA-Net achieved 98.54/96.08/95.92% versus CNN 98.08/94.67/95.33% and LSTM 96.84/92.31/89.79% (Iwana et al., 2017). Gains of $\mathbf{w}$8 to $\mathbf{w}$9 over CNN baselines are typical (Shulman, 2019).
• Vision: DAU-based CoDA-Nets reach parity or exceed conventional CNNs. For instance, XL-CoDA (with squash nonlinearity) on TinyImagenet surpassed VGG16 and ResNet34 and closely matched ResNet-110, with 54.4% top-1 accuracy (no augmentation) and 58.4% (with data augmentation) (Böhle et al., 2021, Böhle et al., 2021).

A concise performance table:

Model	CIFAR-10	TinyImagenet
ResNet-56	93.4%	–
VGG-16	93.7%	–
XL-CoDA-SQ	93.4%	54.4%
XL-CoDA-SQ+aug	–	58.4%

DAU-based CoDA-Nets are also competitive with earlier interpretable models such as SENN and scalable to hybrid ImageNet architectures (Böhle et al., 2021).

5. Training, Hyperparameters, and Complexity

Configuration and training protocols depend on CoDA-Net instantiation:

• Hyperparameters (DTW): Alignment mode (“symmetric”, “x onto w”, “w onto x”); warping window radius $\mathbf{x}_j$0 (typically 5–15% of filter length); choice of cost normalization; optional slope constraints (Shulman, 2019). For small $\mathbf{x}_j$1, computational cost is manageable.
• Hyperparameters (DAU): Filter rank $\mathbf{x}_j$2 controls the expressivity/interpretability tradeoff; normalization function choice; learning rate, regularization (e.g., $\mathbf{x}_j$3 on final maps), and data augmentation (Böhle et al., 2021).
• Complexity (DTW): For $\mathbf{x}_j$4 filters of length $\mathbf{x}_j$5, signal length $\mathbf{x}_j$6, and stride $\mathbf{x}_j$7:
  • Standard Conv1D: $\mathbf{x}_j$8
  • CoDA-Net (DTW): $\mathbf{x}_j$9

CPU runtime: 0.03–0.09 s (CNN) vs. 0.08–0.40 s (CoDA-Nets) per sample (Iwana et al., 2017).

• Implementation: PyTorch or TensorFlow custom CoDA kernel, with DP loop for DTW, patch gather for DAU, and backtracking for gradient routing. Memory overhead is modest (few MB per sample for typical network and batch sizes) (Shulman, 2019, Böhle et al., 2021).

6. Extensions, Hybrid Models, and Scalability

CoDA-Nets can be extended in various directions:

• Hybrid architectures: Integrate CoDA layers atop standard CNN backbones, e.g., swapping final ResNet blocks for CoDA units. This increases the interpretable depth—the number of layers for which explicit, task-aligned contribution maps can be computed—without sacrificing accuracy (Böhle et al., 2021).
• Multichannel and multivariate inputs: CoDA layers are directly applicable to multivariate time series and multichannel imagery by per-channel dynamic alignment (Shulman, 2019).
• Soft-DTW and learnable warping: Replace hard warping by soft-DTW, or regularize alignment to encourage appropriate deformations (Shulman, 2019).
• 2D warping and attention: Extension to local 2D warping and integration with attention mechanisms has been proposed (Shulman, 2019).

Empirical findings indicate that dynamic alignment is critical not just during inference but throughout training: ablation studies show that applying CoDA only during inference, after standard CNN training, degrades performance (Shulman, 2019).

7. Significance and Context

CoDA-Nets address two central limitations of standard CNNs: sensitivity to localized temporal or spatial distortions, and the opaqueness of their internal decision process. By explicitly aligning convolutional filters to task-relevant patterns—through either non-parametric DTW or learned DAUs—these architectures deliver both improved generalization under deformations and model-inherent, high-fidelity attribution maps.

These properties position CoDA-Nets as both robust sequence/image classifiers and compelling architectures for domains where interpretability is paramount and data exhibit non-trivial deformations. They have established competitive baselines on canonical benchmarks and offer architecture-level alternatives to post-hoc attribution pipelines (Iwana et al., 2017, Böhle et al., 2021, Shulman, 2019, Böhle et al., 2021).