DTW-Conv: Dynamic Time Warp Convolution

• DTW-Conv is a convolutional layer variant that employs dynamic time warping to dynamically align filters with input segments.
• It computes an optimal, warped dot product using dynamic programming, enhancing performance on time series with variable pacing and phase shifts.
• Empirical studies demonstrate that DTW-Conv outperforms standard Conv1D on tasks with local temporal deformations, despite moderate computational overhead.

Dynamic Time Warp Convolution (DTW-Conv) generalizes standard 1-D convolutional layers by incorporating local dynamic time warping alignment between filters and input receptive fields. By embedding a non-parametric, differentiable temporal warping directly into the convolution operation, DTW-Conv enhances robustness to local temporal deformations in sequential data—such as variable pacing, stretching, or short-range phase asynchrony—that violate the strict pointwise alignment assumed by conventional convolutional layers. The core innovation is that for each convolutional window, an optimal filter-to-input alignment is computed via dynamic programming, yielding a warped dot product that replaces the fixed linear filtering of standard Conv1D. The operator can be freely inserted into deep architectures, supports backpropagation, and requires only moderate augmentation of compute and memory, while empirical results demonstrate performance gains for time series classification tasks characterized by local deformations (Shulman, 2019, Iwana et al., 2017).

1. Mathematical Formulation and Algorithmic Workflow

Let $$x_{\vec{}} = (x_t, x_{t+1}, \ldots, x_{t+N-1}) \in \mathbb{R}^N$$ denote the input window at time $t$, and $w_{\vec{}} = (w_1, w_2, ..., w_N) \in \mathbb{R}^N$ the learnable filter. DTW-Conv replaces the standard inner product

$z = w_{\vec{}}^T x_{\vec{}}$

with a dynamically aligned, warped dot product:

Cost Matrix and Dynamic Programming

• Cost/product matrix: $C_{i,j} = w_i x_{t+j-1}$ for $i,j = 1, ..., N$.
• Cumulative score matrix: $$G(i, j) = C(i, j) + \max\{G(i-1, j-1), G(i-1, j), G(i, j-1)\}$$, with boundary conditions $G(1,1) = C(1,1)$, row and column initialization as in classical dynamic programming.
• Constraints:
  • Boundary: Path starts at $(1,1)$ and ends at $(N,N)$.
  • Monotonicity & Continuity: Only steps $t$0, $t$1, or $t$2 allowed.
  • Sakoe–Chiba warping window: $t$3 (optional, $t$410% of $t$5 is typical).

Warping Path and Alignment Matrix

• Optimal path $t$6: Retrieved by back-tracking from $t$7 to $t$8 following maximal score neighbors.
• Sparse warping matrix $t$9: Nonzero entries indicate filter-input matches along $w_{\vec{}} = (w_1, w_2, ..., w_N) \in \mathbb{R}^N$0. Normalization can be symmetric ($w_{\vec{}} = (w_1, w_2, ..., w_N) \in \mathbb{R}^N$1 per path element), “$w_{\vec{}} = (w_1, w_2, ..., w_N) \in \mathbb{R}^N$2 onto $w_{\vec{}} = (w_1, w_2, ..., w_N) \in \mathbb{R}^N$3” (row-normalized), or “$w_{\vec{}} = (w_1, w_2, ..., w_N) \in \mathbb{R}^N$4 onto $w_{\vec{}} = (w_1, w_2, ..., w_N) \in \mathbb{R}^N$5” (column-normalized), with specifics dataset-dependent.
• Final DTW-Conv activation: $w_{\vec{}} = (w_1, w_2, ..., w_N) \in \mathbb{R}^N$6; after bias and nonlinearity, this yields $w_{\vec{}} = (w_1, w_2, ..., w_N) \in \mathbb{R}^N$7.

Pseudocode Summary

$$G(i, j) = C(i, j) + \max\{G(i-1, j-1), G(i-1, j), G(i, j-1)\}$$1 Computational complexity per position is $w_{\vec{}} = (w_1, w_2, ..., w_N) \in \mathbb{R}^N$8 with band constraint.

2. Differentiability and Backpropagation

Although the warping path $w_{\vec{}} = (w_1, w_2, ..., w_N) \in \mathbb{R}^N$9 (and thus $z = w_{\vec{}}^T x_{\vec{}}$0) depends non-differentiably on $z = w_{\vec{}}^T x_{\vec{}}$1 and $z = w_{\vec{}}^T x_{\vec{}}$2, gradients propagate as in max-selector networks—the gradient flows only through the “winning” warping path chosen in the forward pass. Sub-gradients are well-defined and compatible with SGD/Adam. Explicitly,

• $z = w_{\vec{}}^T x_{\vec{}}$3,
• $z = w_{\vec{}}^T x_{\vec{}}$4.

Alternative “soft-DTW” relaxations have been proposed, but the max-based approach retains full compatibility with standard deep learning frameworks (Shulman, 2019, Iwana et al., 2017).

3. Hyperparameterization and Design Choices

• Filter length, stride, channel count: as per standard Conv1D.
• Warping window $z = w_{\vec{}}^T x_{\vec{}}$5: Narrow bands ($z = w_{\vec{}}^T x_{\vec{}}$6–$z = w_{\vec{}}^T x_{\vec{}}$7) are usually effective, with larger $z = w_{\vec{}}^T x_{\vec{}}$8 increasing both flexibility and computational cost.
• Slope constraints: Optionally limit consecutive horizontal/vertical steps to prevent degenerate alignments.
• Normalization: Choice among symmetric, $z = w_{\vec{}}^T x_{\vec{}}$9, or $C_{i,j} = w_i x_{t+j-1}$0 can impact results and is recommended as a dataset-dependent hyperparameter.
• Warping regime: Applying DTW-Conv at both train and test time yields best generalization; restricting warping to only test can harm accuracy (Shulman, 2019).

4. Practical Integration and Computational Complexity

DTW-Conv layers serve as drop-in replacements for standard Conv1D, preserving I/O shapes and compositionality with subsequent layers (pooling, BN, FC, etc.). At each forward pass, DTW alignment is recomputed for each receptive field and filter. Complexity is typically $C_{i,j} = w_i x_{t+j-1}$1 per window/position; memory overhead grows as $C_{i,j} = w_i x_{t+j-1}$2 per filter without windowing, but practical usage with Sakoe–Chiba bands or small $C_{i,j} = w_i x_{t+j-1}$3 mitigates this.

Comparison of computational scaling (per filter):

Method	Complexity per Position	Overall Runtime Impact
Conv1D	$C_{i,j} = w_i x_{t+j-1}$4	Baseline
DTW-Conv	$C_{i,j} = w_i x_{t+j-1}$5 or $C_{i,j} = w_i x_{t+j-1}$6	$C_{i,j} = w_i x_{t+j-1}$73x slower empirically (Iwana et al., 2017)

A plausible implication is that for longer filters, runtime and memory costs may pose scaling limitations without further optimization (Iwana et al., 2017).

5. Experimental Evaluations

Time-series classification: DTW-Conv was evaluated on LSST, Crop, InsectWingbeatSound, and TiSeLaC datasets (Shulman, 2019), as well as Unipen, UCI Spoken Arabic Digit, and UCI Activities of Daily Life (Iwana et al., 2017). In all reported cases, DTW-Conv matched or exceeded the test accuracy of standard Conv1D architectures, with maximal gains (up to several percent) on datasets exhibiting strong local deformations.

Ablation and hyperparameter results:

• Small $C_{i,j} = w_i x_{t+j-1}$8 ($C_{i,j} = w_i x_{t+j-1}$9–$i,j = 1, ..., N$0) suffices for most gains; larger windows generally do not improve maximum accuracy but can speed up convergence.
• Choice of normalization and warping regime impacts performance by dataset.
• Compared to LSTM baselines and other elastic-matching models (SVM+GDTW, HMM+DTW), DTW-Conv is competitive or superior on sequence classification.

Method	Unipen 1a	Arabic	ADL	InsectWingbeatSound	LSST
DTW-Conv	98.54%	96.95%	90.0%	$i,j = 1, ..., N$1 several points (LSST, Insect)	$i,j = 1, ..., N$2 several points
Conv1D	98.08%	95.50%	87.1%	baseline	baseline
LSTM	96.84%	96.09%	81.4%

Empirical runtime is typically 2–4x slower per sample than Conv1D, without hardware-specific optimization (Iwana et al., 2017).

6. Approximations and Extensions

Recent work has explored replacing classical DTW-based DP alignment with fast, fully differentiable, convolutional approximations. In "Approximating DTW with a convolutional neural network on EEG data" (Lerogeron et al., 2023), two architectures are introduced:

• DeepDTW Siamese: Learns an embedding for which Euclidean distance approximates DTW using a shared Conv1D backbone and decoder regularization.
• DeepDTW Direct: A regression architecture directly predicts the DTW value for signal pairs using a Conv1D-based encoder-MLP.

Both models scale linearly in sequence length, achieve 10–100× speedups over SoftDTW and classical DTW, and yield competitive or superior performance on EEG retrieval and classification tasks. The “Direct” method consistently outperforms FastDTW and other baselines in nearest neighbor retrieval and sleep staging tasks, with performance within 1–2pp when evaluated under cross-dataset transfer (Lerogeron et al., 2023).

Model	Complexity	End-to-end diff.	Retrieval/Classification F1	CPU Speed (vs SoftDTW)
DeepDTW Siamese	$i,j = 1, ..., N$3	Yes	Matches classical DTW for $i,j = 1, ..., N$4	$i,j = 1, ..., N$5
DeepDTW Direct	$i,j = 1, ..., N$6	Yes	Outperforms FastDTW on EEG retrieval	$i,j = 1, ..., N$7
SoftDTW	$i,j = 1, ..., N$8	Yes	Lower	1x
FastDTW	$i,j = 1, ..., N$9	No	Lower/faster	$$G(i, j) = C(i, j) + \max\{G(i-1, j-1), G(i-1, j), G(i, j-1)\}$$0 faster

A plausible implication is that DeepDTW architectures represent an efficient substitute for classical DTW-based layers in regimes where differentiability and computational speed are critical.

7. Limitations and Future Directions

Current DTW-Conv implementations are limited by quadratic per-filter computational and memory cost in the absence of windowing, making large filter lengths and 2D/3D extension nontrivial (Shulman, 2019, Iwana et al., 2017). Hardware-accelerated DP and further architectural optimizations are anticipated to close these gaps. Extending DTW alignment to spatio–temporal filters and non-Euclidean elasticity measures remains an active area for future research. Recent results also suggest that learned convolutional approximations (e.g., DeepDTW variants) offer a promising path to reduce DP overhead while retaining the recognition robustness of elastic, local temporal alignment (Lerogeron et al., 2023).