Drop-DTW: Robust Sequence Alignment

• Drop-DTW is a sequence alignment algorithm that selectively matches shared inlier content while dropping unmatched outliers using a fixed penalty.
• It formulates the matching process as a dynamic programming problem with soft minimum operations, ensuring efficiency and differentiability for gradient-based optimization.
• Empirical evaluations demonstrate state-of-the-art performance in areas like video localization, unsupervised representation learning, and multimodal retrieval under noisy conditions.

Drop-DTW is an algorithm for sequence-to-sequence alignment that extends Dynamic Time Warping (DTW) to robustly handle signals containing outliers. Unlike standard DTW, which assumes all elements in two temporal sequences correspond to each other (with possible shifts and temporal distortions), Drop-DTW aligns only the shared "inlier" content and automatically drops unmatched (outlier) elements at a fixed penalty. The method is formalized as a dynamic program, is efficient and easily differentiable, and has demonstrated state-of-the-art performance as a sequence similarity measure and a training loss across various vision and multimodal retrieval tasks (Dvornik et al., 2021).

1. Problem Setting and Motivation

Let $X = \{x_1, ..., x_N\}$ and $Y = \{y_1, ..., y_K\}$ be two sequences, such as feature vectors extracted from video or audio, with lengths $N$ and $K$ respectively. In many applications, these sequences are contaminated by outliers—elements not corresponding to any inlier in the other sequence. The principal goal is to:

• Discover a one-to-many monotonic matching between the inlier portions of $X$ and $Y$, as in standard DTW (which is robust to temporal shifts/dilations but not to arbitrary outliers).
• Allow for any subset of elements in $X \cup Y$ to be "dropped" (removed from matching) subject to a fixed penalty, thereby aligning only the shared signal and excluding outliers from the computation.

This setting generalizes classical DTW by relaxing the requirement that every element be aligned, improving robustness in noisy retrieval, localization, and representation learning scenarios.

2. Mathematical Objective

Drop-DTW defines a binary correspondence matrix $M \in \{0,1\}^{K \times N}$ where $M_{i,j} = 1$ iff $y_i$ is matched to $Y = \{y_1, ..., y_K\}$0. A penalty is assigned for each unmatched element (i.e., zero row or column in $Y = \{y_1, ..., y_K\}$1). Using $Y = \{y_1, ..., y_K\}$2 as the pairwise cost (e.g., $Y = \{y_1, ..., y_K\}$3) and $Y = \{y_1, ..., y_K\}$4, $Y = \{y_1, ..., y_K\}$5 as drop penalties for $Y = \{y_1, ..., y_K\}$6 and $Y = \{y_1, ..., y_K\}$7: $Y = \{y_1, ..., y_K\}$8 where $Y = \{y_1, ..., y_K\}$9 is the set of monotonic (no backward matches) but not fully dense matrices.

In shorthand, if $N$0 and $N$1 indicate which rows/columns are entirely zero,

$N$2

This formulation allows alignment with selective skipping of outliers, controlled by tunable drop penalties.

3. Dynamic Programming Recurrence

The Drop-DTW recursion constructs a cost table $N$3 for optimal alignment of prefixes $N$4 and $N$5.

Drop-X variant (dropping only in $N$6; $N$7):

Auxiliary tables:

• $N$8: cost if $N$9 matched to $K$0
• $K$1: cost if $K$2 is dropped

Recurrence:

1. $K$3
2. $K$4
3. $K$5

With $K$6, $K$7, $K$8, $K$9.

Full (bi-directional drop) version:

Four tables track match/drop for each boundary:

1. $X$0 (match), $X$1 (drop-$X$2), $X$3 (drop-$X$4), $X$5 (drop both).
2. Recurrences (see Drop-DTW summary above for full equations).

Standard back-tracing yields optimal match/drop decisions $X$6. This dynamic program accommodates a strictly monotonic but partially matched correspondence.

4. Differentiable ("Soft") Drop-DTW

All "min" operators are replaced by a soft-min: $X$7 As $X$8, soft-min converges to hard min. This yields a fully differentiable dynamic program suitable for gradient-based optimization. Gradients propagate through the Drop-DTW recurrence via the chain rule, as in Soft-DTW and differentiable DP in modern attention models. The entire computation remains $X$9 in time and space.

5. Pseudocode and Implementation

For the drop-$Y$0 variant, the core procedure is:

$M \in \{0,1\}^{K \times N}$1

Pairwise cost $Y$1 can be symmetric ($Y$2) or asymmetric (negative log-softmax). Drop penalty $Y$3 may be a fixed percentile of $Y$4 (tunable $Y$5) or derived via a small neural network predicting per-element costs. The soft-min temperature $Y$6 trades bias versus stability ($Y$7–$Y$8 practical). For very long sequences, windowing/pruning (e.g., Sakoe–Chiba band) can reduce computation and memory to $Y$9.

6. Computational Complexity

Drop-DTW requires $X \cup Y$0 time and space for full alignment and back-trace, equivalent to standard DTW. If only the final alignment cost is needed, memory can be reduced to $X \cup Y$1. Application of windowing or pruning further optimizes runtime for very long sequences while maintaining alignment fidelity.

Step	Time Complexity	Space Complexity
Full DP fill	$X \cup Y$2	$X \cup Y$3
Cost-only computation	$X \cup Y$4	$X \cup Y$5
Windowed DP	$X \cup Y$6	$X \cup Y$7

7. Experimental Applications and Empirical Findings

Drop-DTW has been validated on multiple noisy alignment and retrieval tasks:

• Synthetic trajectory retrieval (TMNIST): Short sub-trajectories with interspersed Gaussian-blurred frames ("noise"). Drop-DTW Recall@1 greatly exceeds DTW (up to $X \cup Y$8 improvement under heavy noise), and achieves 97.1% subsequence localization accuracy with IoU = 91.2%.
• Instructional video localization (CrossTask, COIN, YouCook2): In weakly supervised settings (ordered step descriptions, no frame labels), Drop-DTW (percentile drop cost) outperforms Smooth-DTW, D$X \cup Y$9TW, and OTAM by 3–5 points in IoU. With learned drop costs plus clustering regularizer: CrossTask IoU = 36.9% (vs 30.5% for Smooth-DTW); COIN IoU = 29.5%; YouCook2 IoU = 49.4%.
• Unsupervised representation learning (PennAction): Under heavy outlier contamination (up to 50% distractor frames), models trained with Drop-DTW loss yield high Kendall’s $M \in \{0,1\}^{K \times N}$0 alignment accuracy, while Smooth-DTW degrades rapidly.
• Unsupervised cross-modal audio-visual localization (AVE): Given a 1 s query in one modality, Drop-DTW used as matching cost in a contrastive triplet loss achieves A2V localization of 41.1% (vs Smooth-DTW 39.8% and OTAM 37.5%) and V2A localization of 35.8% (exceeding even a supervised baseline on V2A).

In all applications, Drop-DTW's ability to exclude arbitrary outliers establishes a more robust similarity measure than DTW or previous differentiable DTW methods, supporting weakly and unsupervised sequence localization and retrieval tasks in vision and multimodal domains (Dvornik et al., 2021).