Soft-DTW: a Differentiable Loss Function for Time-Series (1703.01541v2)

Abstract: We propose in this paper a differentiable learning loss between time series, building upon the celebrated dynamic time warping (DTW) discrepancy. Unlike the Euclidean distance, DTW can compare time series of variable size and is robust to shifts or dilatations across the time dimension. To compute DTW, one typically solves a minimal-cost alignment problem between two time series using dynamic programming. Our work takes advantage of a smoothed formulation of DTW, called soft-DTW, that computes the soft-minimum of all alignment costs. We show in this paper that soft-DTW is a differentiable loss function, and that both its value and gradient can be computed with quadratic time/space complexity (DTW has quadratic time but linear space complexity). We show that this regularization is particularly well suited to average and cluster time series under the DTW geometry, a task for which our proposal significantly outperforms existing baselines. Next, we propose to tune the parameters of a machine that outputs time series by minimizing its fit with ground-truth labels in a soft-DTW sense.

• The paper introduces Soft-DTW, a smooth approximation of DTW that enables gradient-based optimization for time-series tasks.
• It details an efficient quadratic-time algorithm where the gradient is computed alongside the loss, facilitating integration into machine learning pipelines.
• Soft-DTW improves time-series averaging, clustering, and model tuning by overcoming the non-differentiable limitations of traditional DTW.

Soft-DTW: A Differentiable Loss Function for Time-Series

The paper introduces Soft-DTW, a novel differentiable loss function derived from the dynamic time warping (DTW) discrepancy, a popular method for calculating the similarity between time series. Traditional DTW is known for its ability to handle non-linear temporal deformations such as shifts and dilations, making it a robust choice for time series comparison. However, it is not differentiable, which limits its use in certain optimization and learning frameworks.

Key Contributions

1. Soft-DTW Formulation: Soft-DTW is presented as a smooth approximation of the DTW discrepancy. This smoothed version computes the soft-minimum of all alignment costs, allowing for differentiation with respect to inputs. Thus, it maintains the essential advantages of DTW while enabling the use of gradient-based optimization techniques.
2. Efficient Computation: The complexity of computing Soft-DTW and its gradient remains quadratic in both time and space, akin to the original DTW computation. The paper demonstrates that Soft-DTW’s gradient can be computed as a by-product of its value, making its integration into machine learning pipelines feasible.
3. Application to Averaging and Clustering: The capability of Soft-DTW to serve as a loss function for averaging and clustering tasks in DTW space is evidenced by experimentation. The regularization it introduces allows for overcoming the non-differentiable and unstable nature of traditional DTW in optimization pipelines.
4. Parameter Tuning in Machine Learning: By leveraging Soft-DTW, the authors propose tuning models that output time series by minimizing discrepancies with ground-truth sequences. This pivotal aspect supports the design of predictive and generative models using a differentiable approach.

Implications and Experiments

• Averaging: Soft-DTW provides an improved method for time series averaging, which outperforms competitive baselines like DBA (DTW Barycenter Averaging) by offering a smoother optimization landscape.
• Clustering: The paper demonstrates the utility of Soft-DTW in clustering frameworks, highlighting its ability to better synthesize time series centroids.
• Learning Frameworks: Soft-DTW can be effectively employed in models like neural networks to predict multistep-ahead time series outputs. This differentiability allows for more effective training compared to non-differentiable versions of DTW.

Conclusion and Future Work

The introduction of Soft-DTW signifies a step forward in time-series analysis, granting DTW-like robustness with the flexibility required by contemporary machine learning applications. Future work might explore the integration of Soft-DTW with various neural architectures, further refining its application in time series prediction and synthesis tasks. The paper's substantial improvements and insights into time-series processing pave the way for advances in how time-dependent data is modeled and utilized within AI frameworks.