Canonical Time Warping (CTW)

• Canonical Time Warping (CTW) is a temporal alignment method that jointly adapts feature representations and aligns time series for robust synchronization.
• CTW integrates Dynamic Time Warping and Canonical Correlation Analysis in an alternating optimization framework, enhancing alignment accuracy across varying domains.
• CTW has been applied in tasks like singing voice correction, though challenges remain in managing high-dimensional, nonlinear, or sparse datasets.

Canonical Time Warping (CTW) is a temporal alignment technique designed to synchronize two multivariate time series by jointly adapting feature representations and aligning temporal structure. CTW integrates Dynamic Time Warping (DTW) and Canonical Correlation Analysis (CCA) into a unified optimization framework, allowing robust alignment across domains with nonlinear time and feature-space distortions. This method is particularly effective when the aligned sequences exhibit differences in pitch, timbre, or other expressive attributes, as demonstrated in expressive singing voice correction, and it serves as the foundation for subsequent developments in deep nonlinear temporal alignment methods (Luo et al., 2017, Steinberg et al., 2024).

1. Mathematical Formulation

Let $X \in \mathbb{R}^{d \times n_x}$ and $Y \in \mathbb{R}^{d \times n_y}$ denote the $d$-dimensional feature sequences of source and target, respectively. The temporal alignment is captured by binary warping-path matrices $W_x \in \{0,1\}^{m \times n_x}$ and $W_y \in \{0,1\}^{m \times n_y}$, where $m$ is the length of the alignment path. Linear projections $V_x, V_y \in \mathbb{R}^{d \times b}$ map the data into a shared $b$-dimensional subspace.

The CTW objective is: $$\min_{V_x,V_y,\,W_x,W_y}\; \left\| V_x^T X W_x^T - V_y^T Y W_y^T \right\|_F^2$$ subject to: $$V_x^T X D_x X^T V_x = I_b,\quad V_y^T Y D_y Y^T V_y = I_b$$ where $Y \in \mathbb{R}^{d \times n_y}$0, $Y \in \mathbb{R}^{d \times n_y}$1 are diagonal weighting matrices. The monotonicity and contiguity constraints are imposed on $Y \in \mathbb{R}^{d \times n_y}$2. The solution seeks projections maximizing cross-correlation within a warped common subspace while enforcing temporal alignment, combining the CCA criterion (maximizing correlation under covariance normalization) and the DTW path-finding criterion (Luo et al., 2017, Steinberg et al., 2024).

2. Optimization Algorithm

The CTW cost function is non-convex in the joint space of projections and warping paths. Optimization proceeds via block coordinate descent:

• Projection Step: With fixed warping paths $Y \in \mathbb{R}^{d \times n_y}$3, update projections $Y \in \mathbb{R}^{d \times n_y}$4 by solving the paired generalized eigenvalue problem:

$Y \in \mathbb{R}^{d \times n_y}$5

• Alignment Step: With projections $Y \in \mathbb{R}^{d \times n_y}$6 fixed, update $Y \in \mathbb{R}^{d \times n_y}$7 by computing ordinary DTW on the projected sequences, minimizing:

$Y \in \mathbb{R}^{d \times n_y}$8

Alternation continues until convergence. Initialization can use uniform length scaling (CTW-uniform) or DTW alignments (CTW-dtw). The latter is advantageous when the sequences exhibit nonlinear temporal discrepancies (Luo et al., 2017).

3. Comparison to Related Methods

A direct comparison reveals CTW’s distinguishing properties:

Method	Alignment Space	Feature Adaptation	Robustness to Nonlinear Distortion
DTW	Input feature space	None	Low
CTW	Learned latent space	Linear (CCA)	Moderate-High
DCTW/CDCTW	Deep/Nonlinear space	Nonlinear + Gated	High

CTW can accommodate global and piecewise time-stretching as well as moderate pitch-shifting, maintaining higher alignment accuracy than DTW in the presence of feature-space distortion. Unlike DTW’s reliance on direct feature distances, CTW first projects into a maximally correlated subspace, reducing sensitivity to mismatches in features such as pitch or timbre. Each alternating step in the optimization guarantees non-increasing loss, leading to reliable convergence to a satisfactory local minimum within a few iterations (Luo et al., 2017).

In high-dimensional, sparse, or highly nonlinear regimes, CTW can be generalized to Conditional Deep Canonical Time Warping (CDCTW), which replaces linear CCA projections with neural nonlinear mappings and introduces dynamic, context-dependent feature selection via stochastic gating, further enhancing robustness and alignment accuracy (Steinberg et al., 2024).

4. Practical Applications

CTW was initially evaluated in singing voice correction—specifically, adapting an amateur's pitch contour to that of a professional performance while preserving timbral characteristics:

1. Feature extraction: Using the WORLD vocoder, F0 (fundamental frequency), 24-dimensional mel-cepstral coefficients (MCEP), and aperiodicity (AP) features are extracted from both source and target vocals.
2. Alignment: CTW aligns MCEP sequences to obtain an optimal warping path $Y \in \mathbb{R}^{d \times n_y}$9.
3. Pitch transfer: The warping path maps the professional’s F0 contour onto the amateur’s time axis, producing a “corrected” F0 sequence.
4. Synthesis: The corrected F0, original spectral envelope, and AP are synthesized to produce the pitch-corrected output.

Parameter choices such as a 1024-point FFT, 5 ms frame hop, and subspace dimension $d$0 were adopted in empirical studies. Objective evaluation demonstrated CTW's robustness to pitch-shifting and time-stretching, while subjective listening tests indicated improved pitch accuracy and fluency relative to DTW and commercial autotuning software (Luo et al., 2017).

5. Limitations and Ongoing Developments

CTW's reliance on linear CCA and its alternating optimization strategy renders it suboptimal for highly nonlinear and high-dimensional sparse data. Extensions such as Conditional Deep Canonical Time Warping (CDCTW) address these limitations by employing deep nonlinear embedding networks and introducing context-dependent stochastic gates for dynamic feature selection.

Key properties of CDCTW include:

• Nonlinear latent embeddings via neural networks.
• Context-dependent feature selection using learned stochastic gates.
• Empirical superiority over CTW and related methods in alignment accuracy on challenging datasets such as Moving MNIST, TCD-TIMIT audio-visual speech, and MMI facial expressions, particularly in high-noise or sparse settings.

Current limitations involve non-differentiability through DTW alignments (future work includes soft relaxation for end-to-end training), variance from gating-based Monte Carlo estimation, and the need for domain-adapted contextual gating features. Extending CTW/CDCTW to multi-view alignment or complex manifold-aware temporal constraints is an open research avenue (Steinberg et al., 2024).

6. Significance and Outlook

Canonical Time Warping serves as a rigorous methodological bridge between feature-space adaptation and temporal alignment, enabling robust synchronization under significant domain shifts. Its impact is evident in applications that require cross-domain or cross-modal sequence alignment, particularly where expressive distortions are present (e.g., singing voice correction, audio-visual event alignment). Its extension to deep, context-sensitive models marks an important direction for scalable, robust temporal alignment in increasingly complex and high-dimensional settings (Luo et al., 2017, Steinberg et al., 2024).