FTDTW: Flexible Feature Trajectory Alignment

• FTDTW is a method that independently aligns each feature dimension within multivariate sequences, accommodating asynchronous temporal variations.
• It applies standard DTW to individual 1-D trajectories and aggregates the costs normalized by the root-sum-square of alignment path lengths.
• FTDTW improves clustering and retrieval performance in speech data, achieving significant gains in F-measure and NMI over classical DTW.

Feature Trajectory Dynamic Time Warping (FTDTW) is a modification of classical Dynamic Time Warping (DTW) devised to enable individual, independent alignment of the trajectories of each feature dimension within multivariate temporal sequences. The primary motivation is the observation that in speech and similar multidimensional signals, spectral or cepstral features often undergo non-uniform temporal stretching and compression. Classical DTW applies a rigid, joint warping path to entire feature vectors, enforcing strict temporal synchrony across all dimensions. By decoupling the alignment of feature trajectories, FTDTW provides a more flexible similarity measure that accommodates asynchronous warping across dimensions, resulting in improved performance for clustering and other similarity-based tasks on speech data (Lerato et al., 2018).

1. Formal Foundations

Classical DTW

Given two temporal sequences, $X_i = \{x_{i1}, x_{i2}, ..., x_{iT_i} \}$ and $X_j = \{x_{j1}, x_{j2}, ..., x_{jT_j} \}$, each $x_{it} \in \mathbb{R}^m$ is an $m$-dimensional feature vector. Classical DTW defines a local distance $D_{ij}(p, q) = d(x_{ip}, x_{jq})$ (typically the Euclidean norm) and recursively computes the accumulated cost:

$$\gamma_{ij}(p, q) = D_{ij}(p, q) + \min\{\gamma_{ij}(p-1, q-1), \gamma_{ij}(p-1, q), \gamma_{ij}(p, q-1)\}, \ \gamma_{ij}(1,1) = D_{ij}(1,1)$$

The normalized DTW similarity is then:

$DTW(X_i, X_j) = \frac{1}{K} \gamma_{ij}(T_i, T_j),$

where $K$ is the length of the optimal alignment path.

FTDTW: Independent Trajectory Alignment

FTDTW treats each feature dimension $l = 1 \dots m$ separately as a $1$-D time series $X_i^{(l)} = \{x_{i1}^{(l)}, ..., x_{iT_i}^{(l)}\}$. Standard (non-normalized) DTW is applied independently to each trajectory pair $(X_i^{(l)}, X_j^{(l)})$, yielding:

• $C_l$: cumulative DTW cost for the $l$-th trajectory,
• $K_l$: length of the optimal alignment path for that trajectory.

The FTDTW similarity is:

$$FTDTW(X_i, X_j) = \frac{1}{\beta} \sum_{l=1}^m C_l,\quad \beta = \sqrt{\sum_{l=1}^m K_l^2}$$

This normalization by the root-sum-square of the path lengths accounts for the differing alignment degrees across feature dimensions.

2. Algorithmic Description

The FTDTW computation proceeds as follows:

1. Initialization: Set $S_{sum} \leftarrow 0$, initialize list $path\_lengths$.
2. For each feature dimension $l = 1$ to $m$:
   • Extract $1$-D trajectories $A \leftarrow (x_{i1}^{(l)}, ..., x_{iT_i}^{(l)})$, $B \leftarrow (x_{j1}^{(l)}, ..., x_{jT_j}^{(l)})$.
   • Compute unnormalized DTW: $$(C_l, K_l) \leftarrow standard\_DTW\_unnormalized(A, B)$$.
   • $S_{sum} \leftarrow S_{sum} + C_l$.
   • Append $K_l$ to $path\_lengths$.
3. Normalize: $\beta \leftarrow \sqrt{\sum_{l=1}^m K_l^2}$.
4. Output: $S = S_{sum} / \beta$.

An explicit note is that $standard\_DTW\_unnormalized$ does not perform per-path normalization.

3. Computational Complexity and Resource Trade-offs

Let $T$ denote the average sequence length, and $m$ be feature dimension.

• Classical DTW: $\mathcal{O}(T^2 m)$ time; each cost matrix cell requires an $m$-dimensional distance computation.
• FTDTW: $\mathcal{O}(m T^2)$; $m$ independent DTW runs over scalar sequences, with $O(1)$ cost per cell.

While both approaches exhibit $\mathcal{O}(T^2 m)$ overall complexity, FTDTW typically incurs a $1.5$–$2\times$ practical slowdown due to multiple $1$-D DTW computations, but this computation is amenable to parallelization across dimensions and may entail lower memory cost by reusing $2$-D DP matrices.

4. Experimental Setup

Feature Parameterization

• MFCC: 13 static coefficients (including log-energy), plus first ($\Delta$) and second ($\Delta\Delta$) derivatives; 39-dimensional total.
• PLP: Analogous configuration (available for TIMIT only).
• Frame Rate: 10 ms with 5 ms overlap.

Datasets

Dataset	Description
1	8772 TIMIT triphones (balanced: 20–25 tokens each)
2	8800 SADD Arabic digits (10 digits $\times$ 880 utterances), 10 speakers, balanced
3	123,182 unbalanced TIMIT triphones, partitioned into 10 random subsets

Clustering Protocol and Evaluation

• Clustering: Agglomerative hierarchical clustering with Ward’s linkage applied to the $N\times N$ proximity matrix of similarity scores.
• Metrics:
  • F-measure: Computed by combining precision and recall: $PR(r,v)=n_{rv}/n_r$, $RE(r,v)=n_{rv}/n_v$, $F(r,v)=2 \cdot PR \cdot RE / (PR + RE)$.
  • Normalized Mutual Information (NMI): $NMI(G,C) = 2I(G;C)/(H(G)+H(C))$.
• Statistical Significance: Paired t-tests applied to F-measures across random splits.

5. Results and Performance Analysis

FTDTW consistently outperformed classical DTW in both F-measure and NMI across balanced and unbalanced datasets.

• TIMIT Triphones (MFCC): Peak F-measure at approximately 500 clusters: DTW ≈ 0.58; FTDTW ≈ 0.68 (∼17% relative gain).
• TIMIT Triphones (PLP): Similar trends, e.g., peak F-measure DTW ≈ 0.56 vs. FTDTW ≈ 0.65.
• SADD Digits (MFCC): Clusterings with FTDTW always surpassed DTW, with peak F-measure gains of ≈10–12%.
• Unbalanced TIMIT (Dataset 3, #clusters fixed at 2394): Mean F-measure (±std): DTW 0.27 ± 0.02, FTDTW 0.32 ± 0.02; paired t-test $p < 0.0001$.

Across the full cluster range, FTDTW exceeded DTW by 0.05–0.15 absolute in F-measure. Statistically significant improvements were confirmed ($p < 0.0001$).

6. Application Considerations and Use Cases

FTDTW is especially suited for clustering or retrieval tasks in speech or other multivariate temporal signals characterized by asynchronous temporal variation in feature dimensions. Specific scenarios include:

• Applications where spectral/cepstral feature dimensions exhibit unaligned stretching/compression across utterances.
• Contexts where classical DTW yields suboptimal clustering or retrieval precision.

While FTDTW entails a modest (approximately $1.5\times$) computational overhead, it benefits from ease of parallelization by feature, and memory requirements may be reduced to storing only $2$-D DP tables. Empirically, FTDTW yields more coherent clusters and more accurate estimates of the latent number of classes, supporting its use for unsupervised sub-word unit discovery and related downstream ASR tasks.

7. Concluding Remarks

FTDTW extends classical DTW by aligning feature trajectories independently, aggregating unnormalized trajectory costs, and normalizing by the root-sum-square of alignment path lengths. By eliminating the enforced temporal synchrony across vector dimensions, FTDTW models the asynchronous dynamics typical of speech features more faithfully. Empirical results on the TIMIT triphone and SADD Arabic digit datasets demonstrate statistically significant improvements in clustering quality, as measured by F-measure and NMI, with only moderate computational overhead (Lerato et al., 2018).