The Conformer Encoder May Reverse the Time Dimension (2410.00680v2)

Abstract: We sometimes observe monotonically decreasing cross-attention weights in our Conformer-based global attention-based encoder-decoder (AED) models, Further investigation shows that the Conformer encoder reverses the sequence in the time dimension. We analyze the initial behavior of the decoder cross-attention mechanism and find that it encourages the Conformer encoder self-attention to build a connection between the initial frames and all other informative frames. Furthermore, we show that, at some point in training, the self-attention module of the Conformer starts dominating the output over the preceding feed-forward module, which then only allows the reversed information to pass through. We propose methods and ideas of how this flipping can be avoided and investigate a novel method to obtain label-frame-position alignments by using the gradients of the label log probabilities w.r.t. the encoder input frames.

• The paper demonstrates that the Conformer encoder reverses the time dimension due to self-attention dynamics, particularly within deeper layers.
• It proposes mitigation strategies such as CTC loss, disabling early self-attention, and centering frame focus to preserve correct sequence order.
• The study introduces gradient-based alignment methods that outperform traditional CTC alignments in reducing time-stamp errors.

Analysis and Insights on Conformer Encoder Time Reversal and Sequence Alignment from Gradients

In the examination of the peculiar tendency of the Conformer encoder within attention-based encoder-decoder (AED) models to reverse the time dimension of input sequences, Schmitt et al. present an intricate paper of the underlying causes, implications, and potential mitigations of this phenomenon. The paper provides a detailed dissection of the encoder's behavior, demonstrating the internal mechanics that facilitate this anomaly and introducing methodologies for both preventing the issue and harnessing gradient-based approaches for sequence alignment.

Observations and Initial Behavior

The authors observe the occurrence of monotonically decreasing cross-attention weights within their Conformer-based AED models, noting that the encoder appears to reverse the input sequence's time dimension. Upon probing, it is discovered that during initial stages of training, the decoder's cross-attention mechanism emphasizes the first few frames, not due to their direct predictive value, but likely because of their distinctiveness resulting from sequence boundary padding and initial silence.

Mechanism of Time Reversal

The paper clearly identifies that the time reversal occurs within the self-attention layers. This flipping is particularly seen in the 10th Conformer block, where the self-attention's dominant activations overshadow the residual connections. The use of a final layer normalization further eliminates the influence of original frame-wise information, thereby only retaining the reversed sequence order. The encoder’s behavior is reinforced over subsequent layers and epochs as gradients show a stronger focus on leading frames early in training.

Factors Contributing to the Phenomenon

Several stages outline the progression toward time reversal:

1. Initial cross-attention is biased towards the first few frames.
2. The decoder functions independently, initially acting similarly to a LLM, leaning on the global information from initial frames.
3. As training progresses, the model shifts attention to non-initial frames while still maintaining a global perspective.
4. The model transitions to fully reversed frame attention due to easier sequential association driven by positional information.

Differences in vocabulary size and sequence length, combined with stochastic training dynamics, sporadically result in varied flipping and shuffling behaviors, adding complexity to the issue.

Mitigating Time Reversal

To counteract the time reversal, the authors propose several strategies:

• CTC Auxiliary Loss: Introducing a Connectionist Temporal Classification (CTC) auxiliary loss imposes a constraint of monotonic alignment, which inherently prevents flipping. This approach emerges as effective, consistently eliminating the issue across multiple trials.
• Disabling Initial Self-Attention: Fixing self-attention weights to the identity matrix during initial training epochs avoids early frame fixation, thus preventing subsequent flipping.
• Hard Attention to Center Frames: Enforcing attention on central frames during early training epochs circumvents the initial model bias toward leading frames, thereby averting the flipping development.

Advancing Sequence Alignment Using Gradients

In an innovative extension, the authors explore using the gradients of label log probabilities with respect to encoder input frames for obtaining label-frame-position alignments. This method leverages gradient-based attribution to establish alignment paths, proving to be robust even when the encoder flips sequences. Comparisons reveal that this gradient-based alignment method outperforms CTC alignments in terms of time-stamp-error (TSE), particularly in models utilizing byte pair encoding (BPE) for output labelling.

Future Implications

The insights from this paper hold significant implications for both theoretical understanding and practical applications in automatic speech recognition (ASR). The discovered encoder behavior underscores the necessity to reassess self-attention mechanisms' interplay with residual connections, offering guidance on neural network architecture adjustments. The introduction of gradient-based alignment offers a new dimension for obtaining precise label-frame associations, benefiting tasks requiring high accuracy in temporal labeling.

In summary, this work illuminates an unprecedented aspect of Conformer encoder dynamics while furnishing actionable strategies to mitigate sequence reversal. Simultaneously, it opens avenues for utilizing gradient information to enhance sequence alignment, marking an important contribution to ASR model efficiencies and interpretability.