Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation (2104.08771v2)

Abstract: We study the power of cross-attention in the Transformer architecture within the context of transfer learning for machine translation, and extend the findings of studies into cross-attention when training from scratch. We conduct a series of experiments through fine-tuning a translation model on data where either the source or target language has changed. These experiments reveal that fine-tuning only the cross-attention parameters is nearly as effective as fine-tuning all parameters (i.e., the entire translation model). We provide insights into why this is the case and observe that limiting fine-tuning in this manner yields cross-lingually aligned embeddings. The implications of this finding for researchers and practitioners include a mitigation of catastrophic forgetting, the potential for zero-shot translation, and the ability to extend machine translation models to several new language pairs with reduced parameter storage overhead.

• The paper shows that updating only cross-attention parameters yields nearly identical BLEU scores to full fine-tuning across multiple language pairs.
• It reveals that pretrained cross-attention values are crucial, as initializing from scratch causes significant performance drops.
• The approach reduces fine-tuning parameters from 75% to 17%, offering a storage-efficient method for scalable machine translation.

Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation

In the field of NLP, the Transformer architecture offers a highly scalable and efficient model for sequential data tasks. Given its proficiency, particularly in natural language tasks, this paper explores a crucial aspect of the Transformer framework—cross-attention—and evaluates its power within the context of transfer learning for machine translation (MT).

Key Insights and Methodologies

The researchers conducted a series of experiments to ascertain whether fine-tuning only the cross-attention parameters of a pretrained Transformer model could match the performance of fine-tuning all model parameters. These experiments were extensive, covering various language pairs and employing fine-tuning strategies that selectively updated specific parts of the model.

The primary fine-tuning strategies investigated were:

1. Training from scratch: No parameters were reused from the pretrained model.
2. Regular fine-tuning: All parameters except for embeddings of the shared language.
3. Cross-attention fine-tuning: Only the cross-attention layers and embeddings were updated.
4. Embedding-only fine-tuning: Only the new embeddings were updated.

Empirical Results and Analysis

The results revealed several critical findings. Most notably, fine-tuning solely the cross-attention layers, along with new embeddings, resulted in MT performance that was competitive with methods involving full model fine-tuning. Specifically, the BLEU scores for cross-attention fine-tuning were consistently close to those obtained via complete fine-tuning across different language pair transfers. For example, when transferring from Fr–En to Ro–En, cross-attention fine-tuning achieved a BLEU score of 30.9, just 0.1 points below the score obtained from full model fine-tuning.

Another crucial result was that starting from randomly initialized cross-attention parameters resulted in a marked performance drop. This underscores the importance of pretrained cross-attention values in effective adaptation to new language pairs.

From a practical standpoint, fine-tuning cross-attention layers markedly reduces the number of parameters that need to be updated and stored. On average, this method required 17% of the parameters as opposed to 75% in full model fine-tuning, significantly reducing the storage overhead.

Implications and Future Directions

The implications of these findings are manifold. Firstly, this approach mitigates catastrophic forgetting, enabling the model to retain previously acquired knowledge while adapting to new tasks. Moreover, the cross-linguistic alignment of embeddings discovered under this fine-tuning strategy can be leveraged for zero-shot translation. The empirical evidence showed that these zero-shot models, achieved without direct parallel training data, yielded respectable BLEU scores, suggesting practical applications in resource-constrained settings.

Additionally, as verified by various experiments using models like mBART, the nature of the pretrained model impacts the effectiveness of cross-attention parameter re-use. mBART, pretrained with a denoising objective rather than translation, did not fare as well in fine-tuning scenarios compared to an MT-pretrained parent model.

Conclusion

The research advances our understanding of the Transformer architecture's cross-attention mechanism within MT. Fine-tuning cross-attention layers offers a promising avenue for efficient and effective model adaptation, reducing computational overhead while maintaining high-quality translation performance. Future work could explore further refinements in embedding strategies, the impact of alternative pretraining objectives, and the extension of these findings to other sequential tasks in NLP and beyond. Such insights will undoubtedly contribute to the ongoing evolution of scalable and adaptable AI systems.