Retrieval-Augmented Machine Translation with Unstructured Knowledge (2412.04342v1)

Abstract: Retrieval-augmented generation (RAG) introduces additional information to enhance LLMs. In machine translation (MT), previous work typically retrieves in-context examples from paired MT corpora, or domain-specific knowledge from knowledge graphs, to enhance models' MT ability. However, a large amount of world knowledge is organized in unstructured documents, and might not be fully paired across different languages. In this paper, we study retrieval-augmented MT using unstructured documents. Specifically, we build RAGtrans, the first benchmark to train and evaluate LLMs' retrieval-augmented MT ability. RAGtrans contains 79K MT samples collected via GPT-4o and human translators. Besides, documents from different languages are also provided to supply the knowledge to these samples. Based on RAGtrans, we further propose a multi-task training method to teach LLMs how to use information from multilingual documents during their translation. The method uses existing multilingual corpora to create auxiliary training objectives without additional labeling requirements. Extensive experiments show that the method improves LLMs by 1.58-3.09 BLEU and 1.00-2.03 COMET scores.

• The paper presents RAGtrans, a novel benchmark that integrates unstructured Wikipedia data into machine translation, achieving notable BLEU and COMET score improvements.
• It proposes a multi-task training framework with objectives such as cross-lingual relevance discrimination and self-knowledge-enhanced translation without extra labeling.
• Experimental results reveal improvements of 1.58–3.09 BLEU and 1.00–2.03 COMET points, demonstrating the robustness of utilizing noisy yet relevant document inputs.

Retrieval-Augmented Machine Translation with Unstructured Knowledge: An Analysis

The focus of this paper is the development of a retrieval-augmented machine translation (MT) system designed to efficiently leverage unstructured knowledge. While prior research efforts primarily utilized structured data sources such as paired MT corpora or knowledge graphs, this paper posits that the vast amount of world knowledge encapsulated in unstructured textual formats, such as Wikipedia pages, remains underutilized for enhancing translation tasks. To address this gap, the authors introduce RAGtrans, a pioneering benchmark aimed at both training and assessing the retrieval-augmented translation capabilities of LLMs.

Key Contributions

The initial achievement of this paper is the creation of the RAGtrans dataset, encompassing 79,000 machine translation samples along with pertinent documents sourced from Wikipedia. This dataset is distinct in its inclusion of multilingual content beyond a direct bilingual structure, providing an innovative approach to enrich translation with auxiliary knowledge in various languages. To construct this dataset, advanced techniques such as GPT-4o for English-to-Chinese translation and human translation are employed, ensuring high-quality translations in both training and evaluation phases.

In conjunction with the dataset, the authors propose a novel multi-task training framework to enhance models' translation capabilities. This method does not necessitate additional labeling and instead repurposes existing multilingual corpora to create training objectives. The training objectives are threefold: cross-lingual information completion, self-knowledge-enhanced translation, and cross-lingual relevance discrimination. Together, these objectives aim to optimize the model's capacity to synthesize and exploit relevant multilingual documents during translation tasks.

Experimental Results

The rigorous experimentation conducted with the RAGtrans dataset yields critical insights. The proposed multi-task training method demonstrates empirical advancements in translation quality, improving BLEU scores between 1.58 and 3.09 points and COMET scores by 1.00 to 2.03 points across various configurations. These numerical outcomes substantiate the efficacy of utilizing unstructured documents as supplementary translation knowledge, reinforcing the model's ability to decipher complex semantics inherent in knowledge-intensive sentences.

Additionally, the paper divulges the robustness augmentation of the proposed system, contrasting the performances with golden and noisy document inputs. The retrieval-augmented model exhibits marked improvements when supplied with contextually relevant documents, even in cases where the retrieved information is in disparate languages relative to the source and target texts.

Practical and Theoretical Implications

This research introduces a significant shift in the paradigm for retrieval-augmented MT that could potentially redefine best practices for employing LLMs in multilingual contexts. Practically speaking, the techniques outlined could facilitate more accurate and culture-sensitive machine translations, pertinent in industrial, academic, and entrepreneurial applications.

From a theoretical perspective, the paper challenges entrenched notions regarding the necessity of parallelism in knowledge sources for translation tasks. By demonstrating the efficacy of integrating unstructured, non-aligned documents, it paves the way for future inquiries into more flexible and inclusive data utilization strategies.

Future Directions

The progression of retrieval-augmented MT models is likely to benefit substantially from further exploration into the integration of disparate information sources, particularly focused on expanding beyond the linguistic boundaries considered. Moreover, future work might investigate the scalability of these methods across other language pairs and explore potential optimization in document retrieval methodologies beyond what has been examined.

In conclusion, this paper contributes a foundational dataset and methodology that advance the frontier of machine translation capabilities. Through innovative application of unstructured knowledge and comprehensive evaluation, it sets a compelling precedent for the continued intersection of sophisticated retrieval techniques and LLM development.