BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models (2306.10968v2)

Abstract: LLMs have demonstrated remarkable prowess in language understanding and generation. Advancing from foundation LLMs to instructionfollowing LLMs, instruction tuning plays a vital role in aligning LLMs to human preferences. However, the existing LLMs are usually focused on English, leading to inferior performance in non-English languages. In order to improve the performance for non-English languages, it is necessary to collect language-specific training data for foundation LLMs and construct language-specific instructions for instruction tuning, both of which are heavy loads. To minimize human workload, we propose to transfer the capabilities of language generation and instruction following from English to other languages through an interactive translation task. We have developed BayLing, an instruction-following LLM by utilizing LLaMA as the foundation LLM and automatically constructing interactive translation instructions for instructing tuning. Extensive assessments demonstrate that BayLing achieves comparable performance to GPT-3.5-turbo, despite utilizing a considerably smaller parameter size of only 13 billion. Experimental results on translation tasks show that BayLing achieves 95% of single-turn translation capability compared to GPT-4 with automatic evaluation and 96% of interactive translation capability compared to GPT-3.5-turbo with human evaluation. To estimate the performance on general tasks, we created a multi-turn instruction test set called BayLing-80. The experimental results on BayLing-80 indicate that BayLing achieves 89% of performance compared to GPT-3.5-turbo. BayLing also demonstrates outstanding performance on knowledge assessment of Chinese GaoKao and English SAT, second only to GPT-3.5-turbo among a multitude of instruction-following LLMs. Demo, homepage, code and models of BayLing are available.

Analysis of BayLing: Enhancing Cross-lingual Capabilities in LLMs Through Interactive Translation

The paper presents BayLing, a sophisticated LLM developed to improve cross-lingual capabilities and instruction-following proficiency in non-English languages. Building on foundational LLMs like LLaMA, BayLing leverages interactive translation tasks to facilitate cross-lingual alignment, mitigate language-specific training burdens, and enhance performance in multilingual contexts. This paper explores various aspects of the BayLing model, articulating both its implementation and empirical evaluations across diverse language tasks.

BayLing is crafted using LLaMA as the foundation model. The prominent innovation in BayLing lies in its method of instruction tuning through interactive translation tasks. This approach eliminates the necessity of gathering extensive language-specific data, transferring English language capabilities to non-English contexts effectively. The paper emphasizes engaging a high-level interaction process between users and the model to refine language generation capabilities seamlessly.

Key Components and Methodology

1. Foundation Model Selection: BayLing is structured upon LLaMA, an established LLM known for its robust English understanding capabilities. By building on this strong foundation, BayLing focuses on cross-lingual proficiency while maintaining a manageable model size.
2. Interactive Translation: The interactive translation mechanism serves dual purposes: it aligns multiple languages with English and reinforces the model's ability to interpret and act on human instructions. This mechanistic tuning bypasses the heavy demand for non-English datasets, leveraging existing English-centric model training to other languages via cross-lingual tasks.
3. Instruction Tuning: The model's excellence in instruction tuning and multi-turn interaction standardizes it for broader NLP tasks. By incorporating interactive translation instructions, BayLing hones its contextual comprehension and instruction-following capabilities within multi-turn dialogue frameworks.

Evaluation and Results

Extensive evaluations reveal BayLing’s proficiency:

• Translation Tasks: BayLing achieves notable performance benchmarks, attaining 95% and 96% of translation capabilities compared to state-of-the-art models like GPT-3.5-turbo across Chinese-English and German-English benchmarks.
• General Tasks: Evaluations on the BayLing-80 test set demonstrate BayLing achieving 89% of the performance of GPT-3.5-turbo, showcasing strengths in generic and knowledge tasks.
• Standardized Tests: Remarkably, BayLing scores competitively on Chinese GaoKao and English SAT tests, emphasizing its effective knowledge transfer from English-centric corpora to other languages.

Key Outcomes and Implications

• Cross-lingual Transfer Without Pre-training: BayLing's use of interactive tasks effectively transfers language generation and instruction compliance between languages, sidestepping the traditional requirement for large-scale non-English language pre-training.
• Integration of Task Capabilities: Through interactive translation, multi-capability enhancement coalesces in BayLing, offering a streamlined methodology to simultaneously elevate language alignment and human instruction adherence.
• Benchmark Setting: BayLing posits itself as a measurable, openly available benchmark in multilingual translation, encouraging onward advancements and model comparisons in translation tasks.

Future Prospective and Considerations

BayLing provides a compelling blueprint for future cross-lingual innovations in LLM research. Its methodology encourages leveraging foundational models and task-specific tuning to expand LLM competencies efficiently. However, it also highlights several areas for future exploration, including enhancing capabilities in math, coding, and reasoning tasks where the performance still lags behind leading LLM models such as GPT-3.5-turbo.

In essence, BayLing exemplifies a balanced and insightful approach to augmenting non-English language capabilities in LLMs. Its elegance lies in streamlining resource input while maximizing linguistic output, paving the way for extensive applications and fostering cross-lingual understanding through intelligent interaction and alignment tactics.