Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent (2407.21646v2)

Abstract: In this paper, we present Cross Language Agent -- Simultaneous Interpretation, CLASI, a high-quality and human-like Simultaneous Speech Translation (SiST) System. Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency. To address the challenge of translating in-domain terminologies, CLASI employs a multi-modal retrieving module to obtain relevant information to augment the translation. Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information. Experimental results show that our system outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate CLASI with a better human evaluation metric, valid information proportion (VIP), which measures the amount of information that can be successfully conveyed to the listeners. In the real-world scenarios, where the speeches are often disfluent, informal, and unclear, CLASI achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese translation directions, respectively. In contrast, state-of-the-art commercial or open-source systems only achieve 35.4% and 41.6%. On the extremely hard dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70% VIP.

Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent: An Expert Overview

The paper "Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent" by the Cross Language Agent Team at ByteDance Research introduces an innovative approach to Simultaneous Speech Translation (SiST). Leveraging the capabilities of LLMs, the paper seeks to address critical issues typically encountered in machine-assisted interpretation tasks.

Summary of Approach and Contributions

The central contribution is the Cross Language Agent - Simultaneous Interpretation (CLASI), which employs an LLM agent designed to perform high-quality, human-like translations. The architecture mimics the professional human interpreters' process by incorporating a data-driven read-write strategy and employing a Multi-Modal Retrieval Augmented Generation (MM-RAG) module to enhance translation quality through external knowledge augmentation.

Key Features of CLASI

1. Data-Driven Read-Write Policy: The read-write policy of CLASI is inspired by human interpreters, which allows the LLM to determine the appropriate timing for segmenting speech into translatable chunks. This dynamic, data-driven approach does not rely on fixed probabilities or heuristic methods, reducing latency and improving translation quality.
2. Multi-Modal Retrieval Augmented Generation (MM-RAG): CLASI integrates a multi-modal retrieval system that retrieves relevant information from an external knowledge database to augment the LLM's understanding and translation capabilities. This component is pivotal in handling domain-specific terminologies and maintaining translation accuracy.
3. Comprehensive Training Pipeline: The training process consists of three stages—pretraining on a large corpus, continual training with synthesized speech translation data, and fine-tuning on human-annotated datasets. This multi-stage approach ensures the model's robustness and alignment with human interpretation behaviors.
4. Evaluation Metrics and Human Parity: The authors introduce a novel human evaluation metric, Valid Information Proportion (VIP), which measures the proportion of information correctly conveyed in real-time translations. CLASI achieves VIP scores of 81.3% for Chinese to English and 78.0% for English to Chinese, significantly surpassing state-of-the-art commercial and open-source systems.

Experimental Results and Implications

The paper highlights substantial numerical improvements in CLASI’s performance:

• On challenging real-world datasets, CLASI attained VIP scores of 81.3% (zh-en) and 78.0% (en-zh), compared to the 35.4% and 41.6% figures achieved by leading commercial systems.
• In extreme conditions where other systems manage less than 13% VIP, CLASI still achieved 70%.

These results suggest that CLASI can handle disfluencies, informal speech, and unclear expressions effectively. The implications of this research are both practical and theoretical:

• Practically, CLASI can be deployed in various real-time translation scenarios, such as international conferences, live streaming, and multi-lingual meetings, potentially easing the reliance on human interpreters.
• Theoretically, the paper demonstrates the prowess of integrating LLMs in complex, real-time applications, opening new avenues for research in machine translation and human-computer interaction.

Latency Analysis

Despite achieving high translation quality, the latency metrics indicate that CLASI performs competitively with state-of-the-art systems. The Average Lagging, Length Adaptive Average Lagging, and First Letter Appearance Lagging metrics showcase CLASI’s acceptable delay in live scenarios, which is crucial for user experience in real-time translation tasks.

Future Directions

Future developments could focus on:

• Expanding language support: Extending CLASI to support more languages, especially low-resource ones, would broaden its applicability.
• Refining latency mechanisms: Further reducing latency without sacrificing translation quality could enhance real-time experience.
• Improved evaluation metrics: Developing more refined automatic and human evaluation metrics for long-speech and multi-modal translation tasks.

In conclusion, this research represents a significant step towards achieving human parity in simultaneous speech translation. By harnessing the power of LLMs and innovative techniques like MM-RAG and a data-driven read-write policy, CLASI sets a new benchmark in the field, addressing long-standing challenges and paving the way for enhanced machine-assisted interpretation.