Monotonic Infinite Lookback Attention for Simultaneous Machine Translation (1906.05218v1)

Abstract: Simultaneous machine translation begins to translate each source sentence before the source speaker is finished speaking, with applications to live and streaming scenarios. Simultaneous systems must carefully schedule their reading of the source sentence to balance quality against latency. We present the first simultaneous translation system to learn an adaptive schedule jointly with a neural machine translation (NMT) model that attends over all source tokens read thus far. We do so by introducing Monotonic Infinite Lookback (MILk) attention, which maintains both a hard, monotonic attention head to schedule the reading of the source sentence, and a soft attention head that extends from the monotonic head back to the beginning of the source. We show that MILk's adaptive schedule allows it to arrive at latency-quality trade-offs that are favorable to those of a recently proposed wait-k strategy for many latency values.

• The paper presents MILk attention, a dual-head mechanism that adaptively schedules source token reading to optimize latency-quality trade-offs.
• It extends the Average Lagging metric into a differentiable training objective, enabling precise control over translation latency.
• Empirical evaluations on WMT datasets show MILk attention outperforms fixed wait-k policies by reducing lag and enhancing BLEU scores.

Overview of Monotonic Infinite Lookback Attention for Simultaneous Machine Translation

The paper presents a novel framework in the domain of simultaneous machine translation (MT), specifically focusing on real-time translation scenarios where predictive translation is required before a sentence is fully articulated by the speaker. The proposed system, termed as Monotonic Infinite Lookback (MILk) attention, introduces an innovative mechanism that bridges the dichotomy between latency and translation quality.

Key Contributions

The authors achieve a significant milestone by integrating an adaptive translation schedule with a neural machine translation (NMT) model through MILk attention. This mechanism incorporates two attention heads: a hard, monotonic attention head which determines the reading schedule of the source text and a secondary soft attention head which spans from this head to the start of the source sentence. The dual nature of MILk allows the system to maintain low latency while improving translation output quality, especially in comparison to existing strategies like wait-$k$.

The paper puts forward three main contributions:

1. MILk Attention Mechanism: The introduction of MILk establishes an NMT system that learns an adaptive schedule and is responsive to source tokens available up to a given point. This contributes to achieving efficient latency-quality trade-offs.
2. Latency Metric Improvement: The extension of the Average Lagging (AL) latency metric into a differentiable form allows it to operate as a training objective, enabling better optimization during model learning.
3. Empirical Advantage: Extensive experimental evaluations demonstrate the superiority of the MILk approach over predefined wait-$k$ policies across multiple latency conditions.

Methodological Insights

The integration of MILk attention into an NMT architecture involves a nuanced modification. It revolves around enhancing the model's ability to perform streaming attention by leveraging monotonic attention paradigms suitable for live translation scenarios. The mechanism involves decisions to advance through source tokens, making it adaptable based on real-time streaming data. Furthermore, the latency-augmented training scheme ensures that models trained with MILk attention balance latency effectively without compromising translation fidelity.

Experimentation and Results

The paper outlines rigorous experimental protocols using datasets from WMT14 English-to-French and WMT15 German-to-English tasks, revealing measurable improvements in BLEU scores. The proposed MILk attention not only maintained quality comparable to conventional full-sequence translation models but excelled in reducing latency to more favorable trade-offs than the traditional approaches operating under fixed schedules. Such advantages are pronounced in lag conditions ranging between 4 to 14 tokens, offering a flexible yet powerful translation option that adapts to varied sentence structures and lengths.

Implications and Future Directions

The development of MILk attention signifies a pivotal step towards more adaptable simultaneous MT systems, particularly beneficial for stream-based applications like live speech translation. By unifying attention mechanisms that cater to both immediate past and streaming future tokens, MILk presents a robust framework adaptable to real-time constraints.

Future work could explore further optimization in variances of adaptive scheduling and potentially extend these findings to other modalities, such as multimodal translations involving both visual and textual data streams. Additionally, given the current landscape where neural architectures are increasingly sophisticated, MILk's principles might inspire broader applications across sequential predictive modeling domains.

Overall, the paper highlights significant advancements in simultaneous NMT, paving the way for further exploration in latency-efficient, quality-preserving translation technologies.