On the Weaknesses of Reinforcement Learning for Neural Machine Translation (1907.01752v4)

Abstract: Reinforcement learning (RL) is frequently used to increase performance in text generation tasks, including machine translation (MT), notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN). However, little is known about what and how these methods learn in the context of MT. We prove that one of the most common RL methods for MT does not optimize the expected reward, as well as show that other methods take an infeasibly long time to converge. In fact, our results suggest that RL practices in MT are likely to improve performance only where the pre-trained parameters are already close to yielding the correct translation. Our findings further suggest that observed gains may be due to effects unrelated to the training signal, but rather from changes in the shape of the distribution curve.

• The paper offers a theoretical critique of Minimum Risk Training, revealing its failure to optimize expected risk in neural machine translation.
• It demonstrates that observed performance improvements arise from increased token peakiness rather than genuine enhancements in learning.
• The study underscores that effective convergence is contingent on near-optimal pre-training, urging the development of more robust reinforcement learning methods.

Analyzing the Efficacy of Reinforcement Learning Practices in Neural Machine Translation

This paper presents a critical examination of the utilization of reinforcement learning (RL) in neural machine translation (NMT), particularly focusing on commonly employed techniques like Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN). While RL has been increasingly integrated into the text generation tasks with the promise of optimizing challenging non-differentiable functions and addressing notable biases such as "exposure bias," its effectiveness and dynamics in the specific context of NMT have not been thoroughly understood or evidenced.

Key Findings

The authors delve into the theoretical underpinnings of RL methods for NMT, revealing inherent weaknesses in optimization approaches. They argue that common RL practices fail to minimize expected risk and that convergence is achieved only under optimal conditions, where pre-trained parameters are near accuracy.

1. Theoretical Analysis of MRT: One of the substantial contributions of this work is a theoretically grounded critique of MRT methods applied in NMT. They demonstrate that these methods are not adequately founded in optimizing expected reward, often failing to reach risk minima. In essence, MRT does not truly approximate the expected reward function $R(\theta)$, hence, may not lead to optimal parameter updates.
2. Empirical Evidence of Performance Gains: Through simulations and controlled experiments, it was revealed that performance gains may not stem from enhancing token probabilities where they are rewarded. Instead, perceived improvements could be derived from a "peakiness effect"—increasing the probability mass of the most probable tokens, which raises questions about genuine advancements attributed to RL.
3. Convergence Rate Concerns: The convergence rate of the enhancement of target tokens to mode is concerningly slow, achievable under stringent conditions with pre-trained models. When the most probable target tokens rank beyond second or third, even significant amounts of training data and steps fail to yield notable configurations.

Implications and Future Prospects

This critical evaluation urges the research community to reassess RL practices in NMT, both theoretically and pragmatically. The implications are manifold:

• Accuracy of RL Optimizations: The paper challenges the community to develop more robust RL methods capable of effectively tackling non-ideal conditions in NMT. Current practices show limitations in exploration and risk minimization, hinting at the need for adopting off-policy learning and enhanced exploration techniques to facilitate better sampling from diverse and rewarding states.
• Policy Adjustments: Alterations in RL policy sampling could substantially aid convergence, smoothing the inherent peakiness, and thus, fostering exploration over exploitation bias observed presently.
• Expansion of RL Theory: The research highlights the potential for more extensive, foundational studies to inform RL adaptation in high-dimensional, discrete spaces of NMT, which pose unique challenges compared to traditional RL domains. Implementing sophisticated exploratory strategies and improving sampling methodologies could offer substantial advances.

Conclusion

The paper makes an incisive contribution to understanding the limitations and effectiveness of reinforcement learning in neural machine translation. By laying bare the inadequacies of existing RL practices, the researchers carve out a path for evolving RL to address specific NMT challenges through theoretically sound and empirically validated approaches. This could fundamentally alter how RL techniques are leveraged to improve performance in text generation tasks beyond just machine translation.