Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better (2212.08597v2)

Abstract: While the problem of hallucinations in neural machine translation has long been recognized, so far the progress on its alleviation is very little. Indeed, recently it turned out that without artificially encouraging models to hallucinate, previously existing methods fall short and even the standard sequence log-probability is more informative. It means that characteristics internal to the model can give much more information than we expect, and before using external models and measures, we first need to ask: how far can we go if we use nothing but the translation model itself ? We propose to use a method that evaluates the percentage of the source contribution to a generated translation. Intuitively, hallucinations are translations "detached" from the source, hence they can be identified by low source contribution. This method improves detection accuracy for the most severe hallucinations by a factor of 2 and is able to alleviate hallucinations at test time on par with the previous best approach that relies on external models. Next, if we move away from internal model characteristics and allow external tools, we show that using sentence similarity from cross-lingual embeddings further improves these results.

Detecting and Mitigating Hallucinations in Machine Translation

The paper "Detecting and Mitigating Hallucinations in Machine Translation" addresses a notable issue within neural machine translation (NMT) systems: the occurrence of hallucinations. These hallucinations manifest when translations are generated that bear little or no relation to the source text, posing a significant challenge in ensuring translation reliability. The authors propose a methodology aimed at enhancing hallucination detection and mitigation by considering both the inner workings of translation models and leveraging external tools.

Core Contributions

The paper reveals that the fundamental mechanisms within translation models provide more information about potential hallucinations than previously recognized. The approach employs a metric evaluating the percentage of source contribution to a generated translation as an indicator of hallucination. The hypothesis is grounded in the idea that hallucinations often arise when translations become "detached" from the source, resulting in low source contribution. This internal method demonstrated a marked improvement, doubling detection accuracy for severe hallucinations compared to using sequence log-probability alone. This advancement signifies that exploring model-based characteristics is a viable direction for enhancing NMT reliability.

Upon allowing external tools, the research further utilizes sentence similarity metrics derived from cross-lingual embeddings to bolster detection capabilities. The implementation of models such as LaBSE (Language-agnostic BERT Sentence Embedding) and XNLI (Cross-lingual Natural Language Inference) demonstrated an 80% improvement in precision over previously established methods. This suggests a broader set of objectives in external models may uncover significant enhancements in handling hallucinations.

Implications and Future Directions

The implications of this research lie in the potential refinement of NMT systems both in theoretical understanding and practical applications. By harnessing internal model insights and external semantic tools, machine translation systems can be better equipped to identify and rectify hallucinations, thus improving their reliability and user trust. Practically, this could benefit fields like multilingual communication, international business, and global media consumption where accurate translation is pivotal.

The authors speculate that future advancements may involve deeper exploration into model interpretability to further understand how internal characteristics can be optimized for hallucination detection. Furthermore, expanding the scope of external tools beyond conventional quality estimation metrics could yield more innovative solutions. The integration of artificial intelligence insights with linguistic expertise stands as a promising direction for enhancing NMT systems.

Conclusion

The paper makes substantial strides in the detection and mitigation of hallucinations through leveraging model internal workings and sentence similarity measures. While challenges remain in the seamless integration of these findings into operational NMT systems, the research opens avenues for enriched translation accuracy and reliability. As artificial intelligence continues to evolve, such methodologies may play a critical role in shaping the future of machine translation.