How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs

Abstract: Recent research has focused on literary machine translation (MT) as a new challenge in MT. However, the evaluation of literary MT remains an open problem. We contribute to this ongoing discussion by introducing LITEVAL-CORPUS, a paragraph-level parallel corpus containing verified human translations and outputs from 9 MT systems, which totals over 2k translations and 13k evaluated sentences across four language pairs, costing 4.5k C. This corpus enables us to (i) examine the consistency and adequacy of human evaluation schemes with various degrees of complexity, (ii) compare evaluations by students and professionals, assess the effectiveness of (iii) LLM-based metrics and (iv) LLMs themselves. Our findings indicate that the adequacy of human evaluation is controlled by two factors: the complexity of the evaluation scheme (more complex is less adequate) and the expertise of evaluators (higher expertise yields more adequate evaluations). For instance, MQM (Multidimensional Quality Metrics), a complex scheme and the de facto standard for non-literary human MT evaluation, is largely inadequate for literary translation evaluation: with student evaluators, nearly 60% of human translations are misjudged as indistinguishable or inferior to machine translations. In contrast, BWS (BEST-WORST SCALING), a much simpler scheme, identifies human translations at a rate of 80-100%. Automatic metrics fare dramatically worse, with rates of at most 20%. Our overall evaluation indicates that published human translations consistently outperform LLM translations, where even the most recent LLMs tend to produce considerably more literal and less diverse translations compared to humans.

• The paper shows that human translations remain superior to LLMs in capturing stylistic richness and literary creativity.
• It employs diverse evaluation metrics including MQM, SQM, and BWS on the newly introduced LitEval-Corpus to assess translation quality.
• Findings underscore the need for improved automatic metrics to better capture the complex nuances inherent in literary texts.

Exploring the Capacities of LLMs in Literary Translation

Introduction

The research investigates the efficacy of LLMs in the domain of literary machine translation (MT). To assess these models, the study presents the LitEval-Corpus, a rigorously curated dataset designed to evaluate literary translations across four languages. This corpus includes human translations and outputs from multiple MT systems, allowing a thorough comparison of various evaluation metrics.

Evaluation Framework

The research evaluates three core annotation schemes—Multidimensional Quality Metrics (MQM), Scalar Quality Metric (SQM), and Best-Worst Scaling (BWS). The study finds that while MQM is a standard in non-literary MT evaluation, it fails in literary contexts given its inability to capture the nuances of literary translations. As a result, the research highlights the need for evaluation frameworks that better align with the complexities inherent in literary translation tasks.

Figure 1: Segment level correlation measured by Kendall's tau between human MQM and evaluation metrics and between human MQM and GEMBA-MQM (Literary) per error categories.

Dataset and Translation Models

The LitEval-Corpus covers an array of translation pairs and includes both classic and contemporary works, ensuring a comprehensive dataset. MT systems analyzed include Google Translate, DeepL, GPT-4o, and other lesser-known models. The study methodically distinguishes between models in terms of their syntactic similarity and lexical diversity, offering insights into the models' behavior beyond mere accuracy.

Findings on LLM Performance

Figure 2: Scatter plots of human MQM per average syntactic similarity between source and system translation and average lexical overlap of the current system to all other systems.

The findings indicate that professional human translations continue to outperform even the most advanced LLMs like GPT-4o, specifically in stylistic richness and creativity. Automatons, while improving, tend to produce more literal and less nuanced translations. Automatic metrics such as GEMBA-MQM show moderate alignment with human assessments but fail to adequately distinguish human-level translations from those produced by machines in more nuanced artistic domains.

Implications and Future Work

The research underscores the limitation of existing automatic metrics in fully capturing the quality of literary translations and suggests improvements for future metric development. The gaps highlighted by the study suggest pathways for enhancing the translation capabilities of LLMs, particularly by focusing on improving diversity and stylistic adaptation.

Conclusion

Overall, the study provides insightful commentary on the current state and direction of LLM capabilities in literary translation. As it stands, while LLMs offer sophisticated capabilities, the richness of human literary translation still eludes them. Future research addressing these identified gaps is imperative for advancing the fidelity and artistic quality of automated literary translation systems. This work lays the foundation for ongoing enhancements in both evaluation methods and LLM architectures to bridge the current disparity in translation quality.