MetaHOPE: A Metaphor-Oriented Evaluation Framework for Analysing MT and LLM Translation Errors

Abstract: In this opinion paper, we propose MetaHOPE, an error severity-aware annotation framework for evaluating metaphor translations. Metaphors present challenges for machine translation (MT) and natural language understanding and processing (NLU, NLP), because it presents the features of semantic complexity, contextual dependency, and cultural embeddings that can lead to ambiguity issues for NLP models. To investigate how state-of-the-art NLP models perform on translating metaphors, we select three representative systems, i.e., GoogleMT, GPT5.4, and Hunyuan-7b as Neural MT (NMT) models and LLMs. We used two human-annotated metaphor corpora, including VUAMC and PSUCMC for English-to-Chinese and Chinese-to-English translation purposes. The original corpora we used are monolingual, where we carried out error annotation using the MetaHOPE framework, and also produced the human post-edited gold reference for bilingual use as a new resource. We believe the MetaHOPE evaluation framework for metaphor translation annotation, the parallel corpora resources, and the error analysis on SOTA automatic translation models can be useful and shed some light for the field of metaphor translation study. We share our resources publicly upon paper acceptance.

• The paper presents MetaHOPE, a novel evaluation framework that systematically categorizes and quantifies translation errors in metaphorical language.
• It applies a linear 5-level severity scale over five error categories, enabling detailed cross-lingual comparisons and highlighting cultural adaptation challenges.
• Pilot experiments on English-Chinese corpora reveal that metaphor errors dominate translation outputs in state-of-the-art systems, stressing the need for specialized evaluation.

MetaHOPE: A Structured Framework for Systematic Evaluation of Metaphor Translation in MT and LLMs

Motivation and Context

Metaphor translation remains an unresolved challenge for machine translation (MT) and LLMs due to the inherent semantic complexity, contextual sensitivity, and cultural specificity of metaphorical language. Current neural MT (NMT) and LLMs exhibit strong results on general translation benchmarks, yet perform suboptimally on figurative language. Existing research indicates that metaphor translation accuracy for these models stagnates at 64–80%, with roughly 20% of metaphorical expressions being mistranslated or undergoing non-equivalent transfer [karakanta2025metaphors; wang2024mmte]. Error analysis reveals a predominance of literal, non-idiomatic, or contextually inappropriate renderings, particularly for multi-word expressions and idioms [mwe-2023-multiword]. Metaphor evaluation in MT research remains largely coarse-grained, lacking robust and severity-sensitive error categorization.

MetaHOPE Framework: Design and Methodology

The paper proposes MetaHOPE, a metaphor-oriented adaptation of the existing HOPE annotation framework, for the systematic detection and severity-based rating of metaphor translation errors from English to Chinese and vice versa.

Figure 1: MetaHOPE Framework: Metaphor corpus preparation, MT, Aligning segments and metaphor-related words (MRWs), and Post-editing with annotations on pilot/development and test sets.

MetaHOPE focuses on five error categories:

• Impact (IMP): Over-literalness or structural shifts altering the intended emphasis or agency.
• Required Adaptation Missing (RAM): Absence of necessary cultural or idiomatic adaptation in target language.
• Mistranslation (MIS): Inaccuracies such as meaning mismatch or incorrect interpretation of the original figurative sense.
• Style (STL): Diminution or loss of rhetorical, emotional, or stylistic force.
• Proofreading Error (PRF): Surface-level unnaturalness or awkwardness not reflecting true meaning.

Severity is quantified on a linear 5-level scale (minor to critical), addressing disagreement and annotation sparsity noted in traditional exponential schemes. Notably, this operationalizes cross-linguistic, cognitively motivated translation theory into fine-grained, empirical evaluation criteria.

The evaluation pipeline involves:

1. Preparation and formatting of metaphor-annotated corpora (VUAMC for English, PSUCMC for Chinese).

   Figure 2: PSUCMC formatting.

   

   Figure 3: VUAMC formatting.

2. Context-aware MT using three representative systems: GoogleMT, GPT-5.4, and Hunyuan-LLM-7B.
3. Manual alignment of metaphor-related words (MRWs) and source-target segment mapping.
4. Pilot annotation and error categorization, followed by expanded evaluation and post-edited human reference creation.

Alignment principles emphasize meaning-preserving transfer rather than simplistic lexical correspondence; figure illustrates diverse realization strategies for metaphor in translation outputs.

Figure 4: Metaphor translation alignment examples realized through different translational patterns.

Experimental Setup and Error Analysis

Corpora and Annotation

Two parallel, monolingual, human-annotated metaphor corpora (VUAMC, PSUCMC) are used for English-Chinese/Chinese-English translation, with 200 metaphor-bearing sentences selected for each direction. Only content words (nouns, verbs, adjectives, adverbs) are analyzed, excluding structure-triggered shifts inherent to function words.

Annotation is performed by two domain experts with linguistic and translation backgrounds. Inter-annotator agreement is assessed using Krippendorff’s $\alpha$, quadratic weighted Cohen’s $\kappa$, and Pearson’s $r$.

Quantitative Results

• High Prevalence of Metaphor-Caused Errors: For the pilot set, metaphor translation errors account for 91.7% (GoogleMT), 93.8% (GPT-5.4), and 61.8% (Hunyuan-7b) of total error penalties. Thus, metaphor remains a primary cause of translation penalty across systems.
• Annotator Agreement: GPT-5.4 yields the highest inter-annotator consistency ($r = 0.726$), while Hunyuan-LLM-7B shows greater annotation difficulty due to higher output variability.

Qualitative Observations and Error Phenomena

• Translation Style Divergence: GoogleMT and GPT-5.4 favor conservative, source-tracking literal translations, while Hunyuan-LLM-7B demonstrates increased flexibility and localization, occasionally resulting in hallucinated or overly liberal interpretations.
• Cultural Adaptation and Metaphor Loss: In several instances, metaphors are paraphrased or omitted, leading to degraded rhetorical effect, diminished emotional resonance, or imprecise conceptual mapping.
• Alignment and Consistency Issues: Inconsistent term translations and difficulties in precise MRW alignment are observed, emphasizing the necessity of context-aware, manual alignment protocols.
• Interchangeable Lexical Choices in Chinese: Modern regulatory standards and the presence of synonyms complicate the establishment of canonical targets for metaphor translation.

Theoretical and Practical Implications

MetaHOPE occupies a unique position at the intersection of cognitive-linguistic theory and empirical MT evaluation. Existing work on metaphor translation evaluation has prioritized either strategic-documentary analysis or overall adequacy/fluency scoring, neglecting the systematic, severity-sensitive assessment of figurative language transfer.

This framework operationalizes cognitive translation concerns—such as conceptual mapping, pragmatic-cultural adaptation, and stylistic preservation—into a robust annotation protocol, facilitating detailed cross-system, cross-lingual error comparisons. The reliance on manual, context-sensitive alignment addresses ongoing challenges in automatic cross-lingual alignment for polysemous, context-dependent language [pallucchini2025lost; miao2024enhancing].

The pragmatic consequences are substantial: MetaHOPE can inform model diagnosis, MT training for underrepresented linguistic features (e.g., metaphor, idiom), and targeted refinements for LLM prompting regarding non-literal expressions. It also supports the creation of new evaluation resources and potentially benchmarks for figurative language in MT, complementing existing efforts in idiom translation [donthi-etal-2025-improving].

Limitations and Future Directions

The presented pilot constitutes a proof-of-concept; annotation scale and agreement require further expansion and refinement for robust generalization. The study is initially circumscribed to the news domain, but application to literary, academic, and political registers will increase coverage and provide broader insights into MT and LLM limitations.

Potential extensions include:

• Scaling to full-sized test sets and higher annotator throughput, with more sophisticated agreement analysis.
• Incorporation of metaphor typology (novelty, deliberateness, universality) for fine-grained system characterization.
• Semi-automated or LLM-assisted annotation to streamline resource creation.
• Cross-model ablation studies and error attribution at the representation, prompting, or architecture layer.

Conclusion

MetaHOPE constitutes an analytically rigorous framework for evaluating metaphor translation in NMT and LLMs, bridging theoretical linguistics and applied MT evaluation. Early results highlight the predominance of metaphor translation errors in current SOTA systems and underscore the need for explicit, context- and severity-aware analysis of figurative language rendering. The framework establishes a foundation for systematic benchmarking, resource development, and rapid diagnostic feedback in the development of metaphor-sensitive MT and LLM systems. Its adoption will enable more nuanced progress in cross-lingual, cross-cultural NLU, and translation system design.

---

References

• "Metaphors in Literary Machine Translation: Close but no cigar?" [karakanta2025metaphors]
• "MMTE: Corpus and metrics for evaluating machine translation quality of metaphorical language" [wang2024mmte]
• "Mind vs. machine: Comparative analysis of metaphor-related word translation by human and AI systems" [li2025mindmachine]
• "Lost in alignment: A survey on cross-lingual alignment methods for contextualized representation" [pallucchini2025lost]
• "Enhancing cross-lingual sentence embedding for low-resource languages with word alignment" [miao2024enhancing]
• "Improving LLM Abilities in Idiomatic Translation" [donthi-etal-2025-improving]