IdiomEval: Chinese Idiom Translation Evaluation
- The paper introduces IdiomEval—a framework using human-annotated span-level error taxonomy to assess Chinese idiom translation quality.
- It evaluates 900 translation pairs from 9 systems over four domains, categorizing errors like mistranslation, literal rendering, and omissions.
- Empirical findings reveal that even top-performing models like GPT-4 make errors in 28% of cases, and standard MT metrics align poorly with human judgments.
IdiomEval is a framework for evaluating Chinese idiom translation, introduced as both an annotation scheme and a human-annotated benchmark for diagnosing how modern MT systems and LLMs translate idiomatic Chinese into English. It is centered on span-level error analysis rather than sentence-level adequacy alone: annotators first identify the English span corresponding to the Chinese idiom, then classify the translation using a structured taxonomy, and finally assign severity and confidence ratings. The benchmark contains 900 translation pairs from nine modern systems across four domains—web, news, Wikipedia, and social media—and the paper reports that even the best-performing system, GPT-4, still makes errors in 28% of cases. It also finds that existing evaluation metrics correlate weakly with human judgments, with Pearson correlation below 0.48, motivating dedicated idiom-sensitive evaluation (Yang et al., 14 Aug 2025).
1. Definition, scope, and motivation
IdiomEval was proposed to address a specific gap in evaluation: Chinese idioms are common, culturally dense, often historically grounded, and frequently non-compositional, yet there had been little systematic evaluation of how contemporary systems translate them into English (Yang et al., 14 Aug 2025). The paper argues that this is not a minor corner case of MT. Chinese idioms often behave differently from better-studied Indo-European idioms; many are proverb-like or allusive, and literal rendering is often misleading.
The framework was introduced because prior Chinese idiom resources did not provide a fine-grained, human-validated evaluation of contextual idiom translation errors across modern systems. Earlier resources emphasized cloze-style understanding, translation lexicons, parallel corpora, or small blacklist-style tests, while broader idiom benchmarks often targeted different capabilities. For example, ChID framed idiom understanding as candidate-constrained cloze selection in context (Tan et al., 2020), whereas Chengyu-Bench later organized Chinese idiom evaluation into Evaluative Connotation, Appropriateness, and Open Cloze tasks (Fu et al., 22 Jun 2025). IdiomEval instead focuses on Chinese→English idiom translation quality, with error categories intended to capture what generic MT metrics and generic idiom datasets miss (Yang et al., 14 Aug 2025).
2. Corpus construction and annotation procedure
The benchmark is built on a much larger idiom-containing Chinese corpus. The authors start from an idiom vocabulary of 30,999 idioms with definitions and collect 623K Chinese sentences containing idioms from four domains, all drawn from 2023 and 2024 to reduce contamination from model pretraining (Yang et al., 14 Aug 2025).
| Domain | Instances | Idioms |
|---|---|---|
| News | 50,845 | 5,333 |
| Web | 463,642 | 15,319 |
| Wikipedia | 39,699 | 5,947 |
| Social Media | 55,315 | 351 |
To analyze whether prior exposure matters, idioms are partitioned into five frequency bands—VH, H, M, L, and N—using an older corpus. The evaluation subset is then sampled by taking 5 instances from each frequency range for each domain, yielding 25 Chinese texts per domain and 100 Chinese texts total. Each text is translated by 9 systems, producing the final 900 translation pairs (Yang et al., 14 Aug 2025).
The evaluated systems are GPT-4, Alma-13B, Qwen-14B-Chat, mT0-13B, Google Translate, GPT-4o, Qwen2.5-7B, Qwen2.5-14B, and Qwen2.5-72B (Yang et al., 14 Aug 2025). The authors also produce manual English reference translations for each Chinese text.
Annotation proceeds in two phases. In the pilot phase, 20 participants receive detailed guidelines and complete a 30-question quiz covering every category and subcategory; the top 5 scorers continue to the main phase. In the main phase, each translation pair is annotated by 3 annotators, final labels are chosen by majority vote, and ties are manually resolved by the first author. Annotators are native Chinese speakers fluent in English, recruited via Prolific, and paid $22/hour (Yang et al., 14 Aug 2025).
Quality control is reported explicitly. On a sample of 100 annotations compared against the authors’ own annotations, the study reports 66% category accuracy and 88% subcategory accuracy. Inter-annotator agreement before tie resolution is Cohen’s $\kappa = 0.47$ for categories and 0.38 for subcategories; after majority vote and manual resolution, it rises to 0.73 and 0.69, respectively (Yang et al., 14 Aug 2025).
3. Error taxonomy
IdiomEval uses 9 high-level categories and 13 subcategories for idiom translation quality (Yang et al., 14 Aug 2025). The main operational categories are as follows.
| Category | Description |
|---|---|
| No Error | High-quality translation |
| Mistranslation | Incorrect meaning |
| Unnatural | Understandable but suboptimal wording |
| Literal | Word-for-word or compositional rendering |
| Addition | Extra irrelevant content |
| Partial | Only part of the idiom’s meaning is translated |
| Repetition | Correct content is redundantly repeated |
| No Translation | Idiom meaning is absent from output |
| More than One | Multiple error types apply |
The category No Error includes both correct figurative renderings and cases where a literal translation is contextually appropriate. The paper gives 欢呼雀跃 translated as “cheered and rejoiced” as a correct figurative rendering, and 项庄舞剑 rendered literally as “Xiang Zhuang's sword dance” as acceptable when the context refers to the historical story itself (Yang et al., 14 Aug 2025). This makes IdiomEval unusual among idiom benchmarks: it does not assume that every idiom must be translated figuratively.
Mistranslation marks semantic failure that disrupts idiom understanding. An example is 自寻烦恼 rendered as “bring them problems,” where the intended reflexive meaning is lost. Unnatural is reserved for understandable but awkward English, such as 赫赫之功 translated as “a glaring victory.” Literal captures compositional renderings that fail to express the idiomatic sense; the appendix example 身无长处 translated as “no merits in this life” is intended instead as “live in poverty” (Yang et al., 14 Aug 2025).
The appendix further refines Partial into cases such as missing modifier, missing core information, and inaccurate modifier. For instance, 罪大恶极 translated as “evil crimes” loses the intended extremity; 惊恐万状 translated as “a bit terrified” weakens an inherently strong expression. No Translation includes both omission and copied Chinese text left untranslated (Yang et al., 14 Aug 2025).
Beyond categorical labeling, IdiomEval asks annotators to rate severity and confidence on a 3-point scale. Severity is defined as minor (1), somewhat (2), and a lot (3). The paper reports that Mistranslation and No Translation are generally the most severe categories, while Unnatural, Addition, and Repetition are usually less severe because they often preserve much of the meaning (Yang et al., 14 Aug 2025).
4. Empirical findings on contemporary systems
The benchmark’s central result is that Chinese idiom translation remains difficult even for strong models. The abstract states that the best system, GPT-4, still makes errors in 28% of cases (Yang et al., 14 Aug 2025). Across the broader analysis, GPT-4 and the Qwen family have the highest overall idiom translation quality, while Alma-13B and mT0-13B perform consistently poorly (Yang et al., 14 Aug 2025).
Performance varies strongly by domain. The paper reports that GPT-4 and GPT-4o achieve more than 80% good translations on News, but this drops to around 50% on Web and around 60% on Social Media (Yang et al., 14 Aug 2025). This domain effect is one of the benchmark’s major findings: idiom translation is easiest in relatively formal news text and substantially harder in noisier, more variable domains.
The most common error types across systems are Mistranslation, Partial Translation, and Literal Translation. However, the profile is system-dependent. Google Translate is especially prone to Literal errors, including 36% literal translations on Social Media. mT0 frequently produces No Translation errors and on Wikipedia fails to translate at least 40% of idioms across frequency ranges. Alma and some Qwen systems produce Addition errors on Web and Social Media (Yang et al., 14 Aug 2025).
The paper also reports that when models do err, more than half of errors significantly impact understanding (Yang et al., 14 Aug 2025). Severity is therefore not merely cosmetic. This is reinforced by category-level analysis: Mistranslation and No Translation are not just frequent but often heavily damaging.
A notable negative finding concerns idiom frequency. IdiomEval partitions idioms into historical frequency bands, but the authors do not find a clear monotonic relationship between idiom frequency and translation quality. Even idioms in the N band—those that never appear in the old domain corpus—are not always the hardest, and on Social Media they can even receive the best translations in some cases (Yang et al., 14 Aug 2025). This suggests that pretraining exposure cannot be reduced to a single domain-specific count.
5. Metric evaluation and automatic error detection
One of IdiomEval’s main contributions is its demonstration that standard MT evaluation is poorly aligned with idiom quality. The study evaluates eight automatic metrics: BLEU, BERTScore, COMET, MetricX-23-XXL, MetricX-24-XXL, COMETKIWI, MetricX-QE-23-XXL, and MetricX-QE-24-XXL (Yang et al., 14 Aug 2025). Human labels are collapsed to a numeric severity scale with Good translation = 0 and errors mapped to 1, 2, 3.
For full-context evaluation over all categories, the highest reported Pearson correlation is 0.386 for COMET. For idiom-only evaluation, the highest is 0.483 for MetricX-23-XXL (Yang et al., 14 Aug 2025). The paper therefore concludes that current metrics are only weakly aligned with human judgments. Some metric-category combinations are even negatively correlated, especially for No Translation and some Partial cases.
The binary formulation used later in the paper is explicit: let be the idiom, the source Chinese text containing , and the English translation of ; the task is to predict whether is correctly translated in (Yang et al., 14 Aug 2025). Even on this easier good-vs-bad distinction, the metrics remain unreliable. The best idiom-level ROC-AUC is 0.78 for MetricX-23-XXL, and the authors note that even the best metrics still rank bad translations above good ones in more than 20% of pairs (Yang et al., 14 Aug 2025).
The abstract adds a further result: the authors develop improved models that achieve for detecting idiom translation errors (Yang et al., 14 Aug 2025). The details excerpt does not elaborate this modeling section, but the abstract makes clear that idiom-specific error detection performs meaningfully better than generic evaluation heuristics.
6. Position within idiom evaluation research
IdiomEval is a translation-specific benchmark, but its design addresses questions central to idiom evaluation more broadly. Relative to cloze-style idiom selection benchmarks such as ChID, it replaces candidate selection with span-level translation diagnosis (Tan et al., 2020). Relative to Chengyu-Bench, which evaluates connotation, contextual appropriateness, and open cloze for Chinese idioms, IdiomEval isolates the separate problem of whether idiomatic meaning survives translation into English (Fu et al., 22 Jun 2025). Relative to IdioLink, which evaluates semantic retrieval across idiomatic and literal expressions, IdiomEval adds human span annotation and a translation-specific error taxonomy (Hashiloni et al., 21 May 2026). Relative to G-IdiomAlign, which studies cross-lingual idiom equivalence via gloss pivots, IdiomEval focuses on observed MT outputs rather than controlled alignment or generation protocols (Ye et al., 17 Jun 2026).
This suggests a broader interpretation: IdiomEval does not treat idioms merely as lexical items to be recognized, but as translation units whose failures can be literal, partial, missing, unnatural, or semantically wrong in different ways. Its most distinctive contribution is therefore methodological. It supplies a human-centered taxonomy for Chinese idiom translation, shows that generic MT metrics are inadequate for this problem, and provides a concrete benchmark in which idiom translation quality can be studied separately from overall sentence fluency (Yang et al., 14 Aug 2025).
