---
title: 'IdiomEval: Chinese Idiom Translation Evaluation'
url: https://www.emergentmind.com/topics/idiomeval
type: topic
---

# IdiomEval: Chinese Idiom Translation Evaluation

IdiomEval is a framework for evaluating **Chinese idiom translation**, introduced as both an annotation scheme and a human-annotated benchmark for diagnosing how modern MT systems and LLMs translate idiomatic Chinese into English. It is centered on span-level error analysis rather than sentence-level adequacy alone: annotators first identify the English span corresponding to the Chinese idiom, then classify the translation using a structured taxonomy, and finally assign severity and confidence ratings. The benchmark contains **900 translation pairs** from **nine modern systems** across **four domains**—**web, news, Wikipedia, and social media**—and the paper reports that even the best-performing system, **GPT-4**, still makes errors in **28%** of cases. It also finds that existing evaluation metrics correlate weakly with human judgments, with **Pearson correlation below 0.48**, motivating dedicated idiom-sensitive evaluation [2508.10421].

## 1. Definition, scope, and motivation

IdiomEval was proposed to address a specific gap in evaluation: Chinese idioms are common, culturally dense, often historically grounded, and frequently non-compositional, yet there had been little systematic evaluation of how contemporary systems translate them into English [2508.10421]. The paper argues that this is not a minor corner case of MT. Chinese idioms often behave differently from better-studied Indo-European idioms; many are proverb-like or allusive, and literal rendering is often misleading. 

The framework was introduced because prior Chinese idiom resources did not provide a fine-grained, human-validated evaluation of **contextual idiom translation errors** across modern systems. Earlier resources emphasized cloze-style understanding, translation lexicons, parallel corpora, or small blacklist-style tests, while broader idiom benchmarks often targeted different capabilities. For example, **ChID** framed idiom understanding as candidate-constrained cloze selection in context [2011.02378], whereas **Chengyu-Bench** later organized Chinese idiom evaluation into **Evaluative Connotation**, **Appropriateness**, and **Open Cloze** tasks [2506.18105]. IdiomEval instead focuses on **Chinese→English idiom translation quality**, with error categories intended to capture what generic MT metrics and generic idiom datasets miss [2508.10421].

## 2. Corpus construction and annotation procedure

The benchmark is built on a much larger idiom-containing Chinese corpus. The authors start from an idiom vocabulary of **30,999 idioms with definitions** and collect **623K Chinese sentences containing idioms** from four domains, all drawn from **2023 and 2024** to reduce contamination from model pretraining [2508.10421].

| Domain | Instances | Idioms |
|---|---:|---:|
| News | 50,845 | 5,333 |
| Web | 463,642 | 15,319 |
| Wikipedia | 39,699 | 5,947 |
| Social Media | 55,315 | 351 |

To analyze whether prior exposure matters, idioms are partitioned into five frequency bands—**VH**, **H**, **M**, **L**, and **N**—using an older corpus. The evaluation subset is then sampled by taking **5 instances from each frequency range** for each domain, yielding **25 Chinese texts per domain** and **100 Chinese texts total**. Each text is translated by **9 systems**, producing the final **900 translation pairs** [2508.10421].

The evaluated systems are **GPT-4**, **Alma-13B**, **Qwen-14B-Chat**, **mT0-13B**, **Google Translate**, **GPT-4o**, **Qwen2.5-7B**, **Qwen2.5-14B**, and **Qwen2.5-72B** [2508.10421]. The authors also produce manual English reference translations for each Chinese text.

Annotation proceeds in two phases. In the **pilot phase**, **20 participants** receive detailed guidelines and complete a **30-question quiz** covering every category and subcategory; the **top 5 scorers** continue to the main phase. In the **main phase**, each translation pair is annotated by **3 annotators**, final labels are chosen by **majority vote**, and ties are manually resolved by the first author. Annotators are **native Chinese speakers fluent in English**, recruited via **Prolific**, and paid **$22/hour** [2508.10421].

Quality control is reported explicitly. On a sample of **100 annotations** compared against the authors’ own annotations, the study reports **66% category accuracy** and **88% subcategory accuracy**. Inter-annotator agreement before tie resolution is **Cohen’s \(\kappa = 0.47\)** for categories and **0.38** for subcategories; after majority vote and manual resolution, it rises to **0.73** and **0.69**, respectively [2508.10421].

## 3. Error taxonomy

IdiomEval uses **9 high-level categories** and **13 subcategories** for idiom translation quality [2508.10421]. The main operational categories are as follows.

| Category | Description |
|---|---|
| No Error | High-quality translation |
| Mistranslation | Incorrect meaning |
| Unnatural | Understandable but suboptimal wording |
| Literal | Word-for-word or compositional rendering |
| Addition | Extra irrelevant content |
| Partial | Only part of the idiom’s meaning is translated |
| Repetition | Correct content is redundantly repeated |
| No Translation | Idiom meaning is absent from output |
| More than One | Multiple error types apply |

The category **No Error** includes both correct figurative renderings and cases where a literal translation is contextually appropriate. The paper gives **欢呼雀跃** translated as “**cheered and rejoiced**” as a correct figurative rendering, and **项庄舞剑** rendered literally as “**Xiang Zhuang's sword dance**” as acceptable when the context refers to the historical story itself [2508.10421]. This makes IdiomEval unusual among idiom benchmarks: it does not assume that every idiom must be translated figuratively.

**Mistranslation** marks semantic failure that disrupts idiom understanding. An example is **自寻烦恼** rendered as “bring **them** problems,” where the intended reflexive meaning is lost. **Unnatural** is reserved for understandable but awkward English, such as **赫赫之功** translated as “a **glaring victory**.” **Literal** captures compositional renderings that fail to express the idiomatic sense; the appendix example **身无长处** translated as “**no merits in this life**” is intended instead as “live in poverty” [2508.10421].

The appendix further refines **Partial** into cases such as **missing modifier**, **missing core information**, and **inaccurate modifier**. For instance, **罪大恶极** translated as “evil crimes” loses the intended extremity; **惊恐万状** translated as “a bit terrified” weakens an inherently strong expression. **No Translation** includes both omission and copied Chinese text left untranslated [2508.10421].

Beyond categorical labeling, IdiomEval asks annotators to rate **severity** and **confidence** on a 3-point scale. Severity is defined as **minor (1)**, **somewhat (2)**, and **a lot (3)**. The paper reports that **Mistranslation** and **No Translation** are generally the most severe categories, while **Unnatural**, **Addition**, and **Repetition** are usually less severe because they often preserve much of the meaning [2508.10421].

## 4. Empirical findings on contemporary systems

The benchmark’s central result is that Chinese idiom translation remains difficult even for strong models. The abstract states that the best system, **GPT-4**, still makes errors in **28%** of cases [2508.10421]. Across the broader analysis, **GPT-4** and the **Qwen** family have the highest overall idiom translation quality, while **Alma-13B** and **mT0-13B** perform consistently poorly [2508.10421].

Performance varies strongly by domain. The paper reports that **GPT-4** and **GPT-4o** achieve **more than 80% good translations on News**, but this drops to around **50% on Web** and around **60% on Social Media** [2508.10421]. This domain effect is one of the benchmark’s major findings: idiom translation is easiest in relatively formal news text and substantially harder in noisier, more variable domains.

The most common error types across systems are **Mistranslation**, **Partial Translation**, and **Literal Translation**. However, the profile is system-dependent. **Google Translate** is especially prone to **Literal** errors, including **36% literal translations on Social Media**. **mT0** frequently produces **No Translation** errors and on **Wikipedia** fails to translate **at least 40% of idioms** across frequency ranges. **Alma** and some **Qwen** systems produce **Addition** errors on Web and Social Media [2508.10421].

The paper also reports that when models do err, **more than half of errors significantly impact understanding** [2508.10421]. Severity is therefore not merely cosmetic. This is reinforced by category-level analysis: **Mistranslation** and **No Translation** are not just frequent but often heavily damaging.

A notable negative finding concerns idiom frequency. IdiomEval partitions idioms into historical frequency bands, but the authors do **not** find a clear monotonic relationship between idiom frequency and translation quality. Even idioms in the **N** band—those that never appear in the old domain corpus—are not always the hardest, and on Social Media they can even receive the best translations in some cases [2508.10421]. This suggests that pretraining exposure cannot be reduced to a single domain-specific count.

## 5. Metric evaluation and automatic error detection

One of IdiomEval’s main contributions is its demonstration that standard MT evaluation is poorly aligned with idiom quality. The study evaluates eight automatic metrics: **BLEU**, **BERTScore**, **COMET**, **MetricX-23-XXL**, **MetricX-24-XXL**, **COMETKIWI**, **MetricX-QE-23-XXL**, and **MetricX-QE-24-XXL** [2508.10421]. Human labels are collapsed to a numeric severity scale with **Good translation = 0** and errors mapped to **1, 2, 3**.

For **full-context** evaluation over all categories, the highest reported **Pearson correlation** is **0.386** for **COMET**. For **idiom-only** evaluation, the highest is **0.483** for **MetricX-23-XXL** [2508.10421]. The paper therefore concludes that current metrics are only weakly aligned with human judgments. Some metric-category combinations are even negatively correlated, especially for **No Translation** and some **Partial** cases.

The binary formulation used later in the paper is explicit: let \(i\) be the idiom, \(s\) the source Chinese text containing \(i\), and \(t\) the English translation of \(s\); the task is to predict whether \(i\) is correctly translated in \(t\) [2508.10421]. Even on this easier good-vs-bad distinction, the metrics remain unreliable. The best idiom-level **ROC-AUC** is **0.78** for **MetricX-23-XXL**, and the authors note that even the best metrics still rank bad translations above good ones in **more than 20%** of pairs [2508.10421].

The abstract adds a further result: the authors develop improved models that achieve **\(F_1 = 0.68\)** for **detecting idiom translation errors** [2508.10421]. The details excerpt does not elaborate this modeling section, but the abstract makes clear that idiom-specific error detection performs meaningfully better than generic evaluation heuristics.

## 6. Position within idiom evaluation research

IdiomEval is a translation-specific benchmark, but its design addresses questions central to idiom evaluation more broadly. Relative to **cloze-style idiom selection** benchmarks such as **ChID**, it replaces candidate selection with span-level translation diagnosis [2011.02378]. Relative to **Chengyu-Bench**, which evaluates connotation, contextual appropriateness, and open cloze for Chinese idioms, IdiomEval isolates the separate problem of whether idiomatic meaning survives translation into English [2506.18105]. Relative to **IdioLink**, which evaluates semantic retrieval across idiomatic and literal expressions, IdiomEval adds human span annotation and a translation-specific error taxonomy [2605.22247]. Relative to **G-IdiomAlign**, which studies cross-lingual idiom equivalence via gloss pivots, IdiomEval focuses on observed MT outputs rather than controlled alignment or generation protocols [2606.18989].

This suggests a broader interpretation: IdiomEval does not treat idioms merely as lexical items to be recognized, but as translation units whose failures can be **literal**, **partial**, **missing**, **unnatural**, or semantically wrong in different ways. Its most distinctive contribution is therefore methodological. It supplies a human-centered taxonomy for **Chinese idiom translation**, shows that generic MT metrics are inadequate for this problem, and provides a concrete benchmark in which idiom translation quality can be studied separately from overall sentence fluency [2508.10421].

Source: https://www.emergentmind.com/topics/idiomeval