---
title: Last Translation Benchmark
url: https://www.emergentmind.com/topics/last-translation-benchmark
type: topic
---

# Last Translation Benchmark

The **Last Translation Benchmark (LTB)** is a live, multimodal benchmark for evaluating machine-translation systems on deliberately difficult, human-authored examples that expose specific failure modes. Introduced in 2026, it combines text, images, audio, video, translation instructions, human reference translations, and handcrafted verification rules. Instead of reducing quality to a continuous metric such as BLEU, COMET, or an unconstrained LLM-judge score, LTB evaluates whether a candidate translation satisfies every concrete requirement attached to an example. Its first release, **LTBv1**, contains accepted contributions submitted before September 1, 2026; subsequent releases are planned as new examples are collected [2609.04173].

## 1. Motivation and evaluation philosophy

LTB was developed in response to the saturation of conventional machine-translation benchmarks. Static test sets sampled from ordinary text distributions increasingly produce high scores for strong models while failing to distinguish important residual weaknesses. A model may translate most sentences fluently yet fail on a single gender distinction, idiom, cultural reference, ambiguity, multimodal cue, register requirement, or output constraint.

The benchmark therefore targets a diagnostic question: whether a system can solve carefully selected examples that human contributors regard as fair and translatable but that leading translation systems currently fail. It is intentionally not representative of average translation traffic. Rather, it functions as a stress test and long-term goalpost whose eventual aspiration is performance approaching \(100\%\) on its examples.

LTB identifies several limitations in existing evaluation methods:

- **Surface-overlap metrics** such as BLEU and ChrF reward similarity to a reference and may penalize valid alternatives.
- **Neural metrics** such as COMET and MetricX generally correlate better with human judgments but can be difficult to interpret, vulnerable to out-of-domain behavior and reward hacking, and subject to evaluator bias.
- **Generic LLM judges** may overlook the very phenomenon that makes an example difficult, prefer outputs resembling their own, or produce scores that do not specify an actionable failure.
- **Human evaluation** is expensive, slow, subjective, difficult to reproduce, and dependent on annotator expertise and campaign-specific calibration.

LTB retains human expertise at the point where it is most informative: example design, reference translation, and verification-rule construction. Evaluation then becomes reusable, scalable, and explicit. The benchmark’s central distinction is between a generic judgment that a translation is “mostly correct” and a verifiable diagnosis that it selected the wrong sense, violated a length limit, failed to preserve ambiguity, mistranslated an image, or used an inappropriate register.

## 2. Dataset construction and governance

LTB is crowdsourced through an online platform. A contributor selects a source and target language, supplies an input, provides a human reference translation, runs the input through available translation models, identifies observed failures, and writes verification rules targeting those failures.

Each submission is reviewed by one contributor sufficiently proficient in the relevant language pair. Reviewers assess whether:

- the example follows the submission guidelines;
- it is genuinely translatable;
- an expert could reasonably produce a passing translation;
- the human reference is correct;
- the failure is perceptible and substantively important; and
- the verification rules are valid, neither excessively narrow nor generic.

Contributors with at least ten accepted examples are invited to become dataset co-authors. The paper reports **244 dataset contributors from 166 institutions** [2609.04173].

An example is considered to break leading systems when most tested models fail at least one attached rule. The submission interface can display up to ten automatic translations. For acceptance, **all but two of the ten automatic translations must fail at least one rule**, while the contributor’s human translation must pass. A model need not produce unintelligible output to fail: a fluent translation can still violate a crucial semantic, grammatical, cultural, multimodal, or user-specified requirement.

LTB is continuously updated rather than frozen. Releases are tagged, the displayed model set can change, and new contributions can be used to maintain a challenging and comparatively uncontaminated evaluation set. LTBv1 and the text-only **LTBv1-eval** subset are the principal releases described in the paper. The dataset is available under **CC BY 4.0**, and the code is released under an **MIT** license [2609.04173].

## 3. Modalities, languages, and failure phenomena

LTB accepts ordinary text, images, audio, video, and multimodal combinations. A non-text modality can be the primary translation input: text embedded in an image or speech in an audio recording must be translated directly. Multimodal context can also disambiguate accompanying text, such as a photograph attached to a social-media post.

Contributors may specify instructions requiring:

- casual or formal language;
- preservation of a particular tone;
- an output-length limit;
- a formatting constraint; or
- a particular output language.

Examples may involve standard languages, dialects, regional varieties, scripts, or user-defined forms, including variants such as “Swiss German (Zurich)” and “Serbian (Cyrillic).”

The benchmark includes examples involving:

- grammatical gender;
- idioms and nonliteral expressions;
- puns and wordplay;
- polysemy;
- cultural knowledge;
- internet terminology;
- register and politeness;
- sarcasm;
- ambiguity preservation;
- code-mixed language;
- onomatopoeia;
- low-resource languages;
- multimodal interpretation;
- formatting and length constraints;
- prompt-injection-like content embedded in source material.

A representative gender example is the English-to-Czech sentence:

> “The two new nurses share the 100m men's world record.”

The surrounding context identifies the nurses as men, but leading systems reportedly selected a feminine Czech form. The corresponding rule requires a masculine form that does not imply feminine gender.

Other reported cases include literal translation of an idiom, incorrect interpretation of a respectful Korean address, failure to preserve a pun in Thai, mistranslation of sarcasm in German audio, confusion of an academic “paper” with physical *Papier* in German, and failure to translate Chinese Sign Language while attending to a textual distractor. LTB also includes cases involving Sandawe, Latin, Catalan, Hausa, Japanese, French, Italian, Czech, and other languages [2609.04173].

## 4. Handcrafted verification rules

Each example contains one or more atomic, pass/fail rules specifying what a successful translation must accomplish. Examples include:

- “The word for ‘paper’ should be ‘Artikel’ or the English loanword ‘Paper,’ but not ‘Papier.’”
- “Check whether the translation conveys that the speaker feels no inner peace.”
- “The translation must preserve the ambiguity as to whether the person will return or die.”

A candidate passes an example only if it satisfies every associated rule. If \(R_i\) denotes the rules for example \(i\), and \(v_{ij}(y_i)\) is the binary result for candidate translation \(y_i\) on rule \(j\), example-level success can be represented as:

$$
s_i(y_i)=\prod_{j\in R_i} v_{ij}(y_i).
$$

The benchmark pass rate for a model \(M\) is:

$$
\mathrm{PassRate}(M)=
\frac{1}{N}\sum_{i=1}^{N}s_i\left(y_i^{(M)}\right)\times 100.
$$

Thus, an example receives credit only when all of its requirements are met. The pass rate has a direct interpretation as the percentage of difficult examples successfully translated.

The verification rules are deliberately asymmetric. The translation model is not shown the rules in the recommended evaluation condition and must infer the relevant difficulty from the input. The verifier, by contrast, receives explicit information about what to inspect. This makes the evaluation reliable even when the verifier could not independently generate a perfect translation.

LTB supports two modes:

- **Blind evaluation**: the translation system sees the input but not the verification rules. This is the recommended mode for realistic system evaluation.
- **Oracle evaluation**: the system receives rules, the human translation, or other privileged information. This measures whether a model can solve a problem once the relevant failure mode has been identified.

The distinction separates two sources of failure: inability to recognize the important phenomenon and inability to produce a translation satisfying a known requirement. The paper reports that providing rules can substantially improve model performance, although some requirements—such as producing a creative pun—remain difficult even when explicitly stated.

## 5. LTBv1 statistics and evaluation resources

LTBv1 contains:

- **3,456 accepted examples**;
- **109 main languages**;
- approximately **94% textual examples**;
- average input length of **19 words** or **104 characters**;
- an average of **1.9 verification rules per example**;
- **10%** of examples containing translation instructions;
- **73%** non-English-to-English examples;
- **14%** English-to-non-English examples;
- **13%** non-English-to-non-English examples;
- **911** text-only examples in LTBv1-eval.

The language distribution is uneven. Reported language counts include English (2,997), German (354), Telugu (299), Chinese (288), Spanish (227), French (222), Hindi (178), Italian (160), Arabic (130), Bengali (117), Catalan (100), Persian (98), and Japanese (98), among many others.

Among non-English languages, the reported resource-level distribution is:

- ultra-high: \(30.5\%\);
- high: \(28.3\%\);
- medium: \(19.0\%\);
- low: \(3.2\%\);
- minimal: \(15.2\%\);
- zero: \(3.8\%\).

The reported language-family distribution, excluding English, is:

- Indo-European: \(58.9\%\);
- Afro-Asiatic: \(8.5\%\);
- Dravidian: \(8.4\%\);
- Sino-Tibetan: \(7.5\%\);
- other: \(16.8\%\).

These statistics demonstrate broad coverage, including low-resource languages, dialects, historical varieties, scripts, and unusual language forms. They also reveal concentration in English and Indo-European languages, so LTBv1 should not be interpreted as a balanced sample of the world’s linguistic diversity.

## 6. Results, reliability, and limitations

LTBv1-eval compares 29 models using handcrafted verification, generic LLM judges, MetricX QE 24 and MetricX 24, COMET QE 22 and COMET 22, ChrF, and human evaluation on a subset.

The principal result is a large discrepancy between targeted verification and generic quality scores. Human translations achieve approximately \(99.0\%-99.9\%\) verifier pass rates. The strongest listed automatic systems achieve substantially lower values:

- GPT-5.6 Sol: approximately \(28.1\%-36.1\%\), depending on verifier;
- Gemini 3.1 Pro: approximately \(37.4\%-41.9\%\);
- GPT-5.6 Luna: approximately \(15.4\%-22.9\%\).

Many conventional systems score below \(10\%\), while Google Translate scores approximately \(1.1\%-3.7\%\), depending on the verifier. Generic LLM judges assign much higher values to some of the same outputs: GPT-5.6 Sol receives generic-judge scores of approximately \(76.4\%-89.7\%\) despite a verifier pass rate of only about \(28\%-36\%\).

The human references score \(100\) on ChrF in the table’s normalized presentation, \(93.5\) on MetricX 24, \(94.0\) on COMET 22, \(81.4\) in human evaluation without rules, and \(90.8\) in human evaluation with rules. The high verifier performance of human references is partly structural, since submissions are accepted only when the reference translation satisfies the attached rules.

Verifier-based model rankings are more stable than rankings from generic judges or neural metrics. The reported average pairwise Kendall ranking similarities are:

| Comparison | Average ranking similarity |
|---|---:|
| Verifier–verifier | 86.9 |
| Judge–judge | 71.3 |
| Metric–metric | 35.1 |
| Verifier–judge | 51.3 |
| Verifier–metric | 31.0 |
| Judge–metric | 34.2 |
| Verifier–human | 90.5 |
| Judge–human | 34.9 |
| Metric–human | 16.2 |

The paper also reports greater low-data stability for verifier-based ranking. On a \(0.1\%\) subsample, similarity with the full-set ranking is 44.2 for verifiers, 23.0 for judges, and 16.2 for metrics.

LTB has several limitations. Its accepted examples are intentionally difficult and therefore do not estimate average translation quality. The language distribution is highly uneven, with substantial English and Indo-European concentration. The benchmark is crowdsourced, and contributors and reviewers may disagree about the importance or difficulty of rules. Human references are accepted through the same rule-based process and are not necessarily definitive translations. The evaluation is also dependent on verifier prompts and models, although the explicit rules make those dependencies more inspectable than generic quality scoring.

Verification rules can be overly narrow, overly broad, or incomplete. Automatically generated rules perform only slightly better than ordinary blind translation and remain well below the performance obtained with human-authored rules. This indicates that identifying the precise failure mode may itself require linguistic, cultural, or multimodal expertise.

A further limitation is the privileged-information asymmetry between generation and verification. The verifier is told exactly what phenomenon to inspect, whereas the translation system is not. This is intentional and supports actionable diagnosis, but verifier pass rates should not be interpreted as direct substitutes for ordinary translation-quality scores.

LTB’s principal methodological contribution is the conversion of expert knowledge into reusable, atomic evaluation rules. Its main empirical finding is that strong generic translation scores can coexist with very low performance on carefully targeted, human-validated requirements. The benchmark therefore complements, rather than replaces, broad resources such as FLORES and WMT: conventional benchmarks estimate general performance, whereas LTB exposes specific residual failures and provides concrete targets for model development, post-training, prompting, and inference-time verification [2609.04173].

Source: https://www.emergentmind.com/topics/last-translation-benchmark