Last Translation Benchmark

Abstract: For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for machine translation are approaching saturation. Further, automatic translation metrics are unreliable, vulnerable to reward-hacking, and provide unactionable assessments. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking objective progress in the field and identifying pathways for improvement. We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. We also present a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation. The Last Translation Benchmark is a live dataset that accepts ongoing contributions. The latest version is LTBv1, containing accepted contributions prior to September 1st 2026, with future releases planned as new data is continuously collected.

• The paper introduces a new benchmark, Last Translation Benchmark, that uses human-authored examples and verification rules to diagnose persistent errors in machine translation.
• The benchmark includes 3,456 accepted examples across 109 languages, focusing on multimodal inputs to expose errors in sense disambiguation, cultural knowledge, and more. This selection method aims to reveal specific system failures not easily identified through generic metrics like BLEU, ChrF, COMET, and generic LLM-based evaluations, which are often opaque and unstable.
• The diagnostic discrimination provides localized error identification, with a significantly lower agreement between generic evalualation metrics and human judgements.

Motivation and central thesis

“Last Translation Benchmark” (2609.04173) addresses a structural problem in contemporary machine-translation evaluation: conventional benchmarks and metrics increasingly fail to discriminate among strong systems or identify the specific errors that remain consequential. Static test sets sampled from ordinary distributions are described as approaching saturation, while automatically generated challenge sets often produce unnatural or adversarial inputs that provide limited evidence about failures encountered in realistic translation. At the same time, BLEU-, ChrF-, COMET-, and LLM-based evaluations compress heterogeneous translation phenomena into scores whose interpretation is unclear and whose ranking stability may be poor.

The paper’s central claim is that progress requires a benchmark deliberately constructed around persistent model weaknesses rather than average-case translation quality. The Last Translation Benchmark (LTB) therefore collects human-authored examples that leading translation systems fail to translate, including text, images, audio, and video. Each example is paired with a correct human translation and one or more handcrafted verification rules specifying the concrete condition that a successful translation must satisfy. A model receives full credit for an example only if it passes every rule associated with that example.

This design makes LTB a stress test rather than an estimator of expected user-facing translation quality. Its intended objective is diagnostic discrimination: exposing failures involving sense disambiguation, cultural knowledge, discourse and pragmatics, linguistic variation, wordplay, multimodality, output constraints, and other phenomena that generic evaluation can obscure. The benchmark is explicitly live, with new contributions and tagged releases intended to prevent eventual saturation.

Benchmark construction

LTB is community-driven. Contributors select a source language and target language, submit a naturalistic input in textual or multimodal form, provide a human translation, inspect outputs from several translation systems, and formulate verification rules around the observed failure. The platform uses automatic translation and verification during submission, but acceptance also requires peer review by a contributor proficient in the relevant language pair.

The construction protocol imposes two important constraints. First, the human translation must pass all verification rules, establishing that the example is fairly translatable. Second, all but at most two of ten automatically generated translations must fail at least one rule. This criterion operationalizes difficulty relative to contemporary systems rather than relying on an a priori linguistic judgment. Reviewers are additionally instructed to verify that the failure is perceptible and significant, and that an expert human translator could reasonably produce a passing translation.

The resulting LTBv1 contains 3,456 accepted examples across 109 languages. The distribution is highly uneven: 73% of examples are non-English to English, 14% are English to non-English, and 13% are non-English to non-English. English is consequently central to the release, although the dataset includes substantial coverage of German, Telugu, Chinese, Spanish, French, Hindi, Arabic, Bengali, and numerous lower-resource languages and language varieties. Excluding English, 15.2% of examples involve minimally resourced languages and 3.8% involve languages categorized as having zero available resources under the adopted taxonomy.

Most examples are textual, but 6% are multimodal. The average textual input contains 19 words or 104 characters, and examples contain an average of 1.9 verification rules. A text-only evaluation subset, LTBv1-eval, contains 911 examples selected for difficulty, output diversity, and language-pair balance. The paper reports an average cost of approximately $0.12 per accepted example, including translation and verification, suggesting that the construction procedure is economically compatible with continuous expansion.

The benchmark’s openness is a methodological feature rather than merely a release decision. Contributors receive coauthorship after ten accepted submissions, and the dataset, code, and platform are released under CC BY 4.0 and MIT licenses, respectively. This structure broadens linguistic and cultural coverage beyond what a small annotation team could plausibly provide, while peer review supplies a quality-control layer absent from purely automated adversarial data generation.

Verification rules as an evaluation paradigm

The paper’s principal methodological contribution is the use of example-specific verification rules. A rule is a concise, binary criterion describing a targeted success condition, such as whether a translation preserves the masculine interpretation of “nurses,” renders an idiom non-literally, retains a specified ambiguity, or satisfies a character-length constraint. The official metric is the verifier pass rate: the percentage of examples for which all rules are passed.

This formulation addresses several deficiencies of scalar translation evaluation. A pass rate has an explicit upper bound and a direct interpretation: the fraction of challenge examples on which the system satisfies the benchmark’s stated requirements. The rules also provide localized diagnostic information. A system can fail because it selected the wrong sense, omitted a culturally specific implication, violated an output constraint, mistranslated an idiom, or ignored multimodal evidence, rather than receiving an opaque reduction in an aggregate quality score.

The rules also introduce an intentional asymmetry between generation and evaluation. A generic LLM judge must independently discover what matters in the source, whereas a verifier is told the specific phenomenon to inspect. This privileged information is crucial in cases where the generator and evaluator would otherwise share the same blind spot. For example, a model that translates a gendered Czech noun incorrectly may also fail to recognize the error under generic judging; a verifier explicitly instructed to check the gender interpretation can identify it without independently solving the full translation problem.

This design does not make verification equivalent to comprehensive translation assessment. The rules are selective by construction and may ignore fluency, terminology, omissions, or other defects not related to the intended challenge. The paper therefore treats verifier pass rate as a measure of targeted robustness, not as a complete quality score.

Empirical difficulty of LTB

The results show substantial separation between LTB and conventional evaluation. On LTBv1-eval, the strongest reported system under the official verifier reaches only 43.8%, while Gemini 3.1 Pro reaches 41.9% and GPT-5.6 Sol reaches 34.4%. Most other systems score below 20%, and several established systems score below 10%. Google Translate reaches only 2.9% with the Gemma 4 verifier and 1.6% with Qwen 3.7 Plus. Even the contributor-provided human translations, which are advantaged by the construction procedure, obtain approximately 99% verifier pass rates.

The contrast with generic judges and standard metrics is pronounced. GPT-5.6 Sol receives generic LLM-judge scores between 84.3% and 89.7%, despite a verifier pass rate between 28.1% and 36.1%. GPT-5.6 Luna receives judge scores as high as 88.5% while passing only 22.9% of examples under the strongest listed verifier. ChrF, COMET, and MetricX likewise produce substantially higher apparent quality than the rule-based evaluation. For Google Translate, for example, the official pass rate is below 4%, while generic judge scores reach approximately 48–67% depending on the judge.

These divergences support the paper’s claim that LTB examples contain failures that generic metrics classify as minor or overlook entirely. The examples are not merely low-quality outputs: they frequently contain a single decisive semantic or pragmatic error that invalidates the translation for the intended use. In the Czech example involving “the two new nurses,” leading systems select feminine forms corresponding to sestra despite the source explicitly specifying that the nurses are men. In the German translation of “paper,” systems use Papier rather than Artikel or Paper, converting an academic submission into a physical sheet of paper. In the Japanese-to-Spanish example involving “impression zombies,” systems render the expression literally rather than explaining its meaning as social-media accounts designed to farm impressions.

The results imply that high generic evaluation scores cannot be interpreted as evidence that models have solved the phenomena represented in LTB. Conversely, low LTB scores should not be interpreted as average translation quality. The benchmark establishes a deliberately difficult frontier, and the paper is explicit that its examples are not sampled to approximate ordinary traffic.

Effect of explicit verification information

The paper investigates whether models fail because they cannot perform the underlying translation operation or because they do not identify the relevant difficulty. Providing the verification rules to the translation model substantially improves performance. The reported pattern indicates that many models can satisfy a specified criterion once it is made explicit, even though they fail to anticipate that criterion during ordinary translation.

This distinction is important. It separates failure of problem recognition from failure of controlled translation. A rule such as “translate babka as pastry in this context” supplies the disambiguating information required to avoid a lexical error. However, rules demanding creative preservation, such as maintaining a pun, are less reliably actionable because they identify the objective without specifying how to achieve it.

Automatically generated verification rules produce only a slight improvement relative to blind translation and remain far below the performance obtained with gold rules. The explanation is theoretically coherent: rule generation requires the model to infer both the latent difficulty and the privileged information needed to state it. In the babka case, the model must first determine the intended sense and recognize that sense selection is the relevant risk. In a wordplay case, it must identify the need to preserve both meaning and phonological structure. Thus, synthetic rule generation functions partly as extended reasoning, but it does not eliminate the underlying recognition problem.

The result also qualifies a common interpretation of prompting gains. Rule-aware improvements do not show that the model possesses robust translation competence in the absence of guidance. They show that explicit diagnostic information can unlock capabilities that are otherwise inconsistently deployed.

Reliability and bias of the verifier

The authors compare rankings produced by verification rules, generic LLM judges, neural metrics, and human evaluation. Verification-based rankings have an average within-method Kendall similarity of 86.9%, compared with 71.3% for generic judges and 35.1% for neural metrics. Agreement between verifier rankings and human rankings reaches 90.5%, whereas agreement between generic judges and human evaluation is 34.9% and agreement between metrics and human evaluation is 16.2%.

The verifier approach is also more stable under data subsampling. The reported similarity between full-data and small-subsample rankings is 44.2% for verifiers, compared with 23.0% for generic judges and 16.2% for metrics. This suggests that binary, targeted decisions aggregate more efficiently than noisy continuous judgments when the benchmark is used to compare systems.

The self-bias analysis yields a related result. Generic LLM judges show substantial self-preference: measured self-bias reaches 28.2% for Gemma 4, 20.2% for Qwen 3.7 Flash, 15.5% for GPT-5.4 Mini, and 15.1% for Gemini 3.5 Flash Lite. Under verification rules, the corresponding values decline to 8.9%, 6.9%, 8.5%, and 5.9%. Gemini 3.1 Pro exhibits negative self-bias in both conditions, with values of -8.9% for verification and -8.3% for generic judging.

The implication is not that LLM verification is intrinsically unbiased. Rather, explicit criteria appear to constrain evaluator discretion and reduce the opportunity for model-specific preferences to affect rankings. The evaluator remains a potential source of error, especially when rules are ambiguous or culturally dependent, which is why the authors recommend open-weight verifiers for reproducibility and report stability across several verifier models.

Human evaluation and the status of the reference translations

The paper recognizes a circularity risk: contributors write both the reference translation and the verification rules, so the strong performance of human translations may partly reflect authorial alignment. To address this concern, the authors conduct a human reevaluation involving 22 bilingual annotators and 317 examples across 19 language pairs. Annotators use a contrastive error-span protocol, first without seeing the rules and then with them.

Human-supplied translations rank highest under this evaluation, broadly agreeing with the rule-based verifier but not with generic judges or standard metrics. Without rules, the gap between human translations and the strongest automatic systems is smaller; after rules are shown, the gap widens. This indicates that independent annotators do not always assign the same importance to the phenomenon identified by the contributor, but that the rules nonetheless expose errors that human evaluators recognize when directed to inspect them.

The evidence supports the claim that LTB references and rules are not merely arbitrary author preferences. However, the sample is small relative to the full benchmark, and the human translations remain advantaged by their role in example construction. The study therefore validates the evaluation framework only partially; it does not establish that every rule is equally reliable across languages, modalities, or annotator populations.

Taxonomy of translation difficulty

LTB introduces a source-oriented taxonomy intended to characterize why an input is difficult, rather than merely cataloguing output errors. The taxonomy is multi-label and inductive: two linguists annotate a subset, and an LLM expands the annotation to the complete release. The paper reports 5,214 linguistic labels, 2,654 extralinguistic labels, and 5,937 broad-level linguistic labels, with multiple labels possible for each example.

The largest linguistic category is sense-related difficulty, containing 1,861 labels. Polysemy accounts for 948 instances, while collocation, style preservation, false friends, lexical gradation, and domain preservation identify more specific mechanisms. Non-compositional phenomena account for 1,437 labels, dominated by metaphor with 1,086 and wordplay with 398. These examples show that strong systems continue to struggle when translation requires non-literal interpretation, contextual sense selection, preservation of pragmatic force, or simultaneous transfer of meaning and form.

The extralinguistic taxonomy contains 2,329 knowledge-related labels, including cultural artifacts, conventions, slang, named entities, and internet cultural artifacts. This category captures knowledge that is often absent from conventional parallel corpora or dictionaries. Examples include Korean honorific conventions in forms of address, internet usage of “impression zombies,” and culturally specific metaphors or idioms.

Constraint-related failures include output-language control, length limits, input formatting, and multimodal interpretation. The English-to-German example requiring fewer than 20 characters illustrates a case in which translation necessarily involves information loss and paraphrastic compression. The Chinese Sign Language example demonstrates that a textual distractor can cause models to ignore the visual or gestural signal entirely. The taxonomy also identifies blockers such as irrelevant output, refusal, incompleteness, instruction injection, and tokenization errors. These labels are analytically useful because they distinguish failures of translation competence from failures that prevent the system from engaging with the translation task.

A parallel set of broad linguistic levels—phonological, morphological, syntactic, lexical-semantic, pragmatic, and orthographic—provides a coarser analytical axis. The framework is intentionally evolving and does not claim exhaustive linguistic coverage. The authors acknowledge that fine-grained phenomena such as gender agreement, reduplication, light-verb constructions, and honorific agreement may be collapsed into broader categories.

Cross-language transfer

The appendix investigates whether difficult source phenomena can be transferred across target languages. The procedure retains the original source, asks an LLM to adapt the verification rules and generate a new target-language reference, and filters examples according to the same acceptance criteria. Across 100 examples transferred into Czech, Chinese, Farsi, Italian, and Hebrew, 48% survive the automatic filters, while manual review accepts 64.1% of the retained candidates. Rule annotations are judged acceptable in 86.4% of cases.

The transfer results reveal a principled asymmetry. Difficulties located in the source—such as idioms, garden-path structures, and cultural artifacts—often persist across target languages. By contrast, target-language-specific phenomena, including grammatical gender and morphology-to-word mappings, frequently become irrelevant or inapplicable. Only 11% of examples transfer successfully across three or more target languages, 17% across language pairs, and 31% for individual target languages on average.

The Hindi-to-Czech transfer of the idiom unees bees ka phark illustrates the favorable case: the challenge is interpreting a source-language idiom meaning that a difference is negligible, so the same failure persists when the target changes. The result supports semi-automatic expansion of LTB, but the manual-review statistics also show that transfer is not a reliable substitute for linguistic validation. A single invalid verification rule can make an otherwise plausible transferred example unusable.

Limitations and open questions

LTB’s difficulty criterion is relative to a selected set of models and therefore depends on model availability, prompting, language support, and possible data contamination. A benchmark designed to break current systems can become less informative as models improve, requiring continual maintenance and new releases. The live-dataset design addresses this operationally but complicates longitudinal comparability unless version-specific reporting is strictly maintained.

The dataset is also distributionally skewed. English-centered language pairs dominate, many languages have very few examples, and 94% of inputs are textual. The inclusion of 109 languages is substantial, but it does not imply balanced multilingual coverage. The release includes dialects, variants, and low-resource languages, yet the number of examples for many of them is too small to support robust language-level conclusions.

Verification rules improve interpretability but introduce authorial and linguistic subjectivity. They may encode one contributor’s conception of acceptable style, cultural adequacy, or pragmatic equivalence, and LLM verifiers may fail on precisely the low-resource or culturally specialized cases that the benchmark is intended to expose. The high human-reference pass rate is partly guaranteed by construction, and the independent human evaluation covers only 317 examples. The paper therefore leaves open how well rule-based judgments generalize to broader expert consensus and how rule quality should be audited at scale.

The taxonomy is likewise dependent on LLM-assisted annotation after initial linguistic analysis. Its counts should be interpreted as descriptive rather than as validated estimates of the prevalence of translation difficulties. An unresolved methodological question is how to distinguish intrinsically difficult phenomena from artifacts of model training, tokenization, prompt sensitivity, or insufficient target-language coverage.

Finally, the paper leaves open how benchmark scores should be aggregated across heterogeneous modalities, language pairs, and rule types. A system passing 50% of examples may fail systematically on one important phenomenon while succeeding on many others. The binary all-rules pass criterion is diagnostically clear, but it can conceal partial success within examples and does not by itself provide a calibrated estimate of severity or utility.

Conclusion

The Last Translation Benchmark proposes a targeted alternative to saturated translation benchmarks and opaque quality metrics. Its combination of community-authored challenge examples, peer review, explicit verification rules, multimodal inputs, and continuous release produces a benchmark on which leading systems perform markedly below their generic evaluation scores. The reported ranking stability, reduced self-bias, and stronger agreement with human judgments support verification-based evaluation as a practical diagnostic instrument.

The paper’s principal contribution is not a new translation model but a reframing of what a useful benchmark should measure. LTB evaluates whether systems satisfy explicit, linguistically and pragmatically meaningful requirements on examples selected because current models fail. Its results show that strong aggregate translation scores coexist with persistent errors in sense selection, cultural interpretation, non-compositional language, linguistic variation, multimodal grounding, and constraint following. The benchmark consequently provides a concrete, versioned goalpost for measuring progress on those failures while making clear that targeted robustness is distinct from typical translation quality.