---
title: 'CrossNLI: Temporal Semantics in NLI'
url: https://www.emergentmind.com/topics/crossnli
type: topic
---

# CrossNLI: Temporal Semantics in NLI

Searching arXiv for the target paper and closely related NLI benchmarks to ground the article in current literature.
CrossNLI most commonly denotes the cross-linguistic Natural Language Inference benchmark introduced for evaluating perfect-aspect reasoning in Chinese and Japanese in "LLMs Struggle with NLI for Perfect Aspect: A Cross-Linguistic Study in Chinese and Japanese" [2508.11927]. It targets a specific temporal-semantic difficulty: unlike English, where perfect aspect forms such as *had*, *has*, and *will have* distinguish past, present, and future perfects, Chinese and Japanese use a single aspect marker—Chinese *-le* and Japanese *-te-i-ta/ru*—regardless of tense, with reference time often recovered from temporal adverbs or discourse context. CrossNLI therefore probes whether NLI systems can compute entailment under shifts in tense and reference time rather than relying on lexical overlap alone. In the broader NLI literature, this resource sits alongside multilingual benchmarks such as XNLI [1809.05053], but it is narrower and more diagnostic, focusing on temporal semantics rather than general cross-lingual transfer.

## 1. Linguistic target and problem formulation

CrossNLI is built around the perfect aspect in Chinese and Japanese, where the absence of separate grammatical tense forms within the perfect aspect complicates entailment judgments [2508.11927]. The central problem is not merely lexical meaning, but whether two sentences impose compatible relations among event time, reference time, and speech time. In English, overt morphology often encodes these distinctions directly. In Chinese and Japanese, the same aspect marker can realize past, present, and future perfect interpretations, so inference depends more heavily on explicit temporal adverbs or subordinate-clause expressions.

The paper formalizes this with Reichenbach’s three-point temporal axis: Speech Time $(S)$, Event Time $(E)$, and Reference Time $(R)$ [2508.11927]. Perfect aspect always imposes $E < R$. The relevant tense configurations are:

- Past perfect: $E < R < S$
- Present perfect: $E < R = S$
- Future perfect: $S < E < R$

This framework makes the benchmark a test of temporal entailment rather than generic sentence-pair classification. A premise and hypothesis may share the same aspect marker while differing in whether $R$ precedes, coincides with, or follows $S$. The benchmark is therefore designed to expose failures that remain invisible in broader NLI evaluations.

## 2. Dataset construction and annotation logic

CrossNLI is a linguistically motivated, template-based NLI dataset with 1,350 pairs per language for Chinese and Japanese [2508.11927]. Its generation procedure combines six template categories, defined by the tense of the subordinate or main clause—Past, Present, Future—and the presence or absence of an explicit temporal adverb in the main clause. For each category, the authors defined one premise template and five hypothesis templates, yielding 30 distinct premise–hypothesis patterns per language.

The templates contain four principal slots:

- **[Event-Time]**: a subordinate clause with a past or future temporal expression
- **[NP]**: a noun phrase
- **[VP]**: a punctual verb phrase
- **[TIME]**: a duration adverbial

An optional temporal adverb in the main clause indicates $R$ relative to $S$ [2508.11927]. The lexical inventory consists of 45 distinct NP–VP–TIME combinations, validated for one-to-one Chinese↔Japanese correspondence. Since there are 45 lexical combinations and 30 template patterns, each language contains $45 \times 30 = 1{,}350$ pairs.

Labeling is rule-governed rather than heuristic. For each premise $P$ and hypothesis $H$, the annotation computes the $S$–$E$–$R$ ordering. If all temporal relations in $H$ are logically guaranteed by $P$, the label is **ENTAILMENT**; otherwise the label is **NON-ENTAILMENT** [2508.11927]. This yields a deliberately asymmetric distribution: 405 ENTAILMENT and 945 NON-ENTAILMENT examples per language.

Inter-annotator agreement was assessed by majority vote over 7 Chinese speakers and 3 Japanese speakers, with reported agreement of 94% for Chinese and 100% for Japanese [2508.11927]. This indicates that, despite the subtlety of the temporal contrasts, the intended inferential relations were stable under human judgment.

## 3. Temporal configurations and example phenomena

The benchmark’s examples illustrate how entailment depends on reference-time preservation rather than simple overlap in lexical content or aspect marking [2508.11927]. One Chinese entailment case maps a past-perfect premise to a present-perfect hypothesis: “When Taro got his PhD last year, Hanako had already been dead for three months” entails “Hanako has already been dead.” The premise imposes $E < R < S$, while the hypothesis imposes $E < R = S$; the hypothesis follows because the premise guarantees that the death event precedes both reference points.

A contrasting Chinese non-entailment case differs only by the addition of a duration in the hypothesis: the same premise does **not** entail “Hanako has already been dead for three months” [2508.11927]. The reason is that the three-month interval in the premise is anchored to a past reference time $R$, not necessarily to the present speech time $S$. The duration claim is therefore not preserved under the shift from past perfect to present perfect.

The Japanese examples show the same logic. A future-perfect premise—“When Taro graduates next year, Hanako will already have quit her job”—entails a hypothesis with the same $S < E < R$ ordering. By contrast, a past-perfect premise without an explicit duration does not entail a hypothesis that inserts a three-day duration, because the duration is not guaranteed [2508.11927].

These cases make clear that CrossNLI is not a test of lexical substitution or paraphrase alone. It is a controlled benchmark for whether a system can propagate only those temporal constraints licensed by the premise.

## 4. Experimental setup and empirical results

The evaluation in [2508.11927] uses zero-shot classification. The model set includes multilingual LLMs—GPT-3.5/GPT-4 (*gpt-4-0613*), Claude 3.5, Deepseek-V3, and Llama 3.1 (8B, 70B)—as well as monolingual-enhanced LLMs: Qwen 3 (8B, 32B) for Chinese and Swallow (9B, 27B) for Japanese. The prompt is minimal: “Given Premise and Hypothesis in Chinese (or Japanese), answer entailment or non-entailment. No explanations.” Accuracy is the primary metric, defined as
$$
\mathrm{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}.
$$

The reported accuracies are as follows [2508.11927]:

| Model | Chinese | Japanese |
|---|---:|---:|
| Claude 3.5 | 91.5 % | 76.7 % |
| GPT-4 | 80.6 % | 72.3 % |
| Deepseek-V3 | 77.3 % | 70.1 % |
| Llama 70B | 75.8 % | 72.3 % |
| Qwen 3 8B | 74.2 % | 68.8 % |
| Swallow 9B | 70.2 % | 43.6 % |
| Llama 8B | 37.3 % | 65.6 % |
| Qwen 3 32B | 51.4 % | 56.6 % |
| Swallow 27B | 54.9 % | 42.7 % |

The headline result is that even advanced LLMs struggle with fine-grained temporal inference in these tense-poor perfect systems [2508.11927]. Claude 3.5 is strongest overall, particularly in Chinese, but the study emphasizes that high aggregate accuracy does not imply robust temporal reasoning. Cross-tense and reference-time shifts remain a major failure mode.

## 5. Systematic error patterns

The paper identifies three recurring error patterns in model behavior [2508.11927]. The first is **tense-mismatch failure**: when the premise is past or future and the hypothesis is present, or vice versa, models often default to ENTAILMENT if the same aspect marker appears in both sentences. This indicates that many predictions are driven by surface matching rather than by computation of the underlying $S$–$E$–$R$ structure.

The second is **heuristic bias toward lexical overlap**. Models frequently rely on the presence of Chinese *-le* or Japanese *-tei-* without checking whether the reference time has shifted. This behavior parallels broader concerns in NLI research that models exploit superficial cues rather than inferential structure. A plausible implication is that CrossNLI functions as a targeted probe for artifact-like shortcuts, but here the shortcut is temporal morphology overlap rather than the better-known annotation artifacts of English NLI datasets.

The third pattern is specific to Japanese: even when the premise uses *-tei-ta* and the hypothesis uses *-tei-ru*, models often predict ENTAILMENT, suggesting that they have not internalized the semantic distinction between these forms [2508.11927]. This is significant because it shows that the benchmark tests not only cross-sentence reasoning but also form–meaning alignment within a language-specific aspectual system.

The paper’s broader conclusion is that state-of-the-art LLMs, including GPT-4 and Claude, still struggle with subtle temporal inference in languages without explicit tense morphology, and that targeted benchmarks are needed because aggregate accuracy can mask systematic failure on cross-tense cases [2508.11927].

## 6. Position within the NLI benchmark landscape

CrossNLI differs substantially from large multilingual NLI resources such as XNLI [1809.05053]. XNLI extends MultiNLI to 15 languages through translation and preserves the original three-way labels—entailment, neutral, contradiction—across 7,500 sentence pairs per language. Its primary purpose is to evaluate cross-lingual sentence understanding and transfer. CrossNLI, by contrast, is a two-label, phenomenon-specific benchmark constructed from controlled templates to isolate perfect-aspect reasoning in Chinese and Japanese [2508.11927].

The name itself is polysemous in the literature. "Baselines and test data for cross-lingual inference" used “CrossNLI” for manually translated SNLI-style multilingual test sets in Arabic, French, Spanish, and Russian [1704.05347]. "Reliable Evaluations for Natural Language Inference based on a Unified Cross-dataset Benchmark" used “CrossNLI” for a unified cross-dataset evaluation framework that measures generalization from SNLI to 14 other NLI datasets [2010.07676]. "Neuro-Symbolic Contrastive Learning for Cross-domain Inference" also used the term in a cross-domain inference setting tied to ILP-guided hard-example generation [2502.09213]. This suggests that, in current scholarship, “CrossNLI” does not denote a single universally fixed benchmark; in the sense most directly associated with [2508.11927], it refers to the Chinese–Japanese perfect-aspect dataset released at the project repository named CrossNLI.

The benchmark also aligns with a broader methodological trend toward targeted diagnostics. Work on Chinese NLI has shown that broad multilingual transfer results can obscure persistent failures on specific phenomena such as negation, quantifiers, and subsequence heuristics [2106.03983]. CrossNLI extends that diagnostic orientation to temporal semantics, with explicit cross-linguistic control over the phenomenon being tested.

## 7. Uses, extensions, and research significance

The paper presents CrossNLI as a diagnostic resource for three main purposes: evaluating and fine-tuning NLI models on temporal semantics, benchmarking multilingual models’ understanding of aspect–tense interactions, and guiding data augmentation methods that supply explicit temporal cues or auxiliary tasks [2508.11927]. The released repository contains template scripts, full Chinese and Japanese sentence files, annotation guidelines, and evaluation code.

The proposed extension path is likewise tightly scoped. The dataset can be expanded by adding new verb classes such as durative verbs and imperfective contexts, by introducing progressive or perfect-progressive contrasts, or by adapting the templates to other tense-poor languages [2508.11927]. These are not yet benchmark results, but they define the intended research trajectory.

In significance, CrossNLI operationalizes a longstanding typological observation: temporal inference can be substantially harder in languages where tense is not morphologically separated within the perfect system. Its contribution is therefore less about scale than about experimental control. It provides a benchmark in which incorrect entailment decisions can be traced to specific failures in temporal interpretation, especially failures to compute shifts in reference time. For research on multilingual reasoning, this makes CrossNLI a compact but technically precise test bed for whether LLMs encode temporal semantics rather than merely matching surface aspect markers [2508.11927].

Source: https://www.emergentmind.com/topics/crossnli