CrossNLI: Temporal Semantics in NLI

• CrossNLI is a diagnostic benchmark that assesses NLI systems' ability to infer temporal relations from perfect aspect markers in Chinese and Japanese.
• The resource comprises 1,350 template-generated sentence pairs per language, employing Reichenbach's temporal model to ensure precise annotation.
• Empirical findings indicate that even advanced LLMs like GPT-4 and Claude 3.5 struggle with fine-grained temporal reasoning, revealing critical error patterns.

Searching arXiv for the target paper and closely related NLI benchmarks to ground the article in current literature. CrossNLI most commonly denotes the cross-linguistic Natural Language Inference benchmark introduced for evaluating perfect-aspect reasoning in Chinese and Japanese in "LLMs Struggle with NLI for Perfect Aspect: A Cross-Linguistic Study in Chinese and Japanese" (Lu et al., 16 Aug 2025). It targets a specific temporal-semantic difficulty: unlike English, where perfect aspect forms such as had, has, and will have distinguish past, present, and future perfects, Chinese and Japanese use a single aspect marker—Chinese -le and Japanese -te-i-ta/ru—regardless of tense, with reference time often recovered from temporal adverbs or discourse context. CrossNLI therefore probes whether NLI systems can compute entailment under shifts in tense and reference time rather than relying on lexical overlap alone. In the broader NLI literature, this resource sits alongside multilingual benchmarks such as XNLI (Conneau et al., 2018), but it is narrower and more diagnostic, focusing on temporal semantics rather than general cross-lingual transfer.

1. Linguistic target and problem formulation

CrossNLI is built around the perfect aspect in Chinese and Japanese, where the absence of separate grammatical tense forms within the perfect aspect complicates entailment judgments (Lu et al., 16 Aug 2025). The central problem is not merely lexical meaning, but whether two sentences impose compatible relations among event time, reference time, and speech time. In English, overt morphology often encodes these distinctions directly. In Chinese and Japanese, the same aspect marker can realize past, present, and future perfect interpretations, so inference depends more heavily on explicit temporal adverbs or subordinate-clause expressions.

The paper formalizes this with Reichenbach’s three-point temporal axis: Speech Time $(S)$, Event Time $(E)$, and Reference Time $(R)$ (Lu et al., 16 Aug 2025). Perfect aspect always imposes $E < R$. The relevant tense configurations are:

• Past perfect: $E < R < S$
• Present perfect: $E < R = S$
• Future perfect: $S < E < R$

This framework makes the benchmark a test of temporal entailment rather than generic sentence-pair classification. A premise and hypothesis may share the same aspect marker while differing in whether $R$ precedes, coincides with, or follows $S$. The benchmark is therefore designed to expose failures that remain invisible in broader NLI evaluations.

2. Dataset construction and annotation logic

CrossNLI is a linguistically motivated, template-based NLI dataset with 1,350 pairs per language for Chinese and Japanese (Lu et al., 16 Aug 2025). Its generation procedure combines six template categories, defined by the tense of the subordinate or main clause—Past, Present, Future—and the presence or absence of an explicit temporal adverb in the main clause. For each category, the authors defined one premise template and five hypothesis templates, yielding 30 distinct premise–hypothesis patterns per language.

The templates contain four principal slots:

• [Event-Time]: a subordinate clause with a past or future temporal expression
• [NP]: a noun phrase
• [VP]: a punctual verb phrase
• [TIME]: a duration adverbial

An optional temporal adverb in the main clause indicates $R$ relative to $(E)$0 (Lu et al., 16 Aug 2025). The lexical inventory consists of 45 distinct NP–VP–TIME combinations, validated for one-to-one Chinese↔Japanese correspondence. Since there are 45 lexical combinations and 30 template patterns, each language contains $(E)$1 pairs.

Labeling is rule-governed rather than heuristic. For each premise $(E)$2 and hypothesis $(E)$3, the annotation computes the $(E)$4–$(E)$5–$(E)$6 ordering. If all temporal relations in $(E)$7 are logically guaranteed by $(E)$8, the label is ENTAILMENT; otherwise the label is NON-ENTAILMENT (Lu et al., 16 Aug 2025). This yields a deliberately asymmetric distribution: 405 ENTAILMENT and 945 NON-ENTAILMENT examples per language.

Inter-annotator agreement was assessed by majority vote over 7 Chinese speakers and 3 Japanese speakers, with reported agreement of 94% for Chinese and 100% for Japanese (Lu et al., 16 Aug 2025). This indicates that, despite the subtlety of the temporal contrasts, the intended inferential relations were stable under human judgment.

3. Temporal configurations and example phenomena

The benchmark’s examples illustrate how entailment depends on reference-time preservation rather than simple overlap in lexical content or aspect marking (Lu et al., 16 Aug 2025). One Chinese entailment case maps a past-perfect premise to a present-perfect hypothesis: “When Taro got his PhD last year, Hanako had already been dead for three months” entails “Hanako has already been dead.” The premise imposes $(E)$9, while the hypothesis imposes $(R)$0; the hypothesis follows because the premise guarantees that the death event precedes both reference points.

A contrasting Chinese non-entailment case differs only by the addition of a duration in the hypothesis: the same premise does not entail “Hanako has already been dead for three months” (Lu et al., 16 Aug 2025). The reason is that the three-month interval in the premise is anchored to a past reference time $(R)$1, not necessarily to the present speech time $(R)$2. The duration claim is therefore not preserved under the shift from past perfect to present perfect.

The Japanese examples show the same logic. A future-perfect premise—“When Taro graduates next year, Hanako will already have quit her job”—entails a hypothesis with the same $(R)$3 ordering. By contrast, a past-perfect premise without an explicit duration does not entail a hypothesis that inserts a three-day duration, because the duration is not guaranteed (Lu et al., 16 Aug 2025).

These cases make clear that CrossNLI is not a test of lexical substitution or paraphrase alone. It is a controlled benchmark for whether a system can propagate only those temporal constraints licensed by the premise.

4. Experimental setup and empirical results

The evaluation in (Lu et al., 16 Aug 2025) uses zero-shot classification. The model set includes multilingual LLMs—GPT-3.5/GPT-4 (gpt-4-0613), Claude 3.5, Deepseek-V3, and Llama 3.1 (8B, 70B)—as well as monolingual-enhanced LLMs: Qwen 3 (8B, 32B) for Chinese and Swallow (9B, 27B) for Japanese. The prompt is minimal: “Given Premise and Hypothesis in Chinese (or Japanese), answer entailment or non-entailment. No explanations.” Accuracy is the primary metric, defined as

$(R)$4

The reported accuracies are as follows (Lu et al., 16 Aug 2025):

Model	Chinese	Japanese
Claude 3.5	91.5 %	76.7 %
GPT-4	80.6 %	72.3 %
Deepseek-V3	77.3 %	70.1 %
Llama 70B	75.8 %	72.3 %
Qwen 3 8B	74.2 %	68.8 %
Swallow 9B	70.2 %	43.6 %
Llama 8B	37.3 %	65.6 %
Qwen 3 32B	51.4 %	56.6 %
Swallow 27B	54.9 %	42.7 %

The headline result is that even advanced LLMs struggle with fine-grained temporal inference in these tense-poor perfect systems (Lu et al., 16 Aug 2025). Claude 3.5 is strongest overall, particularly in Chinese, but the study emphasizes that high aggregate accuracy does not imply robust temporal reasoning. Cross-tense and reference-time shifts remain a major failure mode.

5. Systematic error patterns

The paper identifies three recurring error patterns in model behavior (Lu et al., 16 Aug 2025). The first is tense-mismatch failure: when the premise is past or future and the hypothesis is present, or vice versa, models often default to ENTAILMENT if the same aspect marker appears in both sentences. This indicates that many predictions are driven by surface matching rather than by computation of the underlying $(R)$5–$(R)$6–$(R)$7 structure.

The second is heuristic bias toward lexical overlap. Models frequently rely on the presence of Chinese -le or Japanese -tei- without checking whether the reference time has shifted. This behavior parallels broader concerns in NLI research that models exploit superficial cues rather than inferential structure. A plausible implication is that CrossNLI functions as a targeted probe for artifact-like shortcuts, but here the shortcut is temporal morphology overlap rather than the better-known annotation artifacts of English NLI datasets.

The third pattern is specific to Japanese: even when the premise uses -tei-ta and the hypothesis uses -tei-ru, models often predict ENTAILMENT, suggesting that they have not internalized the semantic distinction between these forms (Lu et al., 16 Aug 2025). This is significant because it shows that the benchmark tests not only cross-sentence reasoning but also form–meaning alignment within a language-specific aspectual system.

The paper’s broader conclusion is that state-of-the-art LLMs, including GPT-4 and Claude, still struggle with subtle temporal inference in languages without explicit tense morphology, and that targeted benchmarks are needed because aggregate accuracy can mask systematic failure on cross-tense cases (Lu et al., 16 Aug 2025).

6. Position within the NLI benchmark landscape

CrossNLI differs substantially from large multilingual NLI resources such as XNLI (Conneau et al., 2018). XNLI extends MultiNLI to 15 languages through translation and preserves the original three-way labels—entailment, neutral, contradiction—across 7,500 sentence pairs per language. Its primary purpose is to evaluate cross-lingual sentence understanding and transfer. CrossNLI, by contrast, is a two-label, phenomenon-specific benchmark constructed from controlled templates to isolate perfect-aspect reasoning in Chinese and Japanese (Lu et al., 16 Aug 2025).

The name itself is polysemous in the literature. "Baselines and test data for cross-lingual inference" used “CrossNLI” for manually translated SNLI-style multilingual test sets in Arabic, French, Spanish, and Russian (Agić et al., 2017). "Reliable Evaluations for Natural Language Inference based on a Unified Cross-dataset Benchmark" used “CrossNLI” for a unified cross-dataset evaluation framework that measures generalization from SNLI to 14 other NLI datasets (Zhang et al., 2020). "Neuro-Symbolic Contrastive Learning for Cross-domain Inference" also used the term in a cross-domain inference setting tied to ILP-guided hard-example generation (Liu et al., 13 Feb 2025). This suggests that, in current scholarship, “CrossNLI” does not denote a single universally fixed benchmark; in the sense most directly associated with (Lu et al., 16 Aug 2025), it refers to the Chinese–Japanese perfect-aspect dataset released at the project repository named CrossNLI.

The benchmark also aligns with a broader methodological trend toward targeted diagnostics. Work on Chinese NLI has shown that broad multilingual transfer results can obscure persistent failures on specific phenomena such as negation, quantifiers, and subsequence heuristics (Hu et al., 2021). CrossNLI extends that diagnostic orientation to temporal semantics, with explicit cross-linguistic control over the phenomenon being tested.

7. Uses, extensions, and research significance

The paper presents CrossNLI as a diagnostic resource for three main purposes: evaluating and fine-tuning NLI models on temporal semantics, benchmarking multilingual models’ understanding of aspect–tense interactions, and guiding data augmentation methods that supply explicit temporal cues or auxiliary tasks (Lu et al., 16 Aug 2025). The released repository contains template scripts, full Chinese and Japanese sentence files, annotation guidelines, and evaluation code.

The proposed extension path is likewise tightly scoped. The dataset can be expanded by adding new verb classes such as durative verbs and imperfective contexts, by introducing progressive or perfect-progressive contrasts, or by adapting the templates to other tense-poor languages (Lu et al., 16 Aug 2025). These are not yet benchmark results, but they define the intended research trajectory.

In significance, CrossNLI operationalizes a longstanding typological observation: temporal inference can be substantially harder in languages where tense is not morphologically separated within the perfect system. Its contribution is therefore less about scale than about experimental control. It provides a benchmark in which incorrect entailment decisions can be traced to specific failures in temporal interpretation, especially failures to compute shifts in reference time. For research on multilingual reasoning, this makes CrossNLI a compact but technically precise test bed for whether LLMs encode temporal semantics rather than merely matching surface aspect markers (Lu et al., 16 Aug 2025).