ImperfectiveNLI: Testing Aspectual Inference

• ImperfectiveNLI is a diagnostic NLI dataset that isolates aspectual entailment based on the imperfective paradox in English, contrasting events in atelic versus telic predicates.
• The dataset employs minimal-pair premise-hypothesis constructions with explicit progressive and simple past forms to rigorously test inference under varied interruption and ambiguity contexts.
• Evaluation across multiple LLMs reveals a strong teleological bias, where models often hallucinate event completion in ambiguous contexts, illustrating limitations in compositional reasoning.

ImperfectiveNLI is a natural language inference (NLI) diagnostic dataset constructed to probe LLMs' (LLMs) sensitivity to the imperfective paradox—a phenomenon in the formal semantics of tense and aspect whereby the English past progressive (“was V-ing”) entails event realization for activities (atelic predicates) but not for accomplishments (telic predicates). ImperfectiveNLI provides minimal-pair premise–hypothesis pairs that strictly isolate aspectual entailment, telicity, and interruption contexts, functioning as a targeted platform for assessing whether state-of-the-art LLMs perform compositionally faithful event inferences or resort to surface-level goal-completion heuristics (Ma et al., 14 Jan 2026).

1. Benchmark Construction and Formal Semantics

ImperfectiveNLI adopts canonical NLI framing, generating premise–hypothesis pairs of the form:

• Premise: “X was V-ing in Y…”
• Hypothesis: “X V-ed in Y.”

Aspectual marking is explicit: the progressive form is indicated with “_prog” (past progressive, “was V-ing”), while the perfective is “_pf” (simple past, “V-ed”). The validation of the imperfective paradox follows Vendler’s classic distinction:

• For atelic (activity) predicates $\varphi_a$:

$$\text{PROG}(\varphi_a)_{\text{prog}} \models \varphi_{a,\text{pf}}$$

(e.g., running_in_the_park_prog $\models$ ran_in_the_park_pf)

• For telic (accomplishment) predicates $\varphi_t$:

$$\text{PROG}(\varphi_t)_{\text{prog}} \not\models \varphi_{t,\text{pf}}$$

(e.g., building_a_house_prog $\not\models$ built_a_house_pf)

This distinction arises from the sub-interval property of activities and the inherent endpoint (“culminated state”) of accomplishments. ImperfectiveNLI systematically varies only aspect, telicity, and interruption context through templating and manual verification, controlling for confounds present in prior datasets.

2. Semantic Class Coverage and Data Organization

The dataset encapsulates 200 verbs, partitioned evenly by Vendlerian aspectual class:

• 100 activities (atelic): predicates with no inherent endpoint (e.g., run, swim)
• 100 accomplishments (telic): predicates with a goal or endpoint (e.g., build a house, write a letter)

To probe fine-grained teleology, telic verbs are subclassified following Levin (1993) and Beavers & Koontz-Garboden (2012):

• Change of State (44 verbs): melt, fix
• Creation (39 verbs): build, paint
• Consumption (9 verbs): eat, burn
• Motion to Goal (8 verbs): arrive, enter

Each verb appears across four inference-contexts:

Group	Telicity × Context	Example Premise	Hypothesis	Gold Label
A	Telic + Interrupted	The carpenter was building a gazebo, but a storm destroyed it.	The carpenter built a gazebo.	False
B	Atelic + Interrupted	The athletes were running on the track, but it started to hail.	The athletes ran on the track.	True
C	Telic + Ambiguous	The carpenter was building a gazebo.	The carpenter built a gazebo.	Unknown
D	Atelic + Ambiguous	The athletes were running on the track.	The athletes ran on the track.	True

The critical probe is Group C (telic, ambiguous): the progressive does not entail event realization, so the correct NLI relation is “Unknown,” but models frequently predict completion (hallucinated entailment).

3. Evaluation Protocols and Metrics

The evaluation suite comprises seven open-weight, instruction-tuned LLMs in the 7–9B parameter class: Llama-3.1-8B, Mistral-7B, Qwen-2.5-7B, deepseek-7B, gemma-2-9B, glm-4-9B, and Yi-1.5-9B. Four prompting paradigms are assessed:

• Zero-shot “Strict Logician”
• Definition-Aware Prompt (DAP)
• Chain-of-Thought (CoT)
• Counterfactual Simulation

Key metrics:

• Group-wise accuracy:

$$\mathrm{ACC}_G = \frac{\#\{\hat y_i = \text{gold}\}}{|G|} \;, \quad G \in \{A, B, C, D\}$$

• Teleological Bias Rate on C:

$$\mathrm{TBR}_C = \frac{\sum_{i \in C} \mathbf{1}(\hat y_i = \text{True})}{|C|}$$

• Aspectual Awareness Gap:

$\Delta_{AA} = \mathrm{ACC}_D - \mathrm{TBR}_C$

Ideal aspectual sensitivity implies $\mathrm{ACC}_D \approx 1$, $$\text{PROG}(\varphi_a)_{\text{prog}} \models \varphi_{a,\text{pf}}$$0, so $$\text{PROG}(\varphi_a)_{\text{prog}} \models \varphi_{a,\text{pf}}$$1.

Table 1: Zero-shot results (select models).

Model	ACC_A	ACC_B	ACC_C	ACC_D	TBR_C	Δ_AA
Llama-3.1-8B-Instruct	0.47	0.85	0.02	0.98	0.98	0.00
Mistral-7B-Instruct-v0.3	0.37	0.92	0.02	1.00	0.97	0.03
Qwen2.5-7B-Instruct	0.20	0.98	0.47	0.97	0.53	0.44

High $$\text{PROG}(\varphi_a)_{\text{prog}} \models \varphi_{a,\text{pf}}$$2 masks catastrophic failure on Group C, evidencing near-total teleological bias in major models.

4. Model Behavior and Teleological Bias

A salient feature is the “teleological bias”: models overwhelmingly predict “True” (entailment) on ambiguous accomplishment cases (Group C), with rates as high as 98–100%. This persists even when models are confronted with explicit negation in interruption contexts (Group A), where accuracy is only partially rescued ($$\text{PROG}(\varphi_a)_{\text{prog}} \models \varphi_{a,\text{pf}}$$3), indicating that the completion bias overrides negative evidence.

Example failure (Llama-3.1 zero-shot):

• Premise (C): “The carpenter was building a gazebo.”
• Hypothesis: “The carpenter built a gazebo.”
• Model prediction: True (98% of instances)
• Gold label: Unknown

Only Qwen2.5-7B demonstrates partial reluctance to predict completion for ambiguous telic progressives ($$\text{PROG}(\varphi_a)_{\text{prog}} \models \varphi_{a,\text{pf}}$$4), resulting in $$\text{PROG}(\varphi_a)_{\text{prog}} \models \varphi_{a,\text{pf}}$$5, but this is an outlier.

5. Representational vs. Decoding Analysis

Contextual embedding analyses, using BERT-base-uncased sentence representations for PROG/PF forms (e.g., “was building” vs. “built”), reveal that LLMs internally encode distinctions between process and result. Cosine similarity between PROG and PF representations is highest for Motion→Goal verbs ($$\text{PROG}(\varphi_a)_{\text{prog}} \models \varphi_{a,\text{pf}}$$6) and lower for Creation verbs ($$\text{PROG}(\varphi_a)_{\text{prog}} \models \varphi_{a,\text{pf}}$$7).

However, a strong negative correlation is observed between embedding similarity and teleological bias ($$\text{PROG}(\varphi_a)_{\text{prog}} \models \varphi_{a,\text{pf}}$$8): classes with the least representational overlap (Creation verbs) show the highest completion hallucinations. This suggests that, despite the internal aspectual encodings, inference decisions are governed by robust priors about event culmination, not logical compositionality.

6. Prompting Interventions and the Calibration Crisis

Four inference-time prompt types were tested for their influence on aspectual inference and teleological bias:

• Zero-shot (Strict Logic): Baseline, minimal instruction.
• Definition-Aware Prompt (DAP): Encodes formal semantic rules for imperfective entailment within system prompt.
• Chain-of-Thought (CoT): Staged “think step by step” reasoning.
• Counterfactual Simulation: Requires explicit enumeration of interruption scenarios.

For Llama-3.1, prompt aggressiveness lowers $$\text{PROG}(\varphi_a)_{\text{prog}} \models \varphi_{a,\text{pf}}$$9 but at the expense of increased rejection of true activity entailments ($\models$0 drop). Extreme prompts (e.g., Counterfactual) eliminate teleological bias ($\models$1) but result in the model rejecting all progressive entailments ($\models$2).

Table 2: Llama-3.1 performance with different prompts.

Prompt	ACC_A	ACC_B	ACC_C	ACC_D	TBR_C	Δ_AA
Zero-shot	0.47	0.85	0.02	0.98	0.98	0.00
Definition-Aware	0.60	0.95	0.36	0.99	0.45	0.54
Chain-of-Thought	0.15	0.39	0.67	0.65	0.33	0.32
Counterfactual	0.35	0.00	0.97	0.00	0.00	0.00

Pearson correlations of r > 0.9 confirm that improved calibration for telic ambiguity comes at direct cost of undermining activity entailments, producing a “calibration crisis”.

7. Significance and Limitations

ImperfectiveNLI provides systematic empirical evidence for a core shortcoming in LLMs’ event reasoning: although internal representations reflect process/result contrasts, output inference is dominated by teleological completion priors, leading to systematic hallucination of event culmination in ambiguous progressive contexts. Prompt-based interventions can alleviate but not eliminate this bias, always trading off true positive rate for atelic entailment against false positive rate for ambiguous telic completion. The results indicate that current LLMs function more as predictive narrative engines than as reliable compositional reasoners with respect to aspectual entailment (Ma et al., 14 Jan 2026).