Clinical Trial Natural Language Inference

• Clinical Trial Natural Language Inference is the process of verifying claims from structured trial reports by classifying them as entailed or contradicted based on retrieved evidence.
• It utilizes both pipeline and joint modeling approaches, integrating domain-specific architectures such as ClinicalBERT and DeBERTa-v3 to handle long, multi-section documents.
• Recent evaluations emphasize robustness through metrics like Faithfulness and Consistency, highlighting challenges in multi-evidence, numerical, and commonsense reasoning.

Clinical Trial Natural Language Inference (NLI) is the task of determining whether a statement, structured query, or eligibility claim is supported by, contradicted by, or otherwise not warranted by evidence drawn from clinical trial text, typically with an accompanying requirement to identify the supporting evidence. In contemporary work, the topic encompasses multi-evidence textual entailment over full Clinical Trial Reports (CTRs), treatment-effect inference from randomized trial articles, safe biomedical NLI under controlled perturbations, and eligibility-style matching between patient profiles and trial criteria (Jullien et al., 2023, DeYoung et al., 2020, Jullien et al., 2024, Aguiar et al., 19 Mar 2025).

1. Task definitions and formal scope

Clinical trial NLI differs from standard sentence-pair NLI chiefly in its premises, evidence structure, and inferential targets. In NLI4CT, the canonical formulation is binary: given a clinical trial document or CTR section and a claim, the system predicts Entailment or Contradiction; a second subtask retrieves the subset of sentences or facts that justify the decision (Jullien et al., 2023). The same benchmark family distinguishes Single instances, where the statement concerns one report, from Comparison instances, where the statement compares a “primary” and a “secondary” trial or two CTR sections (Mandal et al., 2024).

The premise is usually not a short sentence but a long, structured document segment. CTRs are partitioned into four section types: Eligibility Criteria, Intervention, Results, and Adverse Events. Statements may require biomedical reasoning, commonsense reasoning, numerical reasoning, long-document reasoning, and multi-evidence or multi-hop reasoning. Comparison instances intensify the difficulty by introducing a second source of both relevant and distracting information (Jullien et al., 2023).

A broader formulation appears in Evidence Inference 2.0, where the “hypothesis” is an ICO prompt—Intervention, Comparator, and Outcome—and the model must infer whether the intervention resulted in a significantly increased outcome, significantly decreased outcome, or no significant difference, while also identifying a supporting evidence span (DeYoung et al., 2020). This is NLI-like rather than identical to canonical entailment, because the premise is a full biomedical article and the label space is directional and clinically specific.

Clinical trial NLI has also been extended to eligibility inference. In NLI4PR, the trial’s eligibility criteria serve as the premise and a patient profile serves as the statement; Eligible is mapped to Entailment and Excluded to Contradiction (Aguiar et al., 19 Mar 2025). The paper formalizes positive inference through inclusion and exclusion logic: $\forall i \in Inc, \exists f \in Pat; entail(i, f)$

$$\forall e \in Exc, \forall f \in Pat, contradict(e, f)$$

$(1) \wedge (2) \Rightarrow Entailment$

This makes explicit that clinical-trial inference often depends on satisfying an entire set of constraints, not merely locating a single confirming sentence.

2. Benchmark resources and annotation regimes

Clinical trial NLI matured through a sequence of increasingly document-level and evidence-aware datasets. Earlier clinical NLI work such as MedNLI established doctor-annotated entailment, contradiction, and neutral judgment in the clinical domain, grounded in Past Medical History sentences from MIMIC-III; later resources moved directly into trial reports, structured clinical evidence, and controlled robustness testing (Romanov et al., 2018).

Resource	Distinctive properties	Key statistics
Evidence Inference 2.0 (DeYoung et al., 2020)	Full-text trial articles, ICO prompts, rationale spans	12,616 unique prompts, 3,346 unique articles, 25% expansion
NLI4CT (Jullien et al., 2023)	Multi-evidence entailment and evidence retrieval over CTR sections	2,400 annotated instances; 1,700 train, 200 development, 500 test; 60% single-type, 40% comparison-type
NLI4CT-P (Jullien et al., 2024)	Contrast-set extension for safe biomedical NLI	Dev: 1,942 statements; Test: 5,000 statements; perturbations include paraphrase, contradiction rephrasing, numerical paraphrasing, numerical contradiction, and appending text
NLI4PR (Aguiar et al., 19 Mar 2025)	Eligibility inference from medical-language and patient-language profiles	7,007 instances; 3,939 Entailment, 3,068 Contradiction

The NLI4CT annotation pipeline was explicitly evidence-centered. Four domain experts from the Manchester Cancer Institute / DECMT created entailment statements, selected supporting evidence lines, and then rewrote entailment statements into contradictions; three volunteers later reclassified 100 random test instances, achieving 85% accuracy against gold labels and Cohen’s κ = 0.83. Total annotation effort was about 260 person-hours (Jullien et al., 2023). Evidence Inference 2.0 used a three-stage medical-doctor workflow—prompt generation, prompt annotation, and verification—and reported Krippendorff’s α = 0.854 across all stages (DeYoung et al., 2020).

The evidential density of these datasets is one of their defining features. NLI4CT is explicitly multi-evidence: on average, each instance has 7.74 relevant evidence pieces out of 21.67 facts per CTR premise. For some negation cases, annotators provided the full CTR section as evidence, reflecting the fact that absence or distributional patterns can be the relevant justification (Jullien et al., 2023). This design sharply separates the benchmark from span-localized QA or short-premise entailment.

SemEval shared tasks institutionalized these datasets as community benchmarks. SemEval-2023 Task 7 received 643 submissions from 40 participants for the entailment task and 364 submissions from 23 participants for evidence selection (Jullien et al., 2023). SemEval-2024 Task 2, centered on safe biomedical NLI, registered 106 participants, collected over 1,200 individual submissions, and produced 25 system overview papers (Jullien et al., 2024).

3. Evidence-centric modeling paradigms

A central methodological distinction in clinical trial NLI is between systems that separate evidence retrieval from inference and systems that learn them jointly. The Sebis system for SemEval-2023 Task 7 presented both formulations explicitly. In its pipeline model, each candidate sentence $s_i$ is paired with the claim $c$, encoded as

$$a_i = [s_i; SEP; c], \quad h_i = BERT(a_i), \quad P_i = softmax(MLP(h_i)),$$

and selected when

$z_i = p_i > 0.5.$

The retrieved evidence sentences $e_1,\dots,e_k$ are concatenated, the claim is treated as hypothesis and evidence as premise, and the verdict is produced by a second classifier. The ensemble submission combines pipeline and joint outputs as

$$P_{final} = 0.4 \cdot P_{pipeline} + 0.6 \cdot P_{joint}.$$

The joint model instead encodes the whole claim-document sequence with shared representations and separate heads for evidence retrieval and NLI, truncating to 512 or 1024 tokens depending on the model (Vladika et al., 2023).

This evidence-first emphasis reflects a structural fact about the task. Standard NLI models are usually trained on sentence-pair inputs, whereas CTRs are much longer; direct report-plus-claim encoding is often weak, and narrowing the premise to salient evidence improves both feasibility and explainability (Vladika et al., 2023). Yet later analysis in the shared-task overview also found that simply adding a retrieval stage does not automatically improve entailment F1, implying that retrieval and inference must be coupled carefully rather than naively cascaded (Jullien et al., 2023).

Pretrained backbone choice matters, but not uniformly. Sebis compared BERT, BioBERT, ClinicalBERT, UmlsBERT, StructBERT, ERNIE, and DeBERTa-v3, observing that domain-specific models outperform vanilla BERT, ClinicalBERT beats BioBERT, UmlsBERT benefits from injected biomedical knowledge, and DeBERTa-v3 is strongest overall, especially for NLI. These models were additionally fine-tuned on MNLI, ANLI, and MedNLI (Vladika et al., 2023).

Long-context mitigation has emerged as a separate modeling line. TLDR uses a summarize-then-infer pipeline: a T5-family or SciFive model compresses the CTR premise, and a DeBERTa-v3-base cross-encoder performs the final binary classification on the shortened premise plus statement. The method was motivated by the 512-token limit of DeBERTa and by the observation that truncation discards critical evidence, especially in comparison cases (Das et al., 2024).

Generative models introduced prompt-based variants of the same basic problem. IITK used a Retrieval-Augmented Generation (RAG)-style workflow, structured instruction templates, and multi-turn conversation, with the first turn producing an explanation and the second extracting the final label. The system compared PLMs with GPT-3.5 and Gemini Pro in zero-shot settings, used zero-shot CoT by appending “Let’s think step by step,” and used Tree-of-Thoughts with three reasoning paths (Mandal et al., 2024). SEME compared fine-tuned masked LLMs with prompted Flan-T5-large, including Chain-of-Thought and Contrastive Chain-of-Thought, and found that explicit reasoning prompts improved faithfulness but could reduce F1 by biasing the system toward contradiction (Aguiar et al., 2024).

SemEval-2024 systems also diversified fine-tuning strategies. Edinburgh Clinical NLP trained separate LoRA adapters under language modeling and triplet-loss objectives and merged them by averaging weights (Gema et al., 2024). Lisbon Computational Linguists instruction-fine-tuned a 4-bit quantized Mistral-7B-Instruct-v0.2 with LoRA on manually curated and synthetic augmentations, then extracted labels by token matching in generated outputs (Guimarães et al., 2024). DFKI-NLP combined Mistral 7B, acronym and numerical perturbation data, and a MinMax learner-auxiliary objective that upweighted hard examples (Verma et al., 2024). DKE-Research combined semantic perturbation, domain-specific vocabulary replacement using BioWordVec, and an auxiliary numerical question answering task in a multi-task DeBERTa framework (Wang et al., 2024).

4. Evaluation metrics and the shift to safe biomedical NLI

Clinical trial NLI evaluation has expanded from standard label accuracy to explicit robustness measurement. In the original NLI4CT resource, Task 1 is evaluated with Precision, Recall, and Macro F1, while Task 2 treats evidence selection as a ranking problem and evaluates with mAP@K (Jullien et al., 2023). In the SemEval-2023 shared-task implementation, both the entailment task and the evidence-selection task were evaluated with Precision, Recall, and Macro F1-score, with Task 2 treated as fact-level binary classification (Jullien et al., 2023). Evidence Inference 2.0 adds AUROC and Top-1 accuracy for evidence identification (DeYoung et al., 2020).

SemEval-2024 reoriented the field around robustness under controlled interventions. The NLI4CT-P contrast set applies perturbations only to the development and test splits, not the training split, and uses human-in-the-loop correction after semi-automated generation. The intervention categories are Paraphrasing, Contradiction rephrasing, Numerical paraphrasing, Numerical contradiction, and Appending text (Jullien et al., 2024). This converts the benchmark from a pure classification test into a probe of whether the model changes its prediction when meaning changes and preserves it when meaning is unchanged.

The two signature metrics are Faithfulness and Consistency. In the task paper, they are defined as: $$Faithfulness = \frac{1}{N}\sum_{1}^{N}\left| f(y_i)-f(x_i) \right| \qquad x_i\in C:\text{Label}(x_i) \neq \text{Label}(y_i), \text{ and } f(y_i) = \text{Label}(y_i)$$ and

$$\forall e \in Exc, \forall f \in Pat, contradict(e, f)$$0

Faithfulness measures whether a model flips when a semantic-altering perturbation changes the label; consistency measures whether it remains stable under meaning-preserving perturbations (Jullien et al., 2024).

The SemEval-2024 overview quantified the relation among these metrics. Across all systems, average Faithfulness was about 0.719 and average Consistency about 0.67. For the top 10 F1 systems, average Faithfulness rose to 0.835 and Consistency to 0.751. Overall Spearman’s correlation between Consistency and F1 was 0.8, and between Faithfulness and F1 was 0.62; among the top 10 systems, these correlations weakened to -0.12 and 0.319 respectively (Jullien et al., 2024). This suggests that once ordinary label prediction reaches a certain level, robustness behaves as a partially independent capability.

Several system papers underscore the same point qualitatively. IITK explicitly argued that high faithfulness does not automatically imply high F1, and vice versa (Mandal et al., 2024). SEME showed that a model can be accurate but unfaithful, or faithful but overly conservative (Aguiar et al., 2024). Edinburgh Clinical NLP used an average of F1, Faithfulness, and Consistency as a headline competition score, reporting GPT-4 at 0.8328 average (Gema et al., 2024). The field’s evaluation practice has therefore shifted from “Can the model classify?” to “Does it classify for the right reason, and does it behave predictably under semantic control?”

5. Empirical performance and recurrent failure modes

Original NLI4CT baselines exposed the difficulty of the task. In the resource paper, BioBERT and BioMegatron achieved F1 = 0.644 on Task 1, while a statement-only BERT baseline reached F1 = 0.626, a result explicitly treated as a red flag because it implies exploitation of surface artifacts rather than true grounding in CTR evidence. On Task 2, BM25 achieved mAP = 0.786, and using only gold evidence did not substantially solve the task, with the best evidence-only BioBERT result at F1 = 0.667 (Jullien et al., 2023). The same paper notes that one summary reports a maximum F1 of 0.627 across six SOTA NLI models, whereas the detailed table reports 0.644 for the best biomedical-pretrained models; in either case, performance remained modest.

SemEval-2023 showed that community systems could do substantially better under task-specific engineering, but also that evidence retrieval was easier than final entailment. The top system, Zhou et al. (2023) @THiFLY, achieved 0.856 F1 on entailment and 0.853 F1 on evidence selection. The shared-task overview reported that evidence selection scored on average about +0.07 F1 higher than entailment, and that many systems did not significantly outperform the majority-class baseline of 0.667 F1 on entailment (Jullien et al., 2023).

Sebis’s detailed system analysis illustrated the gains obtainable from integrated evidence modeling. On the hidden test set, the Best pipeline achieved evidence F1 79.8 and entailment F1 77.2; the Best joint reached 80.4 and 78.3; and the Best ensemble reached 81.8 and 79.8, ranking 3rd out of 40 participants (Vladika et al., 2023). The practical interpretation was that retrieval and inference are strongly coupled but make different errors, so ensembling improves robustness.

SemEval-2024 complicated the picture by revealing metric dissociations. The official overview reported best F1 = 0.80, highest Faithfulness = 0.95, and highest Consistency = 0.81, with the two best F1 systems using Mixtral-8x7B-Instruct or related Mistral-family setups (Jullien et al., 2024). Lisbon Computational Linguists reached Macro F1 = 0.80, Faithfulness = 0.83, and Consistency = 0.72, tying for 1st place on the leaderboard for macro F1 but remaining below top systems on the auxiliary metrics (Guimarães et al., 2024). Edinburgh Clinical NLP reported GPT-4 at F1 0.7751, Faithfulness 0.9479, Consistency 0.7754, placing it joint-first by the average metric (Gema et al., 2024). IITK reported Gemini Pro at F1 = 0.691, Consistency = 0.712, and Faithfulness = 0.901, with 5th place on faithfulness but only 21st on F1 (Mandal et al., 2024). A plausible implication is that leaderboard position depends strongly on which aspect of “reasoning” is foregrounded.

Long-context handling also produces measurable gains. TLDR’s best configuration, SciFive-base summarization plus DeBERTa, achieved Macro F1 = 0.658, compared with 0.474 for truncated premises and 0.612 for extractive summarization, a 0.184 gain over truncation (Das et al., 2024). Here the improvement did not make the model state of the art, but it did isolate premise compression as a major bottleneck.

The recurring failure modes are highly consistent across papers. NLI4CT and Sebis both identify commonsense reasoning, numerical reasoning, multi-hop reasoning, medical knowledge, and world knowledge as major error sources (Jullien et al., 2023, Vladika et al., 2023). Systems often over-select evidence, especially for quantifiers such as “at least one,” “exactly one,” and “none of the patients,” which hurts precision; Sebis capped selected evidence at 20 sentences to control this (Vladika et al., 2023). DFKI-NLP observed a shortcut in which higher word overlap correlated with entailment and reported a frequent error of confusing cohorts with trials in comparison examples (Verma et al., 2024). Lisbon Computational Linguists found their weakest robustness on text appended to the end, with an error rate of 40.6% (Guimarães et al., 2024). SEME documented pathological metric behavior in which ClinicalBERT predicted almost always Contradiction, yielding F1 = 0.00 but Faithfulness = 1.00 (Aguiar et al., 2024). D-NLP’s model comparison concluded that even strong LLMs still struggle markedly on numerical cases, with visible arithmetic in generated answers correlating with correctness (Altinok, 2024).

Later diagnostic work sharpened these concerns. The CTNLI benchmark decomposed trial reasoning into Causal Attribution, Compositional Grounding, Epistemic Verification, and Risk State Abstraction, then paired each item with a Ground Knowledge and Meta-Level Reasoning Verification (GKMRV) probe. Across six LLMs, mean GKMRV accuracy was 0.918, mean main-task accuracy 0.25, and mean output consistency 0.87 (Jullien et al., 14 Aug 2025). This result was interpreted as a knowledge-reasoning dissociation: models often have access to the relevant clinical facts but fail to deploy them in the structured way the inference task demands.

6. Applications, extensions, and research directions

The practical motivations of clinical trial NLI are uniform across the literature: evidence synthesis, systematic reviews, claim verification, personalized evidence-based care, clinical decision support, and biomedical question answering. NLI4CT was explicitly introduced as a step toward systems that can retrieve relevant evidence from large collections of CTRs and verify claims about eligibility, interventions, outcomes, and adverse events without requiring clinicians to inspect every report manually (Jullien et al., 2023). Sebis argued that automating inference over trial reports is feasible when evidence retrieval and NLI are treated as connected problems rather than isolated ones (Vladika et al., 2023). Evidence Inference 2.0 framed the same objective as building searchable databases of treatment effects from trial articles (DeYoung et al., 2020).

A major extension is patient-facing trial matching. NLI4PR recasts recruitment from the patient’s point of view and shows that open-source LLMs can operate on patient-language descriptions with only a moderate drop relative to medical-language descriptions. The best model, Qwen2.5-14B-Instruct, achieved 71.8 macro-F1 on patient language and 73.1 on medical language under the vanilla prompt, suggesting that patient-initiated recruitment is plausible, though still limited by precision loss in lay expressions (Aguiar et al., 19 Mar 2025).

Another extension appears in cohort selection over patient records rather than trial reports. The n2c2-based study on clinical trial cohort selection treated eligibility checking as an NLI-like comparison between patient text evidence and criterion statements, finding that open LLMs perform reasonably on straightforward criteria but struggle on temporally constrained, negated, or clinically nuanced ones such as MAKES-DECISIONS and MI-6MOS (Tai et al., 19 Jan 2025). This suggests that “clinical trial NLI” now spans at least two directions: inference over trial documents and inference between trial criteria and patient descriptions.

The research agenda emerging from these papers is notably consistent. Recommended directions include better prompt strategies and reasoning control, improved numerical and logical robustness, stronger exposure to diverse biomedical NLI data, and more explicit robustness-oriented training through perturbation or augmentation (Mandal et al., 2024, Jullien et al., 2024). DFKI-NLP’s perturbation training, DKE-Research’s numerical QA auxiliary task, and Edinburgh’s adapter merging all reflect attempts to move beyond lexical shortcuts toward more stable representations (Verma et al., 2024, Wang et al., 2024, Gema et al., 2024).

Later diagnostic work pushes further toward modular or structured reasoning. CTNLI recommends separating retrieval from reasoning, incorporating neuro-symbolic components, and adding counterfactual or probabilistic modules for interventions, conflicting evidence, and latent risk (Jullien et al., 14 Aug 2025). This suggests a broader transition in the field: from optimizing benchmark F1 on static examples to modeling whether systems can integrate constraints, weigh evidence, react appropriately to causal or numerical changes, and remain stable under paraphrase. In high-stakes biomedical settings, that transition is not merely methodological; it is the condition under which clinical trial NLI becomes a credible component of real evidence workflows.