Span NLI BERT: Span-level Inference & Interpretability

Updated 17 November 2025

Span NLI BERT is a framework that enhances BERT by incorporating span-level representations to capture multi-word interactions and improve NLI accuracy.
It employs novel pre-training methods like contiguous span masking and a span-boundary objective to model phrase-level semantics more effectively.
Applications extend from standard GLUE benchmarks to domain-specific tasks like patent claim validation, offering improved interpretability and robust performance.

Span NLI BERT refers to a class of approaches that integrate span-level representations, masking, and reasoning into BERT-family models specifically for Natural Language Inference (NLI) tasks. These methods aim to enhance BERT’s standard architecture by more directly modeling, interpreting, or predicting the relationships between contiguous spans of text, rather than only at the token or aggregated sequence level. The spectrum of Span NLI BERT encompasses advances in pre-training (e.g., SpanBERT), span-sensitive classification and explanation (e.g., SLR-NLI, SpanEx), and span-pair applications (e.g., patent claim entailment). This article reviews the principal methodologies, theoretical motivations, empirical findings, and remaining research challenges.

1. SpanBERT: Span-aware Pre-training and NLI Fine-tuning

SpanBERT (Joshi et al., 2019) augments BERT’s pre-training objectives to focus on the explicit representation and prediction of spans, replacing BERT’s random token masking with random contiguous span masking and introducing the Span Boundary Objective (SBO). This shift is motivated by the observation that many NLI phenomena—such as phrase-level entailment, contradiction, or alignment—arise at the span rather than the word level.

Key pre-training modifications:

Contiguous Span Masking: Instead of masking 15% of WordPiece tokens independently, SpanBERT repeatedly samples geometric-length (mean ≈ 3.8, max 10) word-aligned spans and masks them until 15% of all subword tokens are masked. Each span is replaced with 80% [MASK], 10% random, 10% unchanged tokens, as in BERT.
Span Boundary Objective (SBO): For each masked span $(x_s,\ldots,x_e)$ , SBO predicts the content of each internal token $x_i$ using only the boundary token representations $\mathbf{x}_{s-1}, \mathbf{x}_{e+1}$ and a learned relative position vector, composed and mapped through a small 2-layer MLP.
Loss Function: The overall loss sums the standard masked LLM (MLM) cross-entropy and the SBO cross-entropy over all masked tokens:

$\mathcal{L}(x_i) = -\log P_{\text{MLM}}(x_i|\mathbf{x}_i) - \log P_{\text{SBO}}(x_i|\mathbf{y}_i)$

Fine-tuning for NLI:

The input consists of $[\text{CLS}]$ premise $[\text{SEP}]$ hypothesis $[\text{SEP}]$ , passed through the SpanBERT encoder. The [CLS] vector is fed to a linear+softmax layer over NLI labels.
No architectural modifications are made for NLI beyond the classification head.

Empirical results on GLUE:

MNLI-matched: 88.1%
MNLI-mismatched: 87.7%
QNLI: 94.3%
RTE: 79.0%
These results consistently outperform BERT baselines.

SpanBERT’s span-focused pre-training enhances NLI by enabling the encoder to capture and align multi-word expressions more effectively, with SBO concentrating span semantics into boundary vectors that inform premise–hypothesis matching during downstream NLI fine-tuning (Joshi et al., 2019).

2. Span-Level Logics and Interpretability in NLI

The SLR-NLI model (Stacey et al., 2022) provides a transparent, logic-based NLI framework wherein predictions are decomposed into span-level decisions. The model identifies hypothesis noun-phrase-based spans, encodes them with BERT (masking all but the span’s tokens), and computes class-wise logits per span. Two independent attention modules (for NEUTRAL and CONTRADICTION) assign scores to each span, normalized and combined as sentence-level logits in training. The final classification uses logical rules:

If any span has contradiction attention > 0.5, label CONTRADICTION.
Else if any span has neutral attention > 0.5, label NEUTRAL.
Else label ENTAILMENT.

Loss terms supervise both the sentence-level output and the maximum span activation per class; optional supervision uses human rationales (e-SNLI) to align predicted and annotated spans.

Interpretability and empirical performance:

Explicit identification of spans responsible for predictions.
Zero-shot span accuracy (no span supervision): 84.75%; with e-SNLI supervision: 88.29%.
Test accuracy on SNLI: BERT baseline 90.77%, SLR-NLI 90.33% (zero-shot), SLR-NLI+e-SNLI 90.49%.
The approach yields highly robust out-of-distribution performance and strong accuracy in low-data settings.

This architecture enables direct mapping from decision label to minimal, interpretable set of hypothesis spans, offering a semantically meaningful trace of model reasoning in the NLI setting (Stacey et al., 2022).

3. Span Interactions and Human-Model Alignment

SpanEx (Choudhury et al., 2023) extends span-level analysis to span–span interactions within NLI. Using fine-tuned BERT-family models, SpanEx models input as paired premise and hypothesis sequences. Attention matrices from selected heads are used to construct bipartite graphs of cross-sentence attention. Communities—sets of premise and hypothesis tokens with strong mutual attention—are identified using the Louvain algorithm. Each community is then split into contiguous premise and hypothesis spans; span–span pairs are ranked by summed internal attention. This produces an interpretable mapping from model attention to span-level interactions.

SpanEx dataset and evaluation:

Crowdsourced annotations on SNLI and FEVER label high- and low-level semantic span–span relations (Synonym, Antonym, Hypernym-P→H, etc.).
Metrics such as AOPC-Comp (comprehensiveness), AOPC-Suff (sufficiency), and Post-hoc Accuracy (PHA) quantify faithfulness of span-based explanations.
Human-model alignment is strong for at least 64% of cases, especially for low-level spans. Models with higher test accuracy have higher human alignment.

Applying unsupervised community detection to BERT reveals that span–span interactions extracted from attention match a large subset of the span interactions identified by humans as critical for NLI, particularly in contradiction (Antonym) and entailment (Synonym/Hypernym) cases (Choudhury et al., 2023).

4. Span-Pair NLI for Coherence and Domain-Specific Tasks

Span NLI BERT architectures can be extended to custom domains such as patent claim validation. In patent claim generation (Lee et al., 2019), candidate claims are segmented into adjacent span pairs. Fine-tuned BERT models are used as span-pair NLI classifiers, receiving $[\text{CLS}]$ span $_1$ $[\text{SEP}]$ span $_2$ $[\text{SEP}]$ as input and outputting a “relevant” or “irrelevant” label.

Training data is carefully balanced with positive (intra/patent, cross-claim) and negative (no-shared-subclass, hard negatives) pairs.
Fine-tuned BERT-Base achieves ≥92.7% accuracy on balanced validation and ≥86.2% for positive pairs over multiple years’ patent datasets.
The span-relevancy ratio $R$ is defined as the fraction of relevant span pairs in generated claims and correlates negatively with the diversity of the generator (top- $k$ sampling from GPT-2).

This approach demonstrates the flexibility and transferability of span-level NLI modeling to non-standard textual domains, using the same fundamental BERT classification machinery (Lee et al., 2019).

5. Unified Span-Extraction for Question Answering and NLI

Span extraction can serve as a universal interface for question answering, text classification, and NLI (Keskar et al., 2019). In this approach, the label set (e.g., “entailment,” “contradiction,” “neutral”) is appended as contiguous spans to the input, and the model trains a span decoder (via start/end distributions) to select the correct label span.

For NLI, the combined input is $[\text{CLS}]$ premise $[\text{SEP}]$ hypothesis $[\text{SEP}]$ “entailment, contradiction, or neutral?”.
During inference, the model selects the span ( $s_k, e_k$ ) corresponding to the predicted class.
Empirical results: MNLI BERT-LARGE 86.3% (span extraction) vs. 86.2% (softmax); using MNLI as intermediate training for RTE, span extraction improves accuracy to 85.2%.
Under low-data conditions, span extraction yields pronounced gains (RTE: 82.7% vs. BERT’s 67%).

This demonstrates that span-level modeling naturally unifies several output paradigms within BERT and maintains or exceeds performance relative to specialist heads (Keskar et al., 2019).

6. Practical Implementation Considerations

Span NLI BERT approaches involve several empirically-supported design decisions:

Pre-training with span masking and SBO yields performance improvements with <10% extra computational cost at pre-training, with optimal values $p=0.2$ , $\ell_{\max}=10$ for spans, and a 15% masking budget (Joshi et al., 2019).
For span-logical NLI, hypothesis noun phrase extraction (e.g., via spaCy), span masking, and two independent attention heads suffice; performance is robust even with low training data (Stacey et al., 2022).
Span-interaction explanations require attention head selection, efficient graph construction, and scalable community detection. Empirical faithfulness and human alignment should be measured using established metrics (Choudhury et al., 2023).
Supervision on human rationale spans (where available) further improves alignment, but significant interpretability is achieved even zero-shot with no span labels (Stacey et al., 2022).

7. Research Impact and Directions

Span NLI BERT has yielded both quantitative and qualitative improvements in NLI:

Systematic performance gains on all main GLUE NLI benchmarks (MNLI, QNLI, RTE) (Joshi et al., 2019).
Models that explicitly reason at the span-level exhibit increased alignment with human judgments of entailment, contradiction, and neutrality, and deliver high transparency and faithfulness (Stacey et al., 2022, Choudhury et al., 2023).
Unified span classifiers can match or surpass traditional heads for NLI, QA, and regression, particularly in low-data or multi-task regimes (Keskar et al., 2019).
Application to domain-specific structured text (patent claims) demonstrates generalization of span NLI BERT beyond standard NLU tasks (Lee et al., 2019).

A plausible implication is that future NLI systems will increasingly integrate span-level modeling both at pre-training and logical reasoning layers, enabling improved interpretability, generalization, and domain adaptability. A continuing challenge is to extend span interaction modeling beyond attention-based extraction to more semantically robust span–span reasoning, as well as to scale such models to longer documents and multi-hop contexts.