Next Sentence Prediction (NSP) Overview

• Next Sentence Prediction (NSP) is a technique that evaluates whether one sentence follows another, using methods like binary classification, multiple-choice selection, and retrieval-based ranking.
• Modern adaptations extend NSP from BERT’s binary approach to task-specific variations, including polarity reversal and sentence-level generative modeling to better capture discourse coherence.
• Empirical findings reveal that while NSP can improve contextual understanding, issues such as shallow semantic signals and context splitting continue to challenge its effectiveness.

Next Sentence Prediction (NSP) denotes a family of objectives and tasks centered on modeling sentence-to-sentence continuation. In the canonical BERT formulation, NSP is a binary sentence-pair classification problem: given segments $A$ and $B$, the model predicts whether $B$ is the actual next sentence after $A$. Subsequent work broadened the term substantially. In the literature, NSP may refer to multiple-choice next-sentence selection, retrieval-style ranking of candidate continuations, sentence-level latent prediction in generative LLMs, coherence-based boundary detection, or task-specific pairwise supervision such as dialogue response matching and negation-sensitive continuation (Wijesiriwardene et al., 2022, Popov et al., 2019, An et al., 2024).

1. Core formulations and terminological scope

The expression “NSP” is overloaded. Some papers retain the original BERT meaning, while others use it for tasks that share only the central idea of predicting discourse continuation.

Formulation	Input and prediction target	Representative source
BERT-style binary NSP	Sentence pair $(A,B)$; predict IsNext vs NotNext	(Sun et al., 2021)
Multiple-choice continuation	Context $q$; choose the true next sentence from four candidates	(Popov et al., 2019)
Retrieval-style sentence LM	Context $s_{1:t}$; rank candidate next sentences using a predicted embedding	(Ippolito et al., 2020)
Sentence-level generative NSP	Predict a sentence-level latent token $\boldsymbol{\Omega}_{t+1}$ and decode it back to text	(An et al., 2024)
Boundary-sensitive continuity classification	Consecutive pair $(s_i,s_{i+1})$; predict is_next vs not_next as a proxy for segmentation	(Isidro et al., 7 Jan 2026)

In original BERT, NSP is one of two pre-training objectives alongside masked language modeling. The task is binary: 50\% of the time, segment $B$ actually follows segment $B$0 in a document and in the other 50\% it does not, with the $B$1 representation used for the final decision (Wijesiriwardene et al., 2022). Later work reused the same sentence-pair interface but altered the semantics of the labels. Examples include polarity-reversed negatives for negation robustness, previous-sentence labels for symmetric order modeling, and supervised synonymy labels in biomedical terminology alignment (Rezaei et al., 11 Feb 2025, Xu et al., 2019).

This terminological spread matters because empirical conclusions about one version of NSP do not automatically transfer to another. Critiques aimed at BERT’s pre-training NSP concern random negative sampling and context splitting; they do not directly address sentence-level autoregressive generation or boundary detection by local continuity (Aroca-Ouellette et al., 2020, An et al., 2024).

2. BERT-era NSP: objective, criticisms, and structured variants

In vanilla BERT, NSP is a document-level sentence-pair objective intended to capture inter-sentence relations. Positive examples use actual neighboring sentences; negative examples use random sentences from a different document. The input keeps the standard BERT sentence-pair format, and the model predicts the label from the final $B$2 state (Sun et al., 2021).

A major line of criticism holds that this objective is weakly aligned with semantic reasoning. “On Losses for Modern LLMs” argues that NSP is detrimental to BERT pre-training for two reasons stated explicitly in the paper: context splitting and a shallow semantic signal. In its controlled comparison, MLM only reaches an average GLUE dev score of 78.8, whereas MLM + NSP reaches 77.5; the appendix reports MLM vs NSP p-value = 2.547e-03, supporting the claim that the MLM-only baseline is better (Aroca-Ouellette et al., 2020). The same study reports that alternative auxiliary tasks outperform NSP, including sentence ordering (79.1), adjacent sentence prediction (79.0), FastSent (80.0), and a Quick Thoughts variant (80.3) (Aroca-Ouellette et al., 2020).

A different response to the same problem is to enrich, rather than discard, the sentence-pair objective. “Symmetric Regularization based BERT for Pair-wise Semantic Reasoning” replaces binary NSP with a 3-class task: IsNext, IsPrev, and DiffDoc. It further extends the setup to a 5-class formulation with IsNextInadj and IsPrevInadj, plus label smoothing for noisier near-neighbor cases. Reported results show BERTBase average 79.6, BERTBase-PN 80.4, BERTBase-PN5cls 80.7, and BERTBase-PNsmth 81.0 on GLUE; on RTE, scores rise from 66.4 to 70.6, and on QNLI from 90.5 to 92.1. The paper also reports a 160\% relative improvement on the HANS lexical-overlap non-entailment subset and a 20\% accuracy gain on swapped-entity examples (Xu et al., 2019).

Task replacement is another pattern. In “UBERT,” the original unsupervised NSP objective is replaced by supervised Synonymy Prediction (SP) over pairs of UMLS atom strings. The backbone remains BERT-like, but the pairwise labels change from next/not-next to synonym/non-synonym. Reported UVA results include LexLM 0.9061, UBERT-A 0.9319, UBERT-B2 0.9340, and SapBERT + UBERT 0.9420, with a reported McNemar statistic: 5615042.0 and $B$3 against LexLM (Wijesiriwardene et al., 2022). This suggests that the utility of sentence-pair pre-training depends heavily on whether the pairwise label is aligned with the downstream relation of interest.

3. NSP as sentence continuation and coherence evaluation

Outside pre-training, NSP often serves as a direct evaluation of discourse continuation. In “Sentence Embeddings for Russian NLU,” NSP is formulated as a 4-way multiple-choice next-sentence selection task built from Lenta.ru news. Each instance consists of a context sentence $B$4 and four candidate continuations $B$5; because labels are sampled uniformly, random/constant guessing gives accuracy 0.25. The dataset contains about 54k multiple-choice questions. Under cosine-similarity ranking, the best result is 0.691 accuracy/F1 using ELMo trained on news; the best BERT result is 0.514, and the best FastText result is 0.496 (Popov et al., 2019). The contrast between ELMo and BERT is notable because BERT is used only as a sentence-vector provider rather than fine-tuned end-to-end.

A retrieval-oriented variant appears in “Toward Better Storylines with Sentence-Level LLMs.” There the model predicts an embedding for the next sentence in a story, then ranks a finite candidate set by dot product: $B$6 Sentence representations are the mean of the 768-dimensional contextual token embeddings from the second-to-last layer of BERT. On Story Cloze, MLP + CSLoss reaches 73.0 on Test 2016, compared with 59.9 for GPT-2 fine-tuned. On large candidate ranking over ROC Stories, ResMLP reaches P@10 = 10.3 and MRR = 0.087 (Ippolito et al., 2020). This formulation differs from BERT NSP in that it is multi-way retrieval rather than binary classification.

NSP has also been used as a lightweight probe of cross-lingual comprehension. “Testing Cross-Lingual Text Comprehension In LLMs Using Next Sentence Prediction” constructs 10,000 NSP questions per language for English, Swahili, and Hausa, each from a short story context of 3 to 10 sentences with two candidate next sentences. Baseline accuracies are 81.61\% / 78.35\% / 76.02\% for GPT-4 Turbo, 80.79\% / 77.13\% / 75.04\% for Gemini 1.5 Flash, and 80.71\% / 68.71\% / 59.43\% for LLaMA 3 70B across English/Swahili/Hausa respectively (Chavan et al., 29 Oct 2025). On a 1,000-question Chain-of-Thought subsample, CoT improves LLaMA 3 in low-resource settings (+4.6 in Swahili, +4.8 in Hausa) but often reduces accuracy for stronger models, which the paper describes as “overthinking” (Chavan et al., 29 Oct 2025). The broader implication is that NSP can isolate coherence-tracking ability even when lexical overlap or factual recall are not the primary bottlenecks.

4. Sentence-level generative NSP in LLMs

A more radical reinterpretation treats NSP not as pair classification but as sentence-level autoregressive generation. “SentenceVAE” introduces a Sentence Encoder and Sentence Decoder that turn an entire sentence into a single learned representation and reconstruct it. The input string $B$7 is split by punctuation into sentences $B$8, with $B$9. For a sentence $B$0, token embeddings are encoded into hidden states and fused by

$B$1

This $B$2 becomes the sentence-level unit consumed by the LLM (An et al., 2024).

When SentenceVAE is grafted onto a decoder-only LLM, the result is a Sentence-level LLM (SLLM). The token embedding layer is replaced by a sequence of sentence embeddings,

$B$3

and the vocabulary classifier is replaced by a termination judgment layer that decides whether generation should stop or pass the current sentence-level state onward. Generation is therefore sentence-by-sentence, while the Sentence Decoder reconstructs the actual token sequence from the predicted sentence token (An et al., 2024).

On the Wanjuan dataset and OPT-based models, the paper reports 204\% to 365\% throughput improvement, perplexity reduced to 46\%–75\% of the original metric, and 86\%–91\% lower memory usage for equivalent context length. One concrete example is SLLM-1.3B-H2, which reaches 553.95 toks/s versus 119.07 toks/s for the OPT baseline, while memory falls from 400.01 KB/token to 55.14 KB/token (An et al., 2024). The paper simultaneously notes several limitations: evaluation only on English data, only on OPT-125M, 350M, and 1.3B, no serving optimizations such as PagedAttention, TensorRT-LLM, or LMDeploy, dependence on punctuation-based sentence segmentation, and the fact that sentence reconstruction still requires token-level decoding inside the SentenceVAE module (An et al., 2024).

A related framework appears in “Let’s Predict Sentence by Sentence,” where pretrained LMs are adapted to reason in sentence space by predicting continuous embeddings of next reasoning steps. The paper distinguishes semantic embeddings, learned by reconstruction, from contextual embeddings, trained via next-sentence prediction “to encode anticipatory structure.” In the contextual setup, the input is $B$4, the target is $B$5, and the embedding must contain the predictive cues needed to generate the next reasoning sentence. Under Continuous inference, contextual embeddings are reported to show competitive performance with Chain-of-Thought while reducing inference-time FLOPs on average by half across mathematics, logic, commonsense, and planning (Hwang et al., 28 May 2025). The same paper also reports that pure latent reasoning is “notably fragile,” indicating that sentence-level predictive states improve efficiency but introduce stability issues (Hwang et al., 28 May 2025).

5. Task-specific reinterpretations and domain adaptations

NSP has repeatedly been repurposed as a task-aligned supervision signal rather than a generic pre-training objective. In dialogue evaluation, “Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information” adds NSP to a Conditional Variational Autoencoder (CVAE). Positive examples are true context-reference pairs; with probability 0.5, the reference is replaced by a response from another conversation to form a negative example. NSP is used both during training and at evaluation time, where

$B$6

weights the mutual-information score: $B$7 The paper states that NSP becomes crucial in the diverse set, where responses are semantically far from the reference but still contextually appropriate (Zhao et al., 2023).

For linear text segmentation, “SegNSP” reframes consecutive-sentence continuity as the core boundary signal. The predictor

$B$8

estimates whether $B$9 continues the current segment. Training uses a 70/30 split of intra-segment and inter-segment pairs, adds up to 10 hard negatives per document, and optimizes a segmentation-aware loss combining focal loss, a confidence penalty, and a boundary term. Reported performance is B-$A$0 on CitiLink-Minutes and B-$A$1 on WikiSection, with the latter outperforming TopSeg by 0.17 absolute points (Isidro et al., 7 Jan 2026). This version of NSP is label-agnostic: it does not use explicit topic labels and treats not_next as a proxy for topical discontinuity.

Prompt-based learning provides another reinterpretation. “NSP-BERT” reuses BERT’s original NSP head as a sentence-level prompt scorer rather than a token-level cloze predictor. Because the prompt is an entire second sentence, labels can be words, phrases, or longer descriptions. On FewCLUE zero-shot tasks, reported results include EPRSTMT 86.9 vs PET 60.7, CSLDCP 47.6 vs PET 22.4, and IFLYTEK 41.6 vs PET 34.8. For entity tasks, a two-stage prompt improves DuEL2.0-L from 61.2 to 69.7 and DuEL2.0-T from 31.4 to 40.0 (Sun et al., 2021). Here NSP functions as a sentence-pair matching mechanism that avoids fixed mask positions.

Negation robustness yields a further specialization. “Making LLMs Robust Against Negation” introduces Next Sentence Polarity Prediction (NSPP) and a modified NSP task in which the negative example is not random but a polarity-reversed version of the true next sentence. Rule-based polarity reversal is manually checked on 100 samples and reported to succeed in 96\% of cases. Further pre-training BERT and RoBERTa on these objectives yields 1.8\%–9.3\% improvements on CondaQA; the paper also finds that NSP usually helps more than NSPP, while joint training is not consistently better (Rezaei et al., 11 Feb 2025). This use of NSP directly targets discourse-level effects of negation rather than generic coherence.

These adaptations support a narrower but important conclusion: sentence-pair prediction appears most effective when the label semantics are matched to the task. Generic next-vs-random classification is often weak, whereas polarity-aware continuation, dialogue coherence, topic continuity, synonymy, or sentence-level prompt matching provide more task-relevant supervision (Wijesiriwardene et al., 2022, Rezaei et al., 11 Feb 2025).

6. Theoretical perspective, broader generalizations, and recurring limitations

A formal learning-theoretic perspective appears in “Hardness of Learning Regular Languages in the Next Symbol Prediction Setting.” Although this paper studies Next Symbol Prediction, not sentence prediction in natural language, it isolates a central issue in next-step supervision. The learner receives, for every prefix, both a membership bit and a continuation vector over symbols: $A$2 Despite this richer supervision, the paper proves that if the class $A$3 were efficiently PAC-learnable in the NSP setting, then it would also be efficiently PAC-learnable in the conventional classification setting; under cryptographic assumptions, this yields hardness for learning DFAs in the NSP setting (Bhattamishra et al., 21 Oct 2025). The result is important because it shows that additional continuation labels do not, by themselves, remove computational hardness.

The next-step prediction idea has also generalized beyond text. “Reinforced Fast Weights with Next-Sequence Prediction” argues that fast weight models are poorly served by next-token prediction and instead trains them with a reinforcement-learning next-sequence prediction objective over multi-token rollouts, improving LaCT-760M and DeltaNet-1.3B on needle-in-a-haystack retrieval, long-context question answering, and LongBench (Hwang et al., 18 Feb 2026). “Masked Next-Scale Prediction for Self-supervised Scene Text Recognition” uses Next-Scale Prediction to predict higher-resolution latent features from lower-resolution context and reports 86.2\% average accuracy on Union14M and 96.7\% across six standard datasets (Chen et al., 14 May 2026). “What Happens Next? Next Scene Prediction with a Unified Video Model” introduces Next Scene Prediction for video anticipation and reports causal consistency 0.73 versus 0.23 for LTX on its benchmark (Li et al., 15 Dec 2025). These are not sentence-pair tasks, but they extend the same predictive principle from sentence continuity to longer symbolic, visual, and multimodal futures.

Across this literature, several limitations recur. BERT-style NSP is criticized for shallow semantics and harmful context splitting (Aroca-Ouellette et al., 2020). Sentence-level generative NSP can accelerate inference and reduce memory, but it depends on reliable sentence segmentation, remains partly token-decoded internally, and has only been demonstrated on modest model scales and English data (An et al., 2024). Continuation benchmarks can contain construction artifacts, as in automatically generated Russian multiple-choice data or distractors sampled from the same story in cross-lingual evaluation (Popov et al., 2019, Chavan et al., 29 Oct 2025). Segmentation-by-NSP captures local continuity but lacks explicit global discourse modeling (Isidro et al., 7 Jan 2026). A plausible implication is that “NSP” is best viewed not as a single settled method but as a design space for continuation-sensitive supervision, whose effectiveness depends on how continuation is defined, what the labels encode, and whether the objective matches the downstream structure.

In this sense, the history of NSP is not linear. The original BERT objective was widely criticized and often removed, yet sentence-level prediction has persisted in modified forms: as an order-sensitive pre-training task, a retrieval objective for coherence, a prompt-based sentence matcher, a latent reasoning state, a segmentation criterion, a negation-aware continuation label, and a broader next-step principle extending beyond text (Xu et al., 2019, Sun et al., 2021, An et al., 2024).