---
title: Next Sentence Prediction (NSP) Overview
url: https://www.emergentmind.com/topics/next-sentence-prediction-nsp
type: topic
---

# Next Sentence Prediction (NSP) Overview

Next Sentence Prediction (NSP) denotes a family of objectives and tasks centered on modeling sentence-to-sentence continuation. In the canonical BERT formulation, NSP is a binary sentence-pair classification problem: given segments \(A\) and \(B\), the model predicts whether \(B\) is the actual next sentence after \(A\). Subsequent work broadened the term substantially. In the literature, NSP may refer to multiple-choice next-sentence selection, retrieval-style ranking of candidate continuations, sentence-level latent prediction in generative language models, coherence-based boundary detection, or task-specific pairwise supervision such as dialogue response matching and negation-sensitive continuation [2204.12716] [1910.13291] [2408.00655].

## 1. Core formulations and terminological scope

The expression “NSP” is overloaded. Some papers retain the original BERT meaning, while others use it for tasks that share only the central idea of predicting discourse continuation.

| Formulation | Input and prediction target | Representative source |
|---|---|---|
| BERT-style binary NSP | Sentence pair \((A,B)\); predict **IsNext** vs **NotNext** | [2109.03564] |
| Multiple-choice continuation | Context \(q\); choose the true next sentence from four candidates | [1910.13291] |
| Retrieval-style sentence LM | Context \(s_{1:t}\); rank candidate next sentences using a predicted embedding | [2005.05255] |
| Sentence-level generative NSP | Predict a sentence-level latent token \(\boldsymbol{\Omega}_{t+1}\) and decode it back to text | [2408.00655] |
| Boundary-sensitive continuity classification | Consecutive pair \((s_i,s_{i+1})\); predict **is\_next** vs **not\_next** as a proxy for segmentation | [2601.03474] |

In original BERT, NSP is one of two pre-training objectives alongside masked language modeling. The task is binary: **50\% of the time, segment \(B\) actually follows segment \(A\) in a document and in the other 50\% it does not**, with the \([CLS]\) representation used for the final decision [2204.12716]. Later work reused the same sentence-pair interface but altered the semantics of the labels. Examples include polarity-reversed negatives for negation robustness, previous-sentence labels for symmetric order modeling, and supervised synonymy labels in biomedical terminology alignment [2502.07717] [1909.03405].

This terminological spread matters because empirical conclusions about one version of NSP do not automatically transfer to another. Critiques aimed at BERT’s pre-training NSP concern random negative sampling and context splitting; they do not directly address sentence-level autoregressive generation or boundary detection by local continuity [2010.01694] [2408.00655].

## 2. BERT-era NSP: objective, criticisms, and structured variants

In vanilla BERT, NSP is a document-level sentence-pair objective intended to capture inter-sentence relations. Positive examples use actual neighboring sentences; negative examples use random sentences from a different document. The input keeps the standard BERT sentence-pair format, and the model predicts the label from the final \([CLS]\) state [2109.03564].

A major line of criticism holds that this objective is weakly aligned with semantic reasoning. “On Losses for Modern Language Models” argues that NSP is detrimental to BERT pre-training for two reasons stated explicitly in the paper: **context splitting** and a **shallow semantic signal**. In its controlled comparison, **MLM only** reaches an average GLUE dev score of **78.8**, whereas **MLM + NSP** reaches **77.5**; the appendix reports **MLM vs NSP p-value = 2.547e-03**, supporting the claim that the MLM-only baseline is better [2010.01694]. The same study reports that alternative auxiliary tasks outperform NSP, including **sentence ordering** (**79.1**), **adjacent sentence prediction** (**79.0**), **FastSent** (**80.0**), and a **Quick Thoughts** variant (**80.3**) [2010.01694].

A different response to the same problem is to enrich, rather than discard, the sentence-pair objective. “Symmetric Regularization based BERT for Pair-wise Semantic Reasoning” replaces binary NSP with a **3-class** task: **IsNext**, **IsPrev**, and **DiffDoc**. It further extends the setup to a **5-class** formulation with **IsNextInadj** and **IsPrevInadj**, plus label smoothing for noisier near-neighbor cases. Reported results show **BERTBase average 79.6**, **BERTBase-PN 80.4**, **BERTBase-PN5cls 80.7**, and **BERTBase-PNsmth 81.0** on GLUE; on **RTE**, scores rise from **66.4** to **70.6**, and on **QNLI** from **90.5** to **92.1**. The paper also reports a **160\% relative improvement** on the HANS lexical-overlap non-entailment subset and a **20\% accuracy** gain on swapped-entity examples [1909.03405].

Task replacement is another pattern. In “UBERT,” the original unsupervised NSP objective is replaced by supervised **Synonymy Prediction (SP)** over pairs of UMLS atom strings. The backbone remains BERT-like, but the pairwise labels change from next/not-next to synonym/non-synonym. Reported UVA results include **LexLM 0.9061**, **UBERT-A 0.9319**, **UBERT-B2 0.9340**, and **SapBERT + UBERT 0.9420**, with a reported **McNemar statistic: 5615042.0** and **\(p < 0.001\)** against LexLM [2204.12716]. This suggests that the utility of sentence-pair pre-training depends heavily on whether the pairwise label is aligned with the downstream relation of interest.

## 3. NSP as sentence continuation and coherence evaluation

Outside pre-training, NSP often serves as a direct evaluation of discourse continuation. In “Sentence Embeddings for Russian NLU,” NSP is formulated as a **4-way multiple-choice next-sentence selection task** built from **Lenta.ru** news. Each instance consists of a context sentence \(q\) and four candidate continuations \(\{a_1,a_2,a_3,a_4\}\); because labels are sampled uniformly, **random/constant guessing gives accuracy 0.25**. The dataset contains about **54k multiple-choice questions**. Under cosine-similarity ranking, the best result is **0.691 accuracy/F1** using **ELMo trained on news**; the best BERT result is **0.514**, and the best FastText result is **0.496** [1910.13291]. The contrast between ELMo and BERT is notable because BERT is used only as a sentence-vector provider rather than fine-tuned end-to-end.

A retrieval-oriented variant appears in “Toward Better Storylines with Sentence-Level Language Models.” There the model predicts an embedding for the next sentence in a story, then ranks a finite candidate set by dot product:
\[
\log P(s_{t+1}=i \mid s_{1:t}) = e_i^\top h - \log Z(h).
\]
Sentence representations are the mean of the **768-dimensional contextual token embeddings from the second-to-last layer of BERT**. On Story Cloze, **MLP + CSLoss** reaches **73.0** on **Test 2016**, compared with **59.9** for **GPT-2 fine-tuned**. On large candidate ranking over ROC Stories, **ResMLP** reaches **P@10 = 10.3** and **MRR = 0.087** [2005.05255]. This formulation differs from BERT NSP in that it is multi-way retrieval rather than binary classification.

NSP has also been used as a lightweight probe of cross-lingual comprehension. “Testing Cross-Lingual Text Comprehension In LLMs Using Next Sentence Prediction” constructs **10,000 NSP questions per language** for **English**, **Swahili**, and **Hausa**, each from a short story context of **3 to 10 sentences** with two candidate next sentences. Baseline accuracies are **81.61\% / 78.35\% / 76.02\%** for **GPT-4 Turbo**, **80.79\% / 77.13\% / 75.04\%** for **Gemini 1.5 Flash**, and **80.71\% / 68.71\% / 59.43\%** for **LLaMA 3 70B** across English/Swahili/Hausa respectively [2510.25187]. On a **1,000-question** Chain-of-Thought subsample, CoT improves **LLaMA 3** in low-resource settings (**+4.6** in Swahili, **+4.8** in Hausa) but often reduces accuracy for stronger models, which the paper describes as “overthinking” [2510.25187]. The broader implication is that NSP can isolate coherence-tracking ability even when lexical overlap or factual recall are not the primary bottlenecks.

## 4. Sentence-level generative NSP in language models

A more radical reinterpretation treats NSP not as pair classification but as sentence-level autoregressive generation. “SentenceVAE” introduces a **Sentence Encoder** and **Sentence Decoder** that turn an entire sentence into a single learned representation and reconstruct it. The input string \(S\) is split by punctuation into sentences \(\{s_i \mid 1 \le i \le n\}\), with \(S = s_1 s_2 \cdots s_n\). For a sentence \(s_t\), token embeddings are encoded into hidden states and fused by
\[
\boldsymbol{\Omega}_t = LayerNorm\left(\sum_{i=1}^{L} \boldsymbol{h}_i \right).
\]
This \(\boldsymbol{\Omega}_t\) becomes the sentence-level unit consumed by the language model [2408.00655].

When SentenceVAE is grafted onto a decoder-only LLM, the result is a **Sentence-level LLM (SLLM)**. The token embedding layer is replaced by a sequence of sentence embeddings,
\[
\boldsymbol{E}_{\text{sllm}} = [\boldsymbol{\Omega}_1, \boldsymbol{\Omega}_2, \ldots, \boldsymbol{\Omega}_t],
\]
and the vocabulary classifier is replaced by a **termination judgment layer** that decides whether generation should stop or pass the current sentence-level state onward. Generation is therefore **sentence-by-sentence**, while the Sentence Decoder reconstructs the actual token sequence from the predicted sentence token [2408.00655].

On the **Wanjuan dataset** and **OPT-based models**, the paper reports **204\% to 365\% throughput improvement**, perplexity reduced to **46\%–75\% of the original metric**, and **86\%–91\%** lower memory usage for equivalent context length. One concrete example is **SLLM-1.3B-H2**, which reaches **553.95 toks/s** versus **119.07 toks/s** for the OPT baseline, while memory falls from **400.01 KB/token** to **55.14 KB/token** [2408.00655]. The paper simultaneously notes several limitations: evaluation only on **English data**, only on **OPT-125M, 350M, and 1.3B**, no serving optimizations such as **PagedAttention**, **TensorRT-LLM**, or **LMDeploy**, dependence on punctuation-based sentence segmentation, and the fact that sentence reconstruction still requires token-level decoding inside the SentenceVAE module [2408.00655].

A related framework appears in “Let’s Predict Sentence by Sentence,” where pretrained LMs are adapted to reason in sentence space by predicting continuous embeddings of next reasoning steps. The paper distinguishes **semantic embeddings**, learned by reconstruction, from **contextual embeddings**, trained via next-sentence prediction “to encode anticipatory structure.” In the contextual setup, the input is \(x = (q, s_1, \dots, s_{i-1})\), the target is \(y = s_i\), and the embedding must contain the predictive cues needed to generate the next reasoning sentence. Under **Continuous** inference, contextual embeddings are reported to show competitive performance with **Chain-of-Thought** while reducing inference-time FLOPs **on average by half** across mathematics, logic, commonsense, and planning [2505.22202]. The same paper also reports that pure latent reasoning is “notably fragile,” indicating that sentence-level predictive states improve efficiency but introduce stability issues [2505.22202].

## 5. Task-specific reinterpretations and domain adaptations

NSP has repeatedly been repurposed as a task-aligned supervision signal rather than a generic pre-training objective. In dialogue evaluation, “Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information” adds NSP to a **Conditional Variational Autoencoder (CVAE)**. Positive examples are true context-reference pairs; with probability **0.5**, the reference is replaced by a response from another conversation to form a negative example. NSP is used both during training and at evaluation time, where
\[
g = \sigma(\text{Linear}(\text{Encoder}([c;x])))
\]
weights the mutual-information score:
\[
\text{Score} = g \cdot I(c,x) + I(x,r).
\]
The paper states that NSP becomes crucial in the **diverse set**, where responses are semantically far from the reference but still contextually appropriate [2305.16967].

For linear text segmentation, “SegNSP” reframes consecutive-sentence continuity as the core boundary signal. The predictor
\[
f_\theta : (s_i, s_{i+1}) \mapsto \hat{y}_i
\]
estimates whether \(s_{i+1}\) continues the current segment. Training uses a **70/30 split** of intra-segment and inter-segment pairs, adds up to **10 hard negatives** per document, and optimizes a segmentation-aware loss combining focal loss, a confidence penalty, and a boundary term. Reported performance is **B-\(F_1 = 0.79\)** on **CitiLink-Minutes** and **B-\(F_1 = 0.65\)** on **WikiSection**, with the latter outperforming **TopSeg** by **0.17 absolute points** [2601.03474]. This version of NSP is label-agnostic: it does not use explicit topic labels and treats **not\_next** as a proxy for topical discontinuity.

Prompt-based learning provides another reinterpretation. “NSP-BERT” reuses BERT’s original NSP head as a sentence-level prompt scorer rather than a token-level cloze predictor. Because the prompt is an entire second sentence, labels can be words, phrases, or longer descriptions. On FewCLUE zero-shot tasks, reported results include **EPRSTMT 86.9 vs PET 60.7**, **CSLDCP 47.6 vs PET 22.4**, and **IFLYTEK 41.6 vs PET 34.8**. For entity tasks, a **two-stage prompt** improves **DuEL2.0-L** from **61.2** to **69.7** and **DuEL2.0-T** from **31.4** to **40.0** [2109.03564]. Here NSP functions as a sentence-pair matching mechanism that avoids fixed mask positions.

Negation robustness yields a further specialization. “Making Language Models Robust Against Negation” introduces **Next Sentence Polarity Prediction (NSPP)** and a modified NSP task in which the negative example is not random but a **polarity-reversed** version of the true next sentence. Rule-based polarity reversal is manually checked on **100 samples** and reported to succeed in **96\%** of cases. Further pre-training BERT and RoBERTa on these objectives yields **1.8\%–9.3\%** improvements on **CondaQA**; the paper also finds that **NSP usually helps more than NSPP**, while joint training is not consistently better [2502.07717]. This use of NSP directly targets discourse-level effects of negation rather than generic coherence.

These adaptations support a narrower but important conclusion: sentence-pair prediction appears most effective when the label semantics are matched to the task. Generic next-vs-random classification is often weak, whereas polarity-aware continuation, dialogue coherence, topic continuity, synonymy, or sentence-level prompt matching provide more task-relevant supervision [2204.12716] [2502.07717].

## 6. Theoretical perspective, broader generalizations, and recurring limitations

A formal learning-theoretic perspective appears in “Hardness of Learning Regular Languages in the Next Symbol Prediction Setting.” Although this paper studies **Next Symbol Prediction**, not sentence prediction in natural language, it isolates a central issue in next-step supervision. The learner receives, for every prefix, both a membership bit and a continuation vector over symbols:
\[
{}_L(x,\sigma)=1 \iff \exists s\in\Sigma^* \text{ such that } x\sigma s\in L.
\]
Despite this richer supervision, the paper proves that if the class \(\mathrm{ADFA}^{N}_{p(N)}\) were efficiently PAC-learnable in the NSP setting, then it would also be efficiently PAC-learnable in the conventional classification setting; under cryptographic assumptions, this yields hardness for learning DFAs in the NSP setting [2510.18634]. The result is important because it shows that additional continuation labels do not, by themselves, remove computational hardness.

The next-step prediction idea has also generalized beyond text. “Reinforced Fast Weights with Next-Sequence Prediction” argues that fast weight models are poorly served by next-token prediction and instead trains them with a reinforcement-learning **next-sequence prediction** objective over multi-token rollouts, improving **LaCT-760M** and **DeltaNet-1.3B** on needle-in-a-haystack retrieval, long-context question answering, and LongBench [2602.16704]. “Masked Next-Scale Prediction for Self-supervised Scene Text Recognition” uses **Next-Scale Prediction** to predict higher-resolution latent features from lower-resolution context and reports **86.2\% average accuracy** on **Union14M** and **96.7\%** across six standard datasets [2605.14885]. “What Happens Next? Next Scene Prediction with a Unified Video Model” introduces **Next Scene Prediction** for video anticipation and reports **causal consistency 0.73** versus **0.23** for **LTX** on its benchmark [2512.13015]. These are not sentence-pair tasks, but they extend the same predictive principle from sentence continuity to longer symbolic, visual, and multimodal futures.

Across this literature, several limitations recur. BERT-style NSP is criticized for shallow semantics and harmful context splitting [2010.01694]. Sentence-level generative NSP can accelerate inference and reduce memory, but it depends on reliable sentence segmentation, remains partly token-decoded internally, and has only been demonstrated on modest model scales and English data [2408.00655]. Continuation benchmarks can contain construction artifacts, as in automatically generated Russian multiple-choice data or distractors sampled from the same story in cross-lingual evaluation [1910.13291] [2510.25187]. Segmentation-by-NSP captures local continuity but lacks explicit global discourse modeling [2601.03474]. A plausible implication is that “NSP” is best viewed not as a single settled method but as a design space for continuation-sensitive supervision, whose effectiveness depends on how continuation is defined, what the labels encode, and whether the objective matches the downstream structure.

In this sense, the history of NSP is not linear. The original BERT objective was widely criticized and often removed, yet sentence-level prediction has persisted in modified forms: as an order-sensitive pre-training task, a retrieval objective for coherence, a prompt-based sentence matcher, a latent reasoning state, a segmentation criterion, a negation-aware continuation label, and a broader next-step principle extending beyond text [1909.03405] [2109.03564] [2408.00655].

Source: https://www.emergentmind.com/topics/next-sentence-prediction-nsp