Masked Sentence Prediction (MSP) Explained

Updated 8 July 2026

Masked Sentence Prediction (MSP) is a technique that masks and reconstructs entire sentence units to enhance semantic coherence in document models.
It employs iterative refinement and conditional masked language modeling to predict missing sentence tokens efficiently.
MSP variants leverage sentence bottlenecks and hierarchical models to improve cross-lingual retrieval and universal sentence representation.

Masked Sentence Prediction (MSP) is a family of masked prediction formulations in which the object to be recovered is sentence-level: a missing sentence in a document, arbitrary masked subsets of a target sentence, or a masked sentence representation inside a document model. In the cited literature, MSP is not a single canonical objective. It appears as conditional masked language modeling over full target sentences in machine translation, as adjacent-sentence-conditioned masked prediction for sentence representation learning, as masked sentence-vector prediction for cross-lingual retrieval, and as a sentence-level infilling benchmark for LLMs (Ghazvininejad et al., 2019, Yang et al., 2020, Zhang et al., 2023, Wyatt et al., 11 Aug 2025).

1. Scope, definition, and terminological boundaries

A direct document-level definition treats a document as a sequence of sentences $D = \{s_0, \dots, s_m\}$ , removes one sentence $s_i$ , and asks a model $M$ to generate a replacement $s' = M(D - s_i)$ . In that form, MSP is a sentence-level cloze task whose difficulty comes from discourse coherence, causal order, and compatibility with both preceding and following context (Wyatt et al., 11 Aug 2025).

Other work uses the same label more broadly for masked prediction over a full sentence. In "Mask-Predict" (Ghazvininejad et al., 2019), MSP is instantiated as conditional masked language modeling over a target sentence, where arbitrary target positions may be masked and predicted in parallel given a source sentence and the unmasked target positions. In "Universal Sentence Representation Learning with Conditional Masked LLM" (Yang et al., 2020), the loss remains token-level, but prediction is mediated through a sentence-level bottleneck derived from an adjacent sentence. In "Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval" (Zhang et al., 2023), the masked object is an entire sentence vector inside a hierarchical document encoder.

Instantiation	Masked unit	Representative paper
Conditional masked generation	Target tokens within a full sentence	(Ghazvininejad et al., 2019)
Sentence-bottleneck denoising	Masked tokens in $s_2$ , conditioned on an embedding of $s_1$	(Yang et al., 2020)
Hierarchical document prediction	Sentence vectors	(Zhang et al., 2023)
Evaluation benchmark	One removed sentence from a document	(Wyatt et al., 11 Aug 2025)

A recurring source of confusion is acronym overload. In "Mixture of Soft Prompts for Controllable Data Generation" (Chen et al., 2023), MSP denotes Mixture of Soft Prompts, and the paper explicitly states that it is not Masked Sentence Prediction.

The translation formulation in "Mask-Predict" defines a conditional masked LLM (CMLM) that predicts a masked target subset $Y_{\text{mask}}$ given a source sentence $X$ and observed target tokens $Y_{\text{obs}}$ , with conditional independence inside the masked set: $P(Y_{\text{mask}} \mid X, Y_{\text{obs}}) = \prod_{y_i \in Y_{\text{mask}}} P(y_i \mid X, Y_{\text{obs}}).$ Training masks a random number of target tokens $s_i$ 0, replaces them with a special MASK token, and applies cross-entropy only on masked positions, together with a length prediction loss: $s_i$ 1 Architecturally, the encoder is a standard Transformer encoder, while the decoder removes the causal self-attention mask, so each target position can attend bidirectionally to the entire partially masked target sentence (Ghazvininejad et al., 2019).

Inference uses the mask-predict algorithm. At $s_i$ 2, all target positions are masked. At later iterations, the model remasks the $s_i$ 3 lowest-confidence tokens, with linear decay

$s_i$ 4

and predicts all masked positions in parallel. This yields a non-autoregressive first pass followed by iterative refinement. With a constant number of iterations, the method improves prior non-autoregressive translation models by over 4 BLEU on average, comes within about 1 BLEU point of a typical left-to-right Transformer, and with $s_i$ 5 and $s_i$ 6 reports about a $s_i$ 7 speed-up over a base Transformer with beam size 5 (Ghazvininejad et al., 2019).

"Semi-Autoregressive Training Improves Mask-Predict Decoding" addresses the train-test mismatch in that setup. Standard NART training exposes the model only to gold observed tokens, whereas mask-predict inference conditions on previous model predictions. SMART therefore generates a prediction-based intermediate sentence $s_i$ 8, masks a subset of it, and trains the model to recover the original gold sequence with loss on all positions: $s_i$ 9 That change makes MSP a correction objective, not merely a fill-in-the-blank objective. With 10 iterations, SMART reports 27.65 BLEU on WMT'14 EN–DE and 31.27 on DE–EN, compared with 27.03 and 30.53 for NART and 27.75 and 31.30 for an autoregressive Transformer; on WMT'17 EN–ZH and ZH–EN it reaches 34.06 and 23.78, close to 34.38 and 23.91 for the autoregressive baseline (Ghazvininejad et al., 2020).

3. Sentence bottlenecks and masked autoencoding for sentence representations

A second MSP lineage uses sentence-level bottlenecks to learn universal sentence representations. In "Universal Sentence Representation Learning with Conditional Masked LLM", a Transformer encoder maps an adjacent sentence $M$ 0 to a mean-pooled sentence vector, projects it into $M$ 1 views with a three-layer MLP, and concatenates those projected vectors as global tokens to a masked version of $M$ 2. The loss is standard MLM cross-entropy on masked tokens in $M$ 3, but prediction is conditioned on sentence-level information from $M$ 4. The paper swaps the sentence order 50% of the time, uses 80 masked tokens out of 256, finds that $M$ 5 gives the best overall SentEval performance, and reports SentEval averages of 86.0 for CMLM base and 86.8 for CMLM large, with STS averages of 67.22 and 67.03 respectively (Yang et al., 2020).

This setup is token-level in loss definition but sentence-level in information flow. The model is forced to compress the semantics of $M$ 6 into a small set of global vectors that materially affect masked prediction in $M$ 7. A plausible implication is that MSP need not mask an entire sentence explicitly to act as sentence-level supervision; a sentence bottleneck can induce the same pressure.

"Masked Autoencoders As The Unified Learners For Pre-Trained Sentence Representation" pushes that logic further. RetroMAE first applies light masking at the encoder, then reconstructs a heavily masked sentence with a one-layer decoder from a sentence embedding $M$ 8. The total loss is

$M$ 9

The encoder masks 15–30% of tokens, the decoder masks 50–70%, and the decoder is intentionally weak so that reconstruction pressure concentrates in the sentence embedding. In a two-stage training pipeline, the resulting SentMAE models achieve average NDCG@10 of 0.452 on BEIR for zero-shot retrieval, 0.393 MRR@10 on MS MARCO after continued pre-training, and 82.60 average on STS when combined with NLI-based contrastive learning (Liu et al., 2022). This suggests that masked sentence autoencoding can serve as a unified pre-training principle for both retrieval and sentence similarity.

4. Sequential sentence relation and document-level MSP

A third formulation masks sentence representations rather than surface tokens. The masked sentence model (MSM) consists of a sentence encoder that produces sentence vectors $s' = M(D - s_i)$ 0 and a shallow document encoder that consumes the ordered sequence of those vectors with sentence position embeddings. For each document, the model masks each sentence vector in turn, replaces it with a special $s' = M(D - s_i)$ 1 vector, and uses the document encoder output $s' = M(D - s_i)$ 2 at that position to predict the original sentence vector $s' = M(D - s_i)$ 3 (Zhang et al., 2023).

Prediction uses a hierarchical contrastive loss with two negative types: intra-document negatives from other sentences in the same document and cross-document negatives from other documents. Intra-document negatives are softened by a dynamic bias

$s' = M(D - s_i)$ 4

and the total pre-training objective is

$s' = M(D - s_i)$ 5

The document encoder is shared across all languages and is trained on CC-108 without bilingual parallel data (Zhang et al., 2023).

The central claim is that sentence order is approximately preserved across parallel documents even when word order is not, so sequential sentence relation becomes a language-agnostic signal. Empirically, MSM improves zero-shot cross-lingual retrieval over XLM-R on multiple benchmarks: on Mr. TyDi, MRR@100 rises from 37.7 to 44.7 and R@100 from 72.7 to 78.6; on XOR Retrieve, R@2k rises from 30.6 to 34.9 and R@5k from 39.3 to 44.7; on LAReQA, mAP@20 rises from 29.3 to 33.5 (Zhang et al., 2023). The ablations also show that sharing the document encoder across languages is materially better than separating it.

5. Sentence-level scoring and MSP as an evaluation benchmark

MSP can also be framed as sentence selection rather than sentence generation. "NSP-BERT" reuses BERT’s original Next Sentence Prediction head as a sentence-level compatibility score: $s' = M(D - s_i)$ 6 For a context $s' = M(D - s_i)$ 7 and candidate sentence $s' = M(D - s_i)$ 8, inference selects

$s' = M(D - s_i)$ 9

Because the score is defined over whole sentence pairs, it avoids the fixed-length masking problem of token-level prompt methods and supports variable-length candidates, including entity descriptions and sense glosses (Sun et al., 2021).

The evaluation paper "What am I missing here?: Evaluating LLMs for Masked Sentence Prediction" makes the document-level cloze formulation explicit and treats MSP as a benchmark rather than a training loss. It masks one sentence with <|mask_id|>, asks the model to output only the missing sentence, and evaluates both fidelity and cohesiveness. Fidelity is measured with BLEURT, SBERT cosine similarity, ROUGE-1, and BLEU; cohesiveness is assessed by human preference between the original sentence and the generated sentence inserted back into context (Wyatt et al., 11 Aug 2025).

Across ROCStories, Recipe1M, and Wikipedia, commercial LLMs show only modest fidelity. Claude 3.5 Sonnet has the strongest average fidelity among the tested models, with BLEURT 0.5259 on Recipe1M, 0.4181 on ROCStories, and 0.3416 on Wikipedia. Performance is best when the masked sentence is in the middle of the text and lowest when the masked sentence is final. In ROCStories and Wikipedia, annotators chose “Equal Preference” more than 60% of the time, showing that plausibility in context can remain high even when reconstruction fidelity is low. The paper’s key finding is that commercial LLMs are poor at predicting masked sentences in low-structured domains (Wyatt et al., 11 Aug 2025).

6. Limits, theory, and adjacent masked objectives

Several limitations recur across MSP variants. In CMLM-based translation, the masked set is assumed conditionally independent, length must be predicted explicitly, and model quality is heavily dependent on model distillation; the fully non-autoregressive first pass is especially vulnerable to multi-modality and repetition (Ghazvininejad et al., 2019). In document-level retrieval MSP, the method assumes coherent sentence segmentation and approximately stable sentence order. In evaluation settings, there is a systematic fidelity–plausibility trade-off: a generated sentence may fit the discourse while failing to reconstruct the original (Wyatt et al., 11 Aug 2025).

A theoretical account of masked prediction sharpens the importance of objective design. "Masked prediction tasks: a parameter identifiability view" studies HMMs and Gaussian HMMs and shows that the choice of masked prediction task determines whether underlying parameters are identifiable. For discrete HMMs, pairwise masked prediction tasks such as predicting $s_2$ 0 from $s_2$ 1 are non-identifying, even when several forward and backward pairwise tasks are combined. By contrast, predicting two masked tokens jointly, such as $s_2$ 2, is sufficient for identifiability up to hidden-state permutation; for Gaussian HMMs, even the pairwise task $s_2$ 3 suffices under the paper’s assumptions (Liu et al., 2022). This suggests that MSP is not only a modeling choice but also a structural choice about which aspects of latent sequence organization can, in principle, be recovered.

A related but distinct development is "Masked Thought", which masks certain tokens within chain-of-thought during autoregressive fine-tuning rather than masking a sentence inside bidirectional context. On Llama-2-7B with GSM8K, it reports a 5% improvement in GSM8K accuracy and a 10% improvement in GSM-IC accuracy over standard supervised fine-tuning (Chen et al., 2024). The result does not define MSP in the sentence-infilling sense, but it extends the masked-prediction idea to reasoning trajectories and reinforces a broader conclusion: masking can act as a dependency regularizer even when the masked unit is smaller than a sentence.

Taken together, these lines of work establish MSP as a heterogeneous research area centered on sentence-level reconstruction, refinement, or selection. Its concrete form ranges from parallel target-sentence regeneration to sentence-bottleneck representation learning, hierarchical document denoising, sentence-pair compatibility scoring, and evaluation of missing-sentence recovery. The shared premise is that token-level fluency alone is insufficient for document-level coherence, and that explicitly masking sentence-scale structure exposes that gap.