Semantic-Consistent Token Dropping (ScTD)

• Semantic-Consistent Token Dropping (ScTD) is a framework that reduces tokens while explicitly preserving semantic content during model training.
• It employs mechanisms like self-distillation, semantic scoring, and full-sequence recovery to maintain meaningful representations in reduced computations.
• Empirical results in BERT pretraining and multimodal tasks show significant training time savings and improved performance compared to vanilla token dropping.

Searching arXiv for papers on Semantic-Consistent Token Dropping and closely related token-dropping methods. Semantic-Consistent Token Dropping (ScTD) denotes token-reduction schemes that are designed to preserve semantic information when tokens are skipped, truncated, or masked during training or inference. The term is stated most explicitly in the BERT pretraining work "Revisiting Token Dropping Strategy in Efficient BERT Pretraining," which introduces a semantic-consistent learning method to counter the semantic loss induced by vanilla token dropping (Zhong et al., 2023). In later work, the same idea is used more broadly to describe methods that retain semantic fidelity under token sparsification in latent image generation, semantic segmentation, reference-conditioned diffusion, long-context language modeling, speculative decoding, supervised fine-tuning, and Vision-Language-Action inference (Li et al., 26 Mar 2026).

1. Origin and conceptual scope

The immediate motivation for ScTD is the observation that vanilla token dropping improves efficiency but can damage meaning-bearing representations. In efficient BERT pretraining, token dropping skips a subset of tokens in several middle layers and restores full sequence length before the last layer, reducing training cost but introducing a semantic loss problem that is especially harmful on semantic-intensive downstream tasks (Zhong et al., 2023). The original ScTD proposal therefore treats semantic preservation not as an incidental side effect of pruning, but as an explicit training target.

A broader interpretation of ScTD emerges in later papers. In these works, token dropping is considered semantic-consistent when removing tokens preserves the semantic content needed for reconstruction, generation, dense prediction, downstream supervision, or action execution. This broader usage includes latent prefix truncation in image tokenizers, sparse reference conditioning in diffusion transformers, token early exit in segmentation transformers, semantic compression of long text, and acceptance or rejection of speculative draft tokens according to semantic consistency criteria (Li et al., 26 Mar 2026, Parihar et al., 22 Jun 2026, Tang et al., 2023, Liu et al., 2023, Liu et al., 21 Aug 2025, Lee et al., 11 Feb 2026).

Two features recur across these formulations. First, the retained tokens are intended to be semantically privileged rather than merely high-frequency or high-attention tokens. Second, the system is usually structured so that dropping tokens reduces detail or computational depth before it destroys category identity, instruction relevance, dense spatial coverage, or action-critical information. This suggests a unifying interpretation of ScTD as semantics-first token reduction, although individual papers instantiate that principle with different mechanisms and under different objectives.

2. The original ScTD formulation in efficient BERT pretraining

In the original BERT setting, token dropping operates during pretraining rather than inference. Let $X_i \in \mathbb{R}^{s_i \times d}$ denote the activations after layer $L_i$, where $d$ is the hidden size and $s_i$ is the sequence length. Baseline pretraining processes all tokens at all layers and optimizes the masked language modeling objective

$$\mathcal{L}_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid X_l)\right],$$

where $Y$ are the masked-token labels. In token dropping, the first few layers still process the full sequence, then tokens are split into important and dropped groups in middle layers, only the important group is propagated, and all tokens are merged before the last layer; the corresponding MLM objective is

$$\mathcal{L}^*_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid \tilde{X}_l)\right].$$

The semantic problem is that this shortcut perturbs sentence structure and the interactions available to later layers (Zhong et al., 2023).

The semantic loss diagnosis in that work is unusually explicit. Using a frozen Sentence-BERT encoder, the authors track cosine similarity between original sentences and corrupted versions used during pretraining, treating it as a proxy for semantic preservation; token dropping yields lower similarity than full computation, and the gap increases during training. Probing with Tense and Subject Number tasks shows that semantic scores drop sharply in the dropped layers, reaching deficits of up to $25.2$ points at layer 11 relative to baseline. On semantic-intensive downstream tasks—OntoNotes 5.0 NER, CoNLL-2003 NER, MRPC, and SICK-R—the baseline average is $60.13$ versus $58.22$ for token dropping, a decline of $L_i$0 points (Zhong et al., 2023).

ScTD remedies this by introducing self-distillation between a full-sequence baseline path and a token-dropping path of the same model. The baseline path acts as teacher and the token-dropping path as student. Two KL-based semantic consistency terms are added: $L_i$1 for the final layer, and

$L_i$2

for the penultimate layer. In semantic-align iterations, the combined objective is

$L_i$3

with default $L_i$4. To control overhead, ScTD uses a hybrid schedule with interval $L_i$5: most iterations use only $L_i$6, and every $L_i$7-th iteration activates the dual-path semantic alignment; $L_i$8 is reported as the best trade-off (Zhong et al., 2023).

Empirically, this formulation recovers and often exceeds the lost performance. For BERT-large at 250k steps, baseline GLUE average is $L_i$9, token drop is $d$0, and ScTD reaches $d$1; for BERT-base at 250k steps, the corresponding values are $d$2, $d$3, and $d$4. The paper reports up to $d$5 pretraining time savings and up to $d$6 average improvement over vanilla token dropping (Zhong et al., 2023).

3. Core mechanisms for preserving semantics under token reduction

ScTD methods differ in implementation, but several recurrent mechanisms can be identified. One is semantic scoring: tokens are retained because they are semantically informative, semantically difficult, semantically indispensable for reconstruction, or semantically tied to the task objective. An earlier BERT token-dropping precursor uses cumulative MLM loss $d$7 as a token-type importance score and keeps the top $d$8 positions in reduced layers; this preserves performance up to about $d$9 dropping in half the layers and maintains a full-length interface by reinserting skipped tokens before the last layer (Hou et al., 2022). The original ScTD work can be read as a correction to the fact that such importance-based dropping alone is prone to semantic drift (Zhong et al., 2023).

A second mechanism is full-sequence recovery or dense-output preservation. In semantic segmentation, direct removal of tokens is not viable because every patch requires a prediction. Dynamic Token Pruning finalizes easy tokens early but keeps $s_i$0 highest confidence tokens for each semantic category during each pruning process, ensuring that every category present in the image still has representative context in deeper layers (Tang et al., 2023). Dynamic Token-Pass Transformers similarly stop easy tokens from further self-attention, reconstruct them back to their original positions, and pass a complete feature map to the decoder; they also merge stopped-token information into the class token (Liu et al., 2023). In both cases, semantics are preserved not by forbidding early exit, but by constraining how early-exited tokens remain available to the prediction head.

A third mechanism is structural privileging of semantics over detail. In "Semantic-Aware Prefix Learning for Token-Efficient Image Generation," the SMAP tokenizer injects semantic embeddings $s_i$1 as prefix tokens in both encoder and decoder and samples a prefix length $s_i$2, so that the decoder reconstructs from $s_i$3 even when $s_i$4. This forces semantics to be indispensable and induces an information-ordered latent sequence in which early tokens and the semantic prefix carry global structure while later tokens refine instance-level detail (Li et al., 26 Mar 2026).

A fourth mechanism is dual-objective consistency rather than semantic-only consistency. VLA-Pruner argues that Vision-Language-Action pruning based only on semantic salience metrics discards control-critical information; it therefore combines vision-language prefill attention with action decode attention estimated via temporal smoothing and selects tokens through a union-plus-diversity strategy rather than a scalar fusion score (Liu et al., 20 Nov 2025). This suggests that in multimodal or control settings, ScTD may have to preserve semantics together with an additional task-specific invariance.

4. Representative instantiations across modalities

The term ScTD is used most explicitly in language-model pretraining, but closely related formulations now span multiple modalities and training regimes.

Paper	Setting	Semantic-preservation mechanism
"Revisiting Token Dropping Strategy in Efficient BERT Pretraining" (Zhong et al., 2023)	BERT pretraining	Self-distillation between full and token-dropped paths
"Semantic-Aware Prefix Learning for Token-Efficient Image Generation" (Li et al., 26 Mar 2026)	Latent image generation	Semantic prefix injection plus random tail token dropping
"Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping" (Parihar et al., 22 Jun 2026)	Reference-conditioned diffusion	Random dropping during fine-tuning, task-aware selection at inference
"SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling" (Liu et al., 21 Aug 2025)	Long-context LM	Semantic clustering, covariance-trace entropy, budgeted span selection
"Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation" (Tang et al., 2023)	Semantic segmentation	Confidence-based early exit with per-class top-$s_i$5 retention
"Dynamic Token-Pass Transformers for Semantic Segmentation" (Liu et al., 2023)	Semantic segmentation	Confidence-guided token passing and reconstruction
"VLA-Pruner: Temporal-Aware Dual-Level Visual Token Pruning for Efficient Vision-Language-Action Inference" (Liu et al., 20 Nov 2025)	VLA inference	Dual-level importance from semantic and action signals
"Training-free Dropout Sampling for Semantic Token Acceptance in Speculative Decoding" (Lee et al., 11 Feb 2026)	Speculative decoding	MC-dropout target neighborhood and JS-based token acceptance
"ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning" (Qin et al., 21 Oct 2025)	SFT data selection	Retrospective excess loss plus prompt-attention score
"Contrastive Token Learning with Similarity Decay for Repetition Suppression in Machine Translation" (Dai et al., 2024)	NMT repetition suppression	Attention-similarity and distance-weighted contrastive suppression

These papers do not all use the term identically. In some, token dropping refers to skipping computation in hidden layers; in others it refers to truncating latent prefixes, pruning conditioning grids, masking supervised response tokens, or rejecting speculative draft tokens. What unifies them is the attempt to remove tokens without destroying the semantic content required by the downstream objective.

Several of these variants broaden what counts as “semantic consistency.” Sparse Context shows that reference-conditioned diffusion models can retain only $s_i$6 to $s_i$7 of reference tokens during fine-tuning and then apply edge-aware or saliency-based selection at inference, achieving a $s_i$8 increase in inference speed for multi-reference generation and an $s_i$9 increase for single-reference generation without compromising visual quality (Parihar et al., 22 Jun 2026). SemToken compresses long text by local semantic clustering and semantic entropy $$\mathcal{L}_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid X_l)\right],$$0, reporting up to $$\mathcal{L}_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid X_l)\right],$$1 reduction in token count and $$\mathcal{L}_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid X_l)\right],$$2 speedup with negligible or no degradation in perplexity and downstream accuracy (Liu et al., 21 Aug 2025). DropMatch uses MC dropout only at the LM head to form an empirical target distribution and accepts a draft token if

$$\mathcal{L}_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid X_l)\right],$$3

or if the draft token matches the majority token among the dropout heads, yielding speedups ranging from $$\mathcal{L}_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid X_l)\right],$$4 to $$\mathcal{L}_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid X_l)\right],$$5 over the standard baseline and up to an additional $$\mathcal{L}_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid X_l)\right],$$6 when layered on EAGLE3 (Lee et al., 11 Feb 2026).

Other variants move the concept from computational pruning to optimization. ssToken keeps only a top-$$\mathcal{L}_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid X_l)\right],$$7 fraction of response tokens in the SFT loss according to a combined score

$$\mathcal{L}_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid X_l)\right],$$8

where REL is retrospective excess loss against a history model and $$\mathcal{L}_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid X_l)\right],$$9 is response-to-prompt attention mass. Both self-modulated selection and semantic-aware selection alone outperform full-data fine-tuning, and their combination yields the best average results (Qin et al., 21 Oct 2025). CTSD, in turn, suppresses harmful repetition through a contrastive loss weighted by attention-similarity and distance decay,

$Y$0

so that repetitions are penalized when they are both recent and semantically overlapping, rather than merely repeated in surface form (Dai et al., 2024).

5. Empirical behavior, trade-offs, and common misconceptions

A central misconception is that any token dropping preserving average accuracy is automatically semantic-consistent. The original ScTD paper directly disputes this: vanilla token dropping in BERT maintains efficiency but exhibits stronger semantic drift, reduced semantic probe performance in dropped layers, and a marked deficit on semantic-intensive tasks (Zhong et al., 2023). The difference between token dropping and ScTD is therefore not simply aggressiveness, but whether semantic preservation is made explicit in the objective or architecture.

A second misconception is that semantic consistency implies no information loss. The cited work generally does not claim that. In SMAP, dropping all latent tokens still produces blurry but semantically correct reconstructions, whereas full latents restore fine details; the design goal is that semantics survive aggressive compression while residual detail degrades gracefully (Li et al., 26 Mar 2026). Sparse Context reports that even without retraining, random dropping of up to $Y$1 of reference tokens preserves coarse layout and identity, but fine-grained details degrade, which is why the model is later fine-tuned with random dropping and paired with task-aware token selection (Parihar et al., 22 Jun 2026). Semantically consistent dropping therefore usually preserves category, structure, or task alignment before it preserves texture, instance identity, or exact lexical realization.

A third misconception concerns dense prediction. In segmentation, tokens cannot simply disappear. Dynamic Token Pruning keeps per-class representative tokens in deeper layers, while DoViT reconstructs halted tokens into their original sequence positions before decoding; these systems show that dense-output ScTD depends on reconstruction and context retention rather than on permanent deletion (Tang et al., 2023, Liu et al., 2023). This is reinforced by empirical results: DToP reports average $Y$2 computational savings without notable accuracy degradation, whereas DoViT reports about $Y$3 FLOPs reduction on Cityscapes with the drop of mIoU within $Y$4 and throughput increases to more than $Y$5 on ViT-L/B (Tang et al., 2023, Liu et al., 2023).

A fourth trade-off concerns how semantic signals are chosen. In BERT token dropping, cumulative MLM loss provides a task-aligned importance measure and outperforms random or frequency-based dropping, but it still does not eliminate semantic loss (Hou et al., 2022, Zhong et al., 2023). In ssToken, loss dynamics alone are not sufficient, so prompt-attention is added as a complementary semantic signal (Qin et al., 21 Oct 2025). In VLA-Pruner, semantic-only pruning is explicitly said to be misaligned with action generation, motivating a dual-level criterion that uses both semantic and action importance (Liu et al., 20 Nov 2025). This suggests that “semantic consistency” is often objective-relative: what must remain invariant depends on whether the system is predicting masked words, semantic masks, images, actions, or verified continuations.

6. Limitations and open directions

The limitations identified across the literature are strikingly consistent. Many methods rely on heuristic thresholds, budgets, or schedules. The original ScTD for BERT introduces two additional hyperparameters, $Y$6 for semantic-consistency strength and $Y$7 for the frequency of semantic-align steps (Zhong et al., 2023). DToP depends on confidence threshold $Y$8, prune locations, and $Y$9 for per-category retention; DoViT depends on a global threshold $$\mathcal{L}^*_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid \tilde{X}_l)\right].$$0; SemToken depends on similarity threshold $$\mathcal{L}^*_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid \tilde{X}_l)\right].$$1 and density threshold $$\mathcal{L}^*_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid \tilde{X}_l)\right].$$2; ssToken depends on token ratio $$\mathcal{L}^*_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid \tilde{X}_l)\right].$$3, balance coefficient $$\mathcal{L}^*_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid \tilde{X}_l)\right].$$4, and attention layer choice (Tang et al., 2023, Liu et al., 2023, Liu et al., 21 Aug 2025, Qin et al., 21 Oct 2025).

Another limitation is the narrowness of semantic supervision. SMAP uses class-level semantics on ImageNet rather than richer natural language, and its authors explicitly note that semantic decomposition may be more complex in text-to-image settings (Li et al., 26 Mar 2026). Sparse Context depends on the choice of task-aware heuristic such as Canny edges or saliency maps and can degrade when the heuristic prioritizes the wrong content, as in some object-replacement cases (Parihar et al., 22 Jun 2026). VLA-Pruner relies on temporal continuity of action attention, which may weaken in dynamic environments or egocentric views (Liu et al., 20 Nov 2025). DropMatch assumes that MC-dropout perturbations of the LM head define a useful uncertainty neighborhood; the paper shows this works best at moderate dropout probabilities such as $$\mathcal{L}^*_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid \tilde{X}_l)\right].$$5 to $$\mathcal{L}^*_{\text{MLM}} = \mathbb{E}\left[-\sum \log P(Y \mid \tilde{X}_l)\right].$$6 and can degrade when dropout is too high (Lee et al., 11 Feb 2026).

A further limitation is scope. The original ScTD evidence is concentrated on BERT-base and BERT-large under MLM pretraining (Zhong et al., 2023). Several later methods are likewise domain-specific: DToP and DoViT are specialized to dense segmentation, Sparse Context to reference-conditioned generation, VLA-Pruner to embodied control, and CTSD to repetition suppression in translation (Tang et al., 2023, Liu et al., 2023, Parihar et al., 22 Jun 2026, Liu et al., 20 Nov 2025, Dai et al., 2024). This suggests that ScTD is better understood as a design pattern than as a single algorithm.

The future directions named across these works point toward richer semantics, adaptive policies, and tighter coupling between pruning and downstream objectives. Proposed directions include extending semantic prefixes from class labels to text embeddings and applying semantic-consistent dropping to video (Li et al., 26 Mar 2026); learning better task-aware selection rules for sparse reference conditioning and adapting token budgets across diffusion timesteps (Parihar et al., 22 Jun 2026); joint training of tokenization and modeling, multilingual or code extensions, and integration with retrieval-augmented generation for semantic compression (Liu et al., 21 Aug 2025); adaptive temporal windows or learned predictors of action-attention shifts in VLA systems (Liu et al., 20 Nov 2025); and adaptive token ratios or improved history models for semantic-aware token selection during fine-tuning (Qin et al., 21 Oct 2025). A plausible implication is that future ScTD systems will increasingly combine semantic scoring, downstream-task scoring, reconstruction guarantees, and compute-aware optimization in a single framework rather than treating semantic preservation as an auxiliary regularizer.