---
title: Dynamic Whole-Word Masking Curriculum
url: https://www.emergentmind.com/topics/whole-word-masking-with-dynamic-curriculum
type: topic
---

# Dynamic Whole-Word Masking Curriculum

Whole-word masking with a dynamic curriculum is a pre-training strategy in encoder-only Transformer models—exemplified by Chinese ModernBERT—that synchronizes the masking ratio with training progress, transitioning from high to low packet-level corruption rates. This approach aims to balance global and local reasoning, leveraging masking schedules that evolve with the optimization horizon. The method is operationalized through a mathematically precise schedule, specialized tokenization, integration with system-level optimizations, and is empirically validated on complex Chinese language understanding and retrieval benchmarks [2510.12285].

## 1. Mathematical Basis for the Dynamic Masking Curriculum

The dynamic curriculum is defined by a piecewise linear schedule that adjusts the whole-word mask ratio $\varepsilon(s)$ as pre-training advances. Let $S$ denote the total number of steps, $S_w$ the warmup length, and $s$ the current step. The mask ratios are bounded: $\varepsilon_{\min} = 0.15$, $\varepsilon_{\max} = 0.30$. Formally,
\[
\varepsilon(s) =
\begin{cases}
\varepsilon_{\min} + (\varepsilon_{\max} - \varepsilon_{\min})\,\frac{s}{S_w}, & 0 \le s \le S_w\\[1em]
\varepsilon_{\max} - (\varepsilon_{\max} - \varepsilon_{\min})\, \frac{s - S_w}{S - S_w}, & S_w < s \le S
\end{cases}
\]
During the warmup ($s \le S_w$), the mask ratio linearly increases from 15% to 30%, incentivizing increased masking early in training. Subsequently ($s > S_w$), the ratio linearly decays from 30% down to 15%, gradually lowering the corruption rate to enable local semantic refinement.

## 2. Operational Workflow: Whole-Word Masking with Scheduling

The masking procedure is synchronized with the dynamic ratio $\varepsilon(s)$. The input is tokenized, and contiguous subwords are grouped as full-word spans. Words are randomly sampled, and their constituent tokens are masked until the expected proportion of masked tokens meets $\varepsilon(s)$. Mask application is performed as follows: 80% of word tokens are replaced with [MASK], 10% with a random token, and 10% are left unchanged. This ensures semantic coherence by masking entire words instead of isolated tokens, and guarantees that the running mask ratio tracks the evolving curriculum.

## 3. Motivations, Ablations, and Empirical Validation

Early high corruption rates (approx. 30%) induce the model to recover masked content using global context—promoting global reasoning. As training advances and the corruption rate drops (to 15%), the model is exposed to more intact context, enhancing its ability for token-level, local semantic reconstruction.

Ablation studies contrasted four configurations:
- Token-level masking at 15% (fixed)
- Whole-word masking at 15% (fixed)
- Whole-word masking at 30% (fixed)
- Dynamic curriculum: 15% $\rightarrow$ 30% $\rightarrow$ 15%

Empirical findings indicate that whole-word masking (WWM) outperforms token-level masking on word-level tasks (e.g., CMNLI, CSL in CLUE). Fixed high masking (30%) accelerates early convergence but underperforms long-term. The dynamic curriculum yields both rapid initial optimization and reduced final validation losses, effecting approximately 1–2 point gains on CLUE and $\sim$0.02 increases in Pearson $r$ on SimCLUE when compared to any fixed rate [2510.12285].

## 4. Integration with Tokenization, Attention, and Optimization

The curriculum's effectiveness depends on the underlying tokenization scheme. A 32k BPE vocabulary is used to optimize chars/token (1.41 at 8k context) and enable reliable grouping for WWM. The schedule is employed during both Stage I (1024 tokens) and Stage II (8192 tokens) of pre-training. Lower mask rates in long-context phases facilitate dense local pattern acquisition, while the earlier high masking instills robust global context learning.

The masking curriculum is coordinated with a damped-cosine learning-rate schedule; both schedules peak near the end of warmup and decay afterward. Alternating local/global attention mechanisms and the rotary positional encoding (RoPE) stabilize optimization across the spectrum of mask ratios.

## 5. Quantitative Impact on Downstream Tasks

The downstream effects are assessed on the CLUE benchmark and SimCLUE STS evaluation. Under unified fine-tuning (3 epochs, peak LR $=3\times10^{-5}$), Chinese ModernBERT with dynamic WWM yields competitive or superior results across eight CLUE tasks. Selected results are summarized:

| Model                   | AFQMC | TNEWS | IFLYTEK | CMNLI | WSC   | CSL   | OCNLI | C$^3$   |
|-------------------------|-------|-------|---------|-------|-------|-------|-------|---------|
| Chinese ModernBERT      | 73.87 | 56.90 | 60.15   | 83.96 | 52.10 | 86.20 | 79.10 | 82.65   |
| RoBERTa-wwm-large       | 76.55 | 58.61 | 62.98   | 82.12 | 74.60 | 82.13 | 78.20 | 73.82   |

For semantic textual similarity (STS) on SimCLUE, fine-tuning with both SimCLUE and T2Ranking datasets (∼5M pairs) yields Pearson $r = 0.5050$, Spearman $\rho = 0.5367$, surpassing Qwen-0.6B-embedding under the same evaluation.

| Model                       | Contrastive Data | $r$ (Pearson) | $\rho$ (Spearman) |
|-----------------------------|-----------------|--------------|-------------------|
| ModernBERT+5M pairs         | 5M              | 0.5050       | 0.5367            |
| Qwen-0.6B-embed             | 169M            | 0.4965       | 0.5211            |
| jina-v2-base-zh             | >800M           | 0.5188       | 0.5501            |
| gte-multilingual            | 2.94B           | 0.5384       | 0.5730            |

Pearson’s $r$ and Spearman’s $\rho$ are the principal correlation metrics used.

## 6. Significance for Large-Scale Pre-Training

Whole-word masking with a dynamic curriculum—spanning a 30% to 15% masking schedule—effectively balances the dual objectives of rapid optimization and robust convergence. Its explicit integration with tokenizer design, context-length scaling, and adaptive attention yields consistent improvements in both Chinese language understanding and retrieval tasks. These results indicate that curriculum-aware masking strategies are a potent lever for advancing encoder-based pre-training, particularly in languages with complex morphology and tokenization phenomena [2510.12285].

Source: https://www.emergentmind.com/topics/whole-word-masking-with-dynamic-curriculum