Span-Corruption Denoising for Pre-training

• Span-corruption denoising is a method for pre-training sequence-to-sequence models that masks random contiguous spans, enabling efficient self-supervised learning.
• The SpacTor framework augments this approach by integrating replaced-token detection in a two-stage curriculum to harness early training gains and improve downstream results.
• Empirical studies show it reduces pre-training steps and compute, matching or surpassing benchmarks like SuperGLUE and SQuAD under fixed FLOP budgets.

Span-corruption denoising is a pre-training objective for sequence-to-sequence models wherein random contiguous spans of tokens from an input sequence are masked and replaced with special sentinel tokens, and the model is tasked with reconstructing the original contents of those masked spans. Initially introduced in the T5 framework, span-corruption denoising has proven to be an information-rich and efficient self-supervised learning task. The SpacTor framework extends span-corruption denoising by incorporating a hybrid objective that combines span corruption (SC) with replaced-token detection (RTD) in a two-stage curriculum, resulting in significant efficiency gains and improved downstream task performance without loss of quality (Ye et al., 2024).

1. Span-Corruption Denoising Formulation

Given a tokenized input sequence $X = (x_0,\ldots,x_{N-1})$, the span-corruption objective operates by masking $p$ disjoint spans $S_k = X_{i_k, j_k} = (x_{i_k},\ldots,x_{j_k})$ such that each span’s start position $i_k$ is selected uniformly at random, with the span length $\ell_k = j_k-i_k+1$ drawn from a light-tailed distribution with mean $\mu = 3$; in effect, the average span length is approximately 3 tokens. The spans are constructed to be non-overlapping and to jointly cover a fraction $r$ of the sequence (typically $r \approx 15\%$).

Each masked span $S_k$ is replaced in $X$ by a unique sentinel token $p$0 to produce the corrupted input $p$1. The target sequence $p$2 for the decoder is then the concatenation of masked spans, each prefixed by their corresponding sentinel token.

$p$3

The span-corruption loss is cross-entropy over the outputs of the decoder, conditioned on the corrupted input: $p$4 This can be equivalently viewed as the expected negative log-likelihood over all masked positions: $p$5

2. Integration with Replaced-Token Detection

In the SpacTor framework, span-corruption is augmented with replaced-token detection (RTD), following the ELECTRA paradigm. After constructing $p$6 via span corruption, an additional 15–20% of the remaining (non-span) tokens are masked at random, producing $p$7. A generator $p$8—an encoder-only Transformer of roughly one-quarter the size of the main model—fills in the masked tokens, creating a noised sequence $p$9.

The main T5 model $S_k = X_{i_k, j_k} = (x_{i_k},\ldots,x_{j_k})$0 encodes $S_k = X_{i_k, j_k} = (x_{i_k},\ldots,x_{j_k})$1 and, for each position $S_k = X_{i_k, j_k} = (x_{i_k},\ldots,x_{j_k})$2, predicts with a sigmoid classifier $S_k = X_{i_k, j_k} = (x_{i_k},\ldots,x_{j_k})$3 whether $S_k = X_{i_k, j_k} = (x_{i_k},\ldots,x_{j_k})$4 is the original token ($S_k = X_{i_k, j_k} = (x_{i_k},\ldots,x_{j_k})$5) or has been replaced ($S_k = X_{i_k, j_k} = (x_{i_k},\ldots,x_{j_k})$6): $S_k = X_{i_k, j_k} = (x_{i_k},\ldots,x_{j_k})$7 The corresponding RTD loss is: $S_k = X_{i_k, j_k} = (x_{i_k},\ldots,x_{j_k})$8 The generator is trained with standard cross-entropy for masked positions: $S_k = X_{i_k, j_k} = (x_{i_k},\ldots,x_{j_k})$9

3. Two-Stage Hybrid Curriculum

Empirical analysis shows that the effectiveness of RTD is highest during the early phases of pre-training, where it promotes “all-token” attention and accelerates representation learning within the encoder. As training progresses and generator $i_k$0 produces increasingly plausible—if sometimes misleading—replacements, continuing to optimize both RTD and SC losses concurrently begins to degrade final denoising performance.

To address this, SpacTor adopts a two-stage curriculum:

1. Stage 1 (steps $i_k$1): joint optimization over the full hybrid objective

$i_k$2

with $i_k$3.

2. Stage 2 (steps $i_k$4 end): discard the generator $i_k$5 and RTD head, continuing training under

$i_k$6

Selection of the transition step $i_k$7 is critical. For a 1 million step run (Baseline T5-Base), inflections in validation and downstream scores occur at 200K–250K steps without transition; optimal results are found for $i_k$8, representing $i_k$9–$\ell_k = j_k-i_k+1$0 of total training.

4. Empirical Efficiency and Effectiveness

Span-corruption denoising as operationalized in SpacTor achieves greater efficiency compared to vanilla SC. Key findings include:

• Data efficiency: Matching baseline downstream performance is achieved with only 500K pre-training steps—50% fewer than standard SC alone.
• Compute reduction: Although hybrid steps cost approximately 37.5% more floating point operations (FLOPs) per iteration, total pre-training FLOPs are reduced by ~40% (1.2× baseline FLOPs at 500K steps versus 2.0× at 1M steps).
• Performance under equal compute: For identical FLOP budgets, SpacTor-pretrained T5-Base surpasses the SC-only baseline on a spectrum of benchmarks: SuperGLUE, SQuAD, CNN/DailyMail, GLUE, Rainbow, MMLU, BBH. For example, SuperGLUE performance at SpacTor(120 K) with 1.1× FLOPs matches baseline at 2× FLOPs.
• Ablation studies: Never switching to SC only ($\ell_k = j_k-i_k+1$1) yields an early lead in metrics but underperforms after 250K steps. Immediate switch ($\ell_k = j_k-i_k+1$2) (i.e., pure SC) is the standard baseline. Optimal tradeoff is achieved with intermediate $\ell_k = j_k-i_k+1$3, harnessing early RTD-driven gains and later SC purity.

5. Implementation Specifics

• Model architecture: T5.1.0 encoder-decoder; T5-Base: 12 layers, 768 hidden size, 12 heads; T5-Large: 24/1024/16.
• Generator $\ell_k = j_k-i_k+1$4: encoder-only Transformer, $\ell_k = j_k-i_k+1$5 size of $\ell_k = j_k-i_k+1$6 (Base: 4 layers × 1024 MLP; Large: 6 × 2048).
• Span corruption: fraction $\ell_k = j_k-i_k+1$7, average span length $\ell_k = j_k-i_k+1$8.
• Additional MLM masking for RTD: 15% for Base, 20% for Large.
• Tokenizer: SentencePiece, 32K vocabulary.
• Optimizer: Adafactor.
• Input length: 512 tokens.
• Batch size: 2048.
• RTD head: MLP size matches discriminator MLP (3072 for Base, 4096 for Large), GELU nonlinearity.

6. Significance and Context

Span-corruption denoising, as implemented in SpacTor, is formally equivalent to the original T5 masking-and-fill strategy—a conditional denoising task over continuous spans of text. Its effectiveness is enhanced when combined, during early pre-training, with an encoder-side replaced-token detection objective, which is subsequently phased out in favor of pure SC. This hybridization enables substantial computation and data savings without sacrificing model quality, and yields consistent gains under fixed compute budgets across standard natural language understanding and generation benchmarks (Ye et al., 2024). These results position span-corruption denoising as a foundational objective for efficient pre-training of encoder-decoder models.