The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles

Abstract: Masked Language Modeling (MLM) pre-training is one of the primary ways to initialize Neural Information Retrieval (IR) models prior to retrieval fine-tuning. However, studies show that MLM pre-trained models have limited readiness and transfer learning issues for fine-tuning them into Neural Bi-Encoder models. This paper studies the effect of different pre-training datasets and pre-training options on the MLM pre-trained models for retrieval fine-tuning. The study focuses on the SPLADE-style model, which uses the MLM layer also at fine-tuning time. More specifically, we experimented with Expanded-SPLADE (ESPLADE) models, a specific instance of SPLADE models, and in-house web document titles are used as datasets. Pre-training, fine-tuning, and evaluation with optional test-time pruning of sparse vectors are conducted. Our observations are three-fold: First, fine-tuned models of higher retrieval effectiveness at both unpruned and most strict pruned settings are mostly pre-trained on a general corpus, and pre-trained with a higher learning rate, showing lower MLM accuracies. Second, in the most strict pruned setting, those models show higher-level retrieval cost and a higher variance in the length of the individual postings list. Third, the repetition of the general pre-training dataset does not have much effect on retrieval effectiveness. The experimentation empirically identifies the potential limitations for aligning MLM pre-training to ESPLADE fine-tuning. Also, the experimentation provides an empirical observation that, at most strict pruned settings, the retrieval effectiveness is better maintained by the higher-level retrieval cost, showing the trade-off relationship between the two in our setting.

• The paper reveals that pre-training ESPLADE models on diverse, independent corpora with higher learning rates yields improved retrieval effectiveness despite lower traditional MLM scores.
• The paper finds that enhanced retrieval performance is associated with higher variance in logit outputs and posting list lengths, demonstrating a clear efficiency–effectiveness trade-off.
• The paper emphasizes that data diversity is more critical than data redundancy, suggesting a shift towards retrieval-specific pre-training methods for neural sparse retrieval.

Pre-training Expanded-SPLADE Models on Web Document Titles: An Expert Analysis

Introduction and Motivation

This work investigates the impact of different Masked LLM (MLM) pre-training regimes on the fine-tuning effectiveness of Expanded-SPLADE (ESPLADE) models for neural first-stage Information Retrieval (IR). ESPLADE introduces expanded, custom vocabularies for SPLADE-style models, leveraging the MLM output layer both during pre-training and IR fine-tuning, thereby supporting high-dimensional sparse representations. The study addresses a central challenge in neural IR: the known misalignment between standard MLM objectives and efficient/salient sparse retrieval, particularly in bi-encoder architectures optimized for rank-based objectives.

Prior results indicate that MLM pre-training is not optimally aligned for neural IR representations, often necessitating architectural or pre-training task alterations for improved transfer. However, empirical evidence remains limited on how corpus choice, learning rate, or data repetition in MLM pre-training affect downstream SPLADE-style efficiency–effectiveness trade-offs, especially when the output vocabulary is massively expanded and customized, as in ESPLADE. This paper provides a targeted empirical study using large-scale in-house web document titles.

Methodology

The evaluation leverages the ESPLADE framework, following the methodology in [dudek2023learning], where a custom output vocabulary (top 100K frequent unigrams) is constructed, and the MLM output layer is correspondingly expanded via weight-pooling over WordPiece embeddings. Multiple pre-training corpora are explored:

• Overlap-repeat: Repeated titles from the fine-tuning corpus, with overlapping data.
• Indep-repeat: Repeated titles from a large independent in-house title set.
• Indep-uniq: Unique, non-repeated titles from the independent set.

For each, two learning rates are tested: a standard ("lr-l") and a higher ("lr-h") regime (the latter using fewer training steps with steeper decay, providing effectively higher rates).

MLM pre-training is performed using standard 15% token masking, maintaining close adherence to the original BERT regime but with the custom vocabulary mapping and weight-pooling initialization. Fine-tuning utilizes a large-scale query-document click dataset (centering on Korean with diverse queries), with FLOPS-regularized contrastive loss for learning efficient sparse vector representations. Both training and inference leverage query/document pruning (static top-k logit selection, [lassance2023static]) to explore trade-offs under various efficiency–effectiveness regimes.

Evaluation measures include MRR@10 and Recall@10, along with a custom FLOPS metric intended to emulate first-stage IR cost in terms of postings list traversals. Statistical characterizations of the learned representations (e.g., logit-score-std, posting list length variance) are included for detailed model diagnostics.

Main Findings and Numerical Results

The paper’s findings are grouped along three axes:

1. Pre-training Regime and Retrieval Effectiveness

Fine-tuned ESPLADE models with the highest retrieval effectiveness (both unpruned and strictly pruned) were pre-trained on more general, independent corpora and with higher effective learning rates. These models exhibited higher logit-score-std (greater variance across logits in MLM outputs) but lower MLM pre-training accuracies compared to their counterparts. Notably, lower MLM accuracy (as traditionally defined) does not translate to poorer downstream retrieval—quite the opposite for ESPLADE with joint FLOPS-regularization.

For example, in the strictest retrieval scenario (top-5/q, top-10/d terms retained), the indep-uniq-lr-h model achieved the best MRR@10 (0.251) and Recall@10 (0.319), while overlap-repeat pretraining lagged, especially when the pre-training data overlapped heavily with the fine-tuning set.

2. Efficiency–Effectiveness Trade-off and Postings List Analysis

Under severe query/document pruning, the most effective models (indep-) incurred higher FLOPS (higher retrieval cost) and, notably, **higher variance in individual postings list lengths* (posting lists were less uniform, with more concentrated term matches). This suggests that increased retrieval effectiveness in SPLADE-style sparse models, under high pruning pressure, is maintained by selecting more salient—but also more costly—terms for retrieval, a phenomenon quantifiable by higher std in posting list lengths.

3. Impact of Pre-training Data Repetition

Repetition of general pre-training data (indep-repeat vs. indep-uniq) had negligible impact on retrieval effectiveness. Differences in model statistical profiles (entropy, margin, number of non-negative logits) were marginal compared to the sizable gap between overlap-repeat and independent pre-training. This suggests that, for ESPLADE, model transfer is mainly bottlenecked by the content and diversity of the pre-training, rather than sheer data scale/redundancy.

Additional Diagnostic Insights

• Higher effective learning rates yielded pre-trained models with higher logit-score-std, greater sparsity (lower non-neg-terms-avg), and lower average MLM scores—yet enabled better downstream SPLADE fine-tuning.
• Overfitting or excessive alignment of pre-training and fine-tuning corpora (overlap-repeat) promoted confident, but less adaptable, representations, as reflected in lower retrieval metrics and altered logit vector statistics.
• Scaling the number of pre-training steps without increasing data diversity or altering pre-training task produced diminishing returns or even fine-tuning failures.

Theoretical and Practical Implications

The experiments reinforce that standard MLM pre-training is not naturally aligned with SPLADE/ESPLADE fine-tuning for neural sparse retrieval, especially under efficiency constraints. There is a task-induced selection for transferable, non-overfitted, high-variance representations in the MLM layer, which is not correlated with classic MLM objective attainment (accuracy/loss minimization). This echoes prior findings [gao2021condenser, lassance2023experimental], but is here demonstrated in the high-dimensional, custom-vocabulary regime unique to ESPLADE.

Practically, for neural IR practitioners, pre-training SPLADE-style models on large, independent, and diverse corpora with higher learning rates is preferable—even at the expense of lower MLM classification performance—if the aim is to maximize pruned first-stage IR effectiveness. Aligning pre-training and fine-tuning data distributions, or optimizing for traditional MLM metrics, is insufficient and may be counterproductive.

The study opens several avenues for theoretical development:

1. Better retrieval-aligned pre-training objectives: There's a clear need for pre-training regimes that emphasize inter-sentence discrimination (e.g., contrastive objectives, ICT, or Condenser-style modifications). The classic intra-sentence MLM is limited in preparing models for bi-encoder or SPLADE fine-tuning dynamics.
2. Scaling implications: Results suggest that corpus diversity (not mere volume or repetition) governs transfer effectiveness. This hints at different scaling laws for sparse retrieval pre-training, compared to those seen in language modeling or dense retrieval [kaplan2020scaling, fang2024scaling].
3. Uniformity vs. adaptability: High variance in per-term postings lists underpins retrieval performance in pruned SPLADE settings. This suggests the importance of learning representations with discriminative, information-rich, but non-uniform activations, possibly motivating new regularization avenues.

Future Directions

Key future directions include:

• Development and benchmarking of retrieval-specific pre-training tasks for high-dimensional sparse settings—for example, contrastive or cross-instance prediction tasks integrated into the ESPLADE pipeline.
• Deeper analysis of the interaction between pre-training statistical characteristics and downstream IR regularizations such as dynamic/joint FLOPS, as well as the effect of pruning regimes (adaptive, learned, or static).
• Systematic exploration of scaling laws (dataset size, vocabulary expansion, step count) for large-scale SPLADE-type models, possibly extending to multilingual or cross-domain settings.

Conclusion

This paper provides a rigorous empirical characterization of how pre-training corpus choice, learning rate, and data distribution affect the transferability of large-output-vocabulary SPLADE-style models to sparse neural IR tasks. Strong numerical evidence demonstrates that optimal retrieval effectiveness and efficiency–effectiveness trade-offs are achieved not by maximizing MLM task performance, nor by pre-training on in-fine-tuning data, but by leveraging diverse, independent corpora with less overfitting and higher output variance. The findings reinforce the misalignment between MLM and neural IR objectives and underscore the need for retrieval-aligned pre-training for scalable, efficient, and effective neural sparse retrieval.