- The paper shows that BERT-based rankers require thousands of annotated queries to outperform BM25, highlighting the high cost of manual labeling.
- The paper demonstrates pseudo-labeling, generating labels via BM25, consistently boosts performance by 5–15% when annotated data is scarce.
- The paper finds that while transfer learning can underperform with domain shifts, fine-tuning pseudo-labeled models may achieve superior IR results.
Evaluation of Transfer Learning and Pseudo-labeling in BERT-based Ranking Models
The paper, presented at SIGIR 2021, evaluates the effectiveness of transfer learning and pseudo-labeling techniques in BERT-based neural ranking models across five distinct English datasets in the context of information retrieval (IR). This research diverges from previous studies by focusing on collections with substantial query numbers, allowing for a comprehensive full-shot evaluation. Additionally, it addresses the constraints posed by non-commercial dataset licenses that limit transfer learning's applicability, promoting pseudo-labeling as an alternative strategy.
Summary of Research Questions and Findings
- Data Needs for Training from Scratch:
- A BERT-based ranker requires a significant number of annotated queries to outperform BM25. On datasets like Yahoo! Answers, thousands of queries are necessary, emphasizing the high cost of manual annotation and the importance of strategies that minimize this need.
- Effectiveness of Pseudo-labeling:
- Models trained on pseudo-labels, generated via BM25, consistently surpassed BM25's performance by 5-15%. This suggests pseudo-labeling as a viable approach when annotated data is scarce, though improvements were smaller compared to earlier studies.
- Transfer Learning Performance:
- Transfer learning doesn't universally outperform BM25. In fact, transferred models can sometimes underperform compared to both BM25 and models trained on pseudo-labels, especially when dataset characteristics differ significantly from the original training data.
- Comparison with Pseudo-labeling:
- Transferred models generally show better performance than those trained solely on pseudo-labels. However, by fine-tuning pseudo-labeled models with a moderate amount of annotated data, performance can not only match but occasionally exceed that of transferred models.
- Impact of Few-shot Training:
- Few-shot training, while capable of enhancing pseudo-labeled models, can paradoxically degrade the performance of transfer-learned models. This effect is particularly stark for datasets like Yahoo! Answers, indicating potential overfitting and a loss of learned knowledge from the transfer step.
Implications and Future Research Directions
The paper provides insights that question the reliability of transfer learning without substantial in-domain data, highlighting the potential of pseudo-labeling supported by limited annotated data as a competitive alternative. This approach might be more robust to domain shifts given that it does not rely on data from external sources which may introduce distribution mismatches.
Future research should focus on enhancing the stability and effectiveness of few-shot learning to mitigate instances of severe performance degradation observed in this paper. Additionally, the exploration of hybrid models that dynamically integrate transfer learning and pseudo-labeling, adapting the strategy based on available data and task-specific needs, represents an intriguing avenue for development in neural ranking techniques.
In conclusion, this paper underscores the complexity of applying transfer learning in IR tasks and advocates for a nuanced approach that combines the strengths of transfer learning and pseudo-labeling to achieve optimal results across diverse datasets.