Understanding Performance of Long-Document Ranking Models through Comprehensive Evaluation and Leaderboarding (2207.01262v3)

Abstract: We evaluated 20+ Transformer models for ranking of long documents (including recent LongP models trained with FlashAttention) and compared them with a simple FirstP baseline, which applies the same model to the truncated input (at most 512 tokens). We used MS MARCO Documents v1 as a primary training set and evaluated both the zero-shot transferred and fine-tuned models. On MS MARCO, TREC DLs, and Robust04 no long-document model outperformed FirstP by more than 5% in NDCG and MRR (when averaged over all test sets). We conjectured this was not due to models' inability to process long context, but due to a positional bias of relevant passages, whose distribution was skewed towards the beginning of documents. We found direct evidence of this bias in some test sets, which motivated us to create MS MARCO FarRelevant (based on MS MARCO Passages) where the relevant passages were not present among the first 512 tokens. Unlike standard collections where we saw both little benefit from incorporating longer contexts and limited variability in model performance (within a few %), experiments on MS MARCO FarRelevant uncovered dramatic differences among models. The FirstP models performed roughly at the random-baseline level in both zero-shot and fine-tuning scenarios. Simple aggregation models including MaxP and PARADE Attention had good zero-shot accuracy, but benefited little from fine-tuning. Most other models had poor zero-shot performance (sometimes at a random baseline level), but outstripped MaxP by as much as 13-28% after fine-tuning. Thus, the positional bias not only diminishes benefits of processing longer document contexts, but also leads to model overfitting to positional bias and performing poorly in a zero-shot setting when the distribution of relevant passages changes substantially. We make our software and data available.

• The paper demonstrates that simple truncation methods (FirstP baseline) can achieve competitive performance compared to sophisticated Transformer-based models.
• It assesses 13 models on MS MARCO and Robust04 collections, highlighting challenges in evaluation and reproducibility.
• The study questions the reliance on popular datasets for long-document tasks, urging the development of improved benchmarks for future research.

Performance of Long-Document Ranking Models: An Analytical Overview

The paper, entitled "Understanding Performance of Long-Document Ranking Models through Comprehensive Evaluation and Leaderboarding," conducted a systematic evaluation of recent models for long-document ranking. The paper assessed thirteen models using two well-acquainted document collections, MS MARCO and Robust04. The highlight of the paper was the exploration of specialized Transformer models like Longformer, which can process long documents directly. However, the research found that simpler methods, such as the FirstP baseline that truncates documents, deliver surprisingly effective results.

Key Insights and Findings

The paper questions several core assumptions and practices in long-document ranking:

1. Effectiveness of Simple Baselines:
   • The authors demonstrate that the FirstP baseline is unexpectedly competitive. This baseline involves truncating documents to satisfy typical transformer input constraints, offering practical performance with reduced computational costs.
2. Challenges in Evaluation:
   • The evaluation of long-document ranking models faces various challenges, including training complexities and reproducibility issues due to differences in initial models and training sets.
3. Dataset Suitability:
   • The authors argue that popular datasets like MS MARCO and Robust04 are not well-suited for benchmarking long-document models. This is a critical insight as it challenges the current research practices predominantly relying on these datasets.
4. General Performance:
   • Long-document models, including those using sparsified attention mechanisms like Longformer and BigBird, showed marginal improvements over simpler baselines. This questions the additional complexity and computational demands of using such models versus their effectiveness.

Methodology and Experiments

The authors conducted an extensive series of experiments to validate their claims:

• Model Zoo: The paper included a diverse set of models with different architectures and configurations.
• Reproducibility Concerns: The evaluation identified challenges in replicating results, attributed to differences in seed selection, initial starting models, and pre-training differences.
• Architecture Insights: Although models like PARADE-Transformer slightly improved when incorporating query embeddings into aggregation Transformers, the gains were modest and sometimes context-dependent.

Implications and Future Directions

The findings of this paper open up several avenues for further exploration:

1. Dataset Analysis and Development: Given the identified limitations of existing popular datasets in benchmarking long-document models, the field may benefit from developing new datasets or refining annotation techniques to better capture long-document nuances.
2. Model Simplification: The effectiveness of simple models like FirstP suggests that model complexity should be carefully justified against performance and computational gains, especially when the latter is marginal.
3. Investigation of Biases: The potential biases in the position of relevant passages within documents signify that further research is necessary to evaluate their effects on model training and performance.

Concluding Remarks

This comprehensive evaluation of long-document ranking models offers critical insights into their performance and limitations. The finding that simple baselines often perform comparably well urges the community to critically assess the complexity and data requirements of more sophisticated models. It also compels the community to rethink the assumptions inherent in the prevalent use of particular benchmark datasets for long-document evaluations. As such, the research initiates a meaningful dialogue about achieving balance in model complexity, computational efficiency, and real-world applicability in information retrieval tasks.