CoRanking: Collaborative Ranking with Small and Large Ranking Agents
In their scholarly work, the authors present a novel framework dubbed "CoRanking," which effectively addresses the significant computational inefficiencies associated with LLMs in the context of passage ranking tasks. The framework skillfully integrates both small and large ranking models to achieve an optimal balance between efficiency and effectiveness. The crux of the innovation lies in reducing the reliance on LLMs for exhaustive listwise reranking processes through a strategic division of tasks between small rerankers and LLM rerankers.
The impetus for this research arises from the recognized limitations of employing LLMs, such as GPT-4, which, despite high performance, are fraught with inefficiency due to their large-scale parameter requirements and the sliding window approach's iterative nature. This methodology often incurs a substantial computational burden, which becomes an efficiency bottleneck particularly when real-time application is a priority.
The CoRanking framework is structured in a three-stage process:
- Small Listwise Reranker (SLR): Initially, a small-sized reranker pre-screens candidate passages to elevate the most relevant ones to the top of a candidate list. This pre-ranking limits the scope of passages that need be processed by the LLM, mitigating extensive computational overhead.
- Passage Order Adjuster (POA): To counteract the potential sub-optimality in the passage order known to influence LLM reranking effectiveness, a passage order adjuster based on reinforcement learning is introduced. This component rearranges the top passages according to the preferences ingrained in the LLM during pre-training, ensuring alignment with the LLM's inherent biases towards passage order.
- LLM Listwise Reranker (LLR): Finally, the LLM is applied exclusively to the top-k passages identified by the preceding stages. This targeted application not only enhances efficiency but also maintains, or indeed improves, the reranking effectiveness when compared to full-scale LLM listwise reranking.
The researchers validate CoRanking's efficacy through extensive experiments conducted on three information retrieval (IR) benchmarks—TREC, BEIR, and BRIGHT. The numerical results are compelling: the framework reportedly improves ranking latency by approximately 70% without sacrificing the effectiveness, as demonstrated by superior performance metrics (NDCG@10) compared to using only the LLM-based listwise reranker.
From a theoretical perspective, CoRanking offers meaningful insights into the dynamic interplay between different sizes of ranking models. It highlights the potential of leveraging smaller, less computationally intensive models to offset the burdens associated with larger models, facilitating more scalable and sustainable AI solutions.
Practically, this research has immediate implications for deploying AI solutions in environments where computational resources are limited or costly. By reducing the computational load without compromising performance, CoRanking enables the deployment of advanced ranking systems on a more flexible range of hardware infrastructures.
Looking forward, the principles underpinning CoRanking could catalyze further advancements in AI resource management. Future work might explore broader applications of this collaborative framework across other context-heavy AI tasks, optimizations for even greater efficiency, or adaptations that account for more diverse LLM architectures beyond those presently available. By bridging the efficiency-gap of LLMs, CoRanking sets a precedent for harmonizing model performance with practical deployment constraints in AI research and beyond.