Open-World Evaluation for Retrieving Diverse Perspectives (2409.18110v2)

Abstract: We study retrieving a set of documents that covers various perspectives on a complex and contentious question (e.g., will ChatGPT do more harm than good?). We curate a Benchmark for Retrieval Diversity for Subjective questions (BERDS), where each example consists of a question and diverse perspectives associated with the question, sourced from survey questions and debate websites. On this data, retrievers paired with a corpus are evaluated to surface a document set that contains diverse perspectives. Our framing diverges from most retrieval tasks in that document relevancy cannot be decided by simple string matches to references. Instead, we build a LLM-based automatic evaluator that decides whether each retrieved document contains a perspective. This allows us to evaluate the performance of three different types of corpus (Wikipedia, web snapshot, and corpus constructed on the fly with retrieved pages from the search engine) paired with retrievers. Retrieving diverse documents remains challenging, with the outputs from existing retrievers covering all perspectives on only 40% of the examples. We further study the effectiveness of query expansion and diversity-focused reranking approaches and analyze retriever sycophancy.

• The paper presents BeRDS, a benchmark evaluating retrieval diversity with metrics like MRecall to assess multiple perspectives.
• It compares various retrievers and corpora, showing that re-ranking and query expansion techniques significantly enhance both diversity and precision.
• The paper highlights retriever sycophancy and advocates for advanced frameworks to address limitations in current information retrieval systems.

Open-World Evaluation for Retrieving Diverse Perspectives

The paper "Open-World Evaluation for Retrieving Diverse Perspectives" by Hung-Ting Chen and Eunsol Choi addresses the task of retrieving documents that encompass a variety of perspectives on complex, contentious questions. Unlike traditional information retrieval (IR) tasks, where document relevancy is typically determined by simple string matches to reference questions, this research explores the challenge of measuring the diversity of perspectives without assuming a gold-standard corpus. The central contribution is the introduction of a Benchmark for Retrieval Diversity for Subjective questions (BeRDS), alongside the development of novel evaluation metrics and strategies to enhance retrieval diversity.

Key Contributions

1. Benchmark Development (BeRDS):
   • Dataset Composition: BeRDS includes 3,000 questions paired with an average of 2.3 perspectives per question. The questions are sourced from survey question collections and debate websites, thus inherently promoting diverse perspectives.
   • Evaluation Metrics: Evaluates retrievers on their ability to surface a set of documents that covers all potential perspectives rather than just factoid answers. Metrics include MRecall at k, which reflects whether the top-k retrieved documents encompass all perspectives, and precision to ensure document relevancy.
2. Corpus and Retriever Comparison:
   • Corpora Analysis: The paper utilizes three different corpora—Wikipedia, a web snapshot (Sphere), and documents fetched from the Google Search API.
   • Retriever Performance: Multiple retrievers, including BM25, DPR, and Contriever, are evaluated. Contriever paired with the Sphere corpus demonstrates the best performance, yet only achieves a MRecall of 30.64%, indicating significant scope for improvement in current retrieval systems.
3. Diversity Enhancement Techniques:
   • Re-ranking with MMR: Utilizes Maximal Marginal Relevance to balance relevancy and diversity in the re-ranked document set. This method improves the diversity for specific retriever-corpus settings, notably for Contriever.
   • Query Expansion: Generates multiple perspectives using GPT-4 and uses them to diversify retrieval queries, resulting in consistent improvements in both MRecall and precision metrics across different settings.
4. Retriever Sycophancy Analysis:
   • Sycophancy Effects: Retrievers show a tendency toward sycophancy, favoring documents aligned with the input query's perspective. This discovery partially accounts for the effectiveness of the query expansion strategy.

Methodological Innovations

• Perspective Detection Model: The paper builds a model to determine if a document supports a given perspective. Fine-tuned on GPT-4 outputs, a Mistral-7B model achieves notable performance, enabling efficient automatic evaluations.
• Open-World Evaluation Framework: Moving beyond predefined datasets, this framework supports comparisons across different knowledge sources, emphasizing the adaptability and robustness in varying informational landscapes.

Numerical Highlights and Observations

• Strong Corpus Effect: Sphere outperforms Wikipedia significantly in terms of coverage and diversity, as shown by its average MRecall improvements.
• Retrievers' Limitations: Even the best-performing retrievers face challenges, covering all perspectives in only a fraction of cases within the top-100 documents.
• Practical Diversity: Simple techniques like re-ranking and query expansion improve diversity, demonstrating that effective enhancements are practicable yet need further refinement.

Implications and Future Directions

The implications of this research are vast, both practically and theoretically. From a practical standpoint, retrieval systems better attuned to capturing diverse perspectives can significantly enhance user satisfaction in applications that range from search engines to retrieval-augmented LLMs (RALMs). Theoretically, the findings highlight critical limitations in current retriever architectures, suggesting fruitful directions for future research.

Future Research Directions

1. Enhanced Evaluator Models: Developing more sophisticated evaluators that balance efficiency with accuracy while reducing the dependency on costly models like GPT-4.
2. Incorporating Fine-grained Perspectives: Extending datasets to include more nuanced and numerous perspectives, thus better reflecting the complexity of real-world issues.
3. Cross-domain Applications: Expanding the scope beyond debate and opinion data to include diverse fields such as healthcare and legal domains, where multiple perspectives are critical.

In sum, this paper provides a comprehensive foundation for the task of retrieving diverse perspectives, offering valuable insights, metrics, and strategies that can spur further advancements in the field. The combination of a robust benchmark, innovative evaluation techniques, and practical diversity enhancements marks a significant step towards more intelligent and inclusive information retrieval systems.