A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look (2411.08275v1)

Abstract: The application of LLMs to provide relevance assessments presents exciting opportunities to advance information retrieval, natural language processing, and beyond, but to date many unknowns remain. This paper reports on the results of a large-scale evaluation (the TREC 2024 RAG Track) where four different relevance assessment approaches were deployed in situ: the "standard" fully manual process that NIST has implemented for decades and three different alternatives that take advantage of LLMs to different extents using the open-source UMBRELA tool. This setup allows us to correlate system rankings induced by the different approaches to characterize tradeoffs between cost and quality. We find that in terms of nDCG@20, nDCG@100, and Recall@100, system rankings induced by automatically generated relevance assessments from UMBRELA correlate highly with those induced by fully manual assessments across a diverse set of 77 runs from 19 teams. Our results suggest that automatically generated UMBRELA judgments can replace fully manual judgments to accurately capture run-level effectiveness. Surprisingly, we find that LLM assistance does not appear to increase correlation with fully manual assessments, suggesting that costs associated with human-in-the-loop processes do not bring obvious tangible benefits. Overall, human assessors appear to be stricter than UMBRELA in applying relevance criteria. Our work validates the use of LLMs in academic TREC-style evaluations and provides the foundation for future studies.

• The paper demonstrates that fully automatic, LLM-based relevance assessments exhibit high correlation (Kendall’s τ up to 0.944) with manual evaluations.
• It compares four assessment strategies—fully manual, manual with filtering, post-edited, and fully automatic—to evaluate cost versus quality trade-offs.
• The study finds that hybrid human-LLM methods offer no significant advantage over automatic processes, prompting further refinement of LLM prompting.

Insights from the Large-Scale Study of Relevance Assessments with LLMs

The paper "A Large-Scale Study of Relevance Assessments with LLMs: An Initial Look" provides a comprehensive examination of using LLMs in the field of relevance assessments for information retrieval systems. This paper, conducted as part of the TREC 2024 Retrieval-Augmented Generation (RAG) Track, analyzes the viability of employing LLMs in a domain traditionally dominated by human evaluators.

Summary of Research and Methodology

The paper investigates four distinct approaches to relevance assessment:

1. Fully Manual Process: This represents the established method using human assessors at the National Institute of Standards and Technology (NIST).
2. Manual Assessment with Filtering: UMBRELA, an LLM-based tool, pre-filters documents before they are assessed by humans.
3. Manual Post-Editing of Automatic Assessment: UMBRELA provides initial labels, which are then reviewed and possibly revised by human assessors.
4. Fully Automatic Assessment: Utilizing UMBRELA's automatically generated labels without human input.

These methodologies were rigorously evaluated across a dataset covering 77 runs from 19 teams, generated for the TREC 2024 RAG Track.

Analysis of Results

A key finding is that LLM-generated relevance judgments using the UMBRELA tool highly correlate with fully manual assessments, suggesting that these automatic judgments can be viable substitutes for manual ones (Kendall’s τ ranging from 0.811 to 0.944 across different metrics and conditions). However, the anticipated benefits of combining human efforts with LLM-generated data in a hybrid approach did not surface as an improvement in correlation strength or quality of assessment. In fact, the mixed-methodologies did not yield significant advantages over the fully automatic process—highlighting a critical insight in the cost vs. quality trade-off discussion.

Moreover, while the analysis shows that human assessors typically apply stricter relevance criteria, UMBRELA occasionally provided overly broad relevance labels, leading to discrepancies primarily in the assignment of higher relevance grades.

Implications and Future Directions

This research demonstrates the potential of LLMs to significantly reduce the cost and time typically associated with human relevance assessment processes without substantively sacrificing quality. Given the high correlation between UMBRELA-induced rankings and those generated by traditional methods, it suggests a strong case for the strategic deployment of LLMs in large-scale relevance assessment tasks.

However, the research also provokes further exploration into refining LLM prompting strategies for more precise relevance judgments, particularly in ensuring that machine-generated evaluations align more closely with human reasoning. Additionally, the use of LLMs opens discussions around ethical implications, particularly in contexts where nuanced human judgment is invaluable.

Future studies might focus on expanding the model's training datasets to encompass more diverse query interpretations and continue probing into different combinations of automated processes and human oversight to balance performance with efficiency. As LLM capabilities grow and evolve, their role in evaluation might extend beyond relevance assessments to more complex tasks within the information retrieval spectrum.

The insights from this paper mark a significant step toward integrating AI-assisted methodologies into selection and evaluation processes, thereby broadening the horizon for cost-effective and scalable solutions in information systems evaluations.