LongEval: Temporal & Extended Evaluation
- LongEval is a collection of frameworks that assess temporal persistence and long-context processing in varied tasks like IR, summarization, and text generation.
- It employs dynamic test collections with evolving snapshots, leveraging metrics such as nDCG and Relative nDCG Drop to evaluate system robustness over time.
- The framework also promotes reproducibility and standardization through enhanced tooling and methodologies applicable to both web and scholarly retrieval.
LongEval is a name used for several evaluation frameworks in contemporary NLP and IR, but it is most prominently associated with the CLEF lab on longitudinal evaluation of IR systems, where retrieval models are trained on earlier snapshots and tested on later, evolving collections in order to measure temporal persistence under document churn, query drift, and changing relevance signals (Cancellieri et al., 11 Mar 2025). The same name also appears in work on human evaluation of long-form summarization faithfulness, long-context language-model evaluation, and plan-based long-text generation, so the term is polysemous rather than designating a single benchmark family (Krishna et al., 2023, Fang et al., 2024, Wu et al., 26 Feb 2025).
1. Multiple uses of the name
In current research usage, “LongEval” designates several distinct artifacts rather than a single canonical benchmark.
| LongEval usage | Domain | Representative paper |
|---|---|---|
| CLEF LongEval | Longitudinal IR evaluation | (Cancellieri et al., 11 Mar 2025) |
| LongEval guidelines | Human evaluation of long-form summarization faithfulness | (Krishna et al., 2023) |
| LongEval framework built on LongPPL/LongCE | Long-context language modeling | (Fang et al., 2024) |
| LongEval benchmark for plan-based generation | Long-text generation | (Wu et al., 26 Feb 2025) |
The CLEF usage is historically the most elaborate as a shared-task ecosystem. It studies how IR systems behave when the evaluation environment changes over time, rather than assuming a static Cranfield-style collection. That framing has generated a sequence of benchmark papers, lab overviews, participant reports, and methodological extensions, including web retrieval, scientific retrieval, classification, and later retrieval-augmented generation settings (Deveaud et al., 2023, Cancellieri et al., 22 Sep 2025, Ninalga, 2023, Yang et al., 4 Jul 2026).
This multiplicity is itself a source of confusion. In IR, LongEval concerns temporal persistence under evolving corpora. In the other lines of work, the name refers to evaluation of long-form or long-context behavior. The shared label therefore signals a concern with longitudinal or extended-context evaluation, but not a shared dataset or metric stack.
2. LongEval as a longitudinal IR paradigm
The original IR benchmark, “LongEval-Retrieval,” defines a dynamic test collection composed of successive sub-collections, each representing the state of a web search environment at a given time step. In the formalization introduced for the 2023 web benchmark, a sub-collection at time is denoted , where topics remain fixed across time, while queries, documents, and relevance evolve (Deveaud et al., 2023).
The web benchmark was built from Qwant, a privacy-preserving web search engine with predominantly French traffic, and initially provided both a French “source” collection and an English “mirror” created through translation. Queries were extracted from logs through topic-based filtering; documents combined SERP-exposed pages with topic-conditioned random samples; and relevance assessments were derived from a simplified Dynamic Bayesian Network with session length $1$, equivalent to the Cascade Model. The click-based attractiveness proxy was estimated as
and then mapped into graded labels for offline IR evaluation (Deveaud et al., 2023).
This design was intended to bridge offline and online evaluation. Rather than treating time as a nuisance variable, LongEval makes temporal divergence the object of study. The lab’s central claim is that conventional single-snapshot evaluation can obscure a basic fact of deployed IR: collections are updated continuously, users develop new information needs, and relevance perceptions change over time (Cancellieri et al., 11 Mar 2025). Earlier LongEval editions already showed that system rankings can vary across time and that the highest-scoring system on one snapshot is not necessarily the most temporally robust, so effectiveness and stability can diverge (Cancellieri et al., 11 Mar 2025).
Although retrieval has been the core of the lab, the CLEF ecosystem was not limited to ad-hoc search. LongEval-Classification, for example, evaluated temporal robustness in sentiment analysis over timestamped Twitter data; a Bernice-based system using date-prefixed inputs and self-labeling ranked second with an overall score of 0.6923 and the best Relative Performance Drop of -0.0656 over the short evaluation set (Ninalga, 2023).
3. Tasks, datasets, and temporal splits in the CLEF lab
The 2025 edition was the third iteration of the lab and organized LongEval around two retrieval tasks sharing a single-training, evolving-test protocol: LongEval-WebRetrieval and LongEval-SciRetrieval (Cancellieri et al., 11 Mar 2025). In the web task, systems are trained on earlier Qwant data and evaluated on multiple later monthly snapshots. The call paper described a training set with 18 million French documents from June 2022 to February 2023, translated to English, together with 9,000 queries and click-model-derived relevance assessments, followed by seven monthly test lags from March 2023 to August 2023 (Cancellieri et al., 11 Mar 2025). A later overview reported post-processed 2025 WebRetrieval data with 19 million French documents and 119,341 queries for training, 14 million documents and 63,416 queries for test, and 33 million documents and 182,757 queries overall; it also noted that similar queries had been merged and identifiers unified, which improved continuity but made direct comparison with earlier releases unsafe despite overlapping periods (Cancellieri et al., 22 Sep 2025).
The scientific task extended LongEval beyond web search into scholarly retrieval. Its data source is CORE, a large open-access aggregation of scholarly documents with logs capturing queries, returned results, and clicks. The task processes Apache server logs, removes bot traffic, and records anonymized session IDs, search queries, first-ten-result SERPs, clicked URLs, result positions, and timestamps (Cancellieri et al., 11 Mar 2025). In the later 2025 overview, the SciRetrieval training snapshot covered mid-November 2024 to mid-December 2024 and contained about 2 million documents, 393 queries, and 4,262 relevance assessments, while the test snapshot for January 2025 contained more than 1 million documents together with 492 new queries and 99 held-out queries from the training period (Cancellieri et al., 22 Sep 2025).
The longitudinal structure is the defining feature. Training data always precede test data temporally; the time distance between datasets is the lag; and participants submit runs for each lag using the same trained system (Cancellieri et al., 11 Mar 2025). Retraining on test periods is not part of the core protocol in the 2025 call paper. In the web task, the lags are consecutive monthly snapshots, enabling measurement of degradation over increasing temporal distance. In the scientific task, the first year exposed fewer lags, but preserved the same principle of training on earlier data and evaluating on later snapshots (Cancellieri et al., 11 Mar 2025).
Across editions, the lab has also had to manage substantial overlap across snapshots. Earlier releases revealed that large overlaps in documents and queries were difficult to analyze because of identifier conventions, and later iterations therefore improved document and query identifiers and collection descriptions to better quantify overlap and drift (Cancellieri et al., 11 Mar 2025). A later overview explicitly reported that consecutive monthly web snapshots shared the highest proportion of documents and that overlap declined across longer gaps, making document overlap itself a useful empirical indicator of temporal divergence (Cancellieri et al., 22 Sep 2025).
4. Evaluation methodology, robustness measures, and empirical findings
LongEval’s basic effectiveness measure is nDCG, typically with emphasis on top ranks such as nDCG@10 for web search (Cancellieri et al., 11 Mar 2025). The early benchmark paper also introduced Relative nDCG Drop (RnD) as a longitudinal indicator:
with the intended interpretation that systems combining high nDCG and small RnD are more robust to temporal evolution (Deveaud et al., 2023).
Subsequent work reframed temporal persistence as a replicability problem. In that view, a later time slice functions as a changed evaluation environment, and persistence is assessed by comparing a system’s effect relative to a pivot baseline such as BM25. The adapted measures include Relative Improvement (RI), Delta Relative Improvement (DRI), and Effect Ratio (ER), with the key conclusion that the most effective systems are not necessarily the ones with the most persistent performance (Keller et al., 2024). This replicability framing was already explored in CLEF 2023 participation work, which showed that ARP-level stability can hide large per-topic volatility and that ER and RI reveal robustness properties missed by raw score deltas alone (Keller et al., 2023).
The empirical record reinforces the distinction between snapshot effectiveness and temporal robustness. Lessons from the 2024 lab, summarized in the 2025 call paper, reported high correlation between system rankings by nDCG at Lag6 and Lag8 but low correlation between rankings by Relative nDCG Drop, showing that systems preserving performance over time are not necessarily those with the highest snapshot effectiveness (Cancellieri et al., 11 Mar 2025). The completed 2025 overview reported similarly high rank correlations for WebRetrieval—Pearson 0.954 and Kendall 0.930 for the short-term comparison, Pearson 0.965 and Kendall 0.893 for the long-term comparison—while SciRetrieval showed weaker but still substantial cross-snapshot agreement, with Pearson 0.708 and Kendall 0.500 (Cancellieri et al., 22 Sep 2025).
LongEval 2025 also moved toward cross-snapshot comparability through empirical standardization of nDCG@10, producing Emp(nDCG@10) values standardized relative to the ensemble of participating systems in each snapshot (Cancellieri et al., 22 Sep 2025). That later overview emphasized that high effectiveness does not imply low temporal change, that RI, DRI, and ER do not always agree, and that systems exploiting historical qrels or query variants often display strong standardized stability even when raw effectiveness trends differ (Cancellieri et al., 22 Sep 2025).
A recurrent limitation is the nature of the judgments. WebRetrieval relies heavily on click-derived signals such as sDBN, while test sets may lack fresh relevance labels and instead reuse human annotations from earlier iterations or behavioral signals (Cancellieri et al., 11 Mar 2025). The lab explicitly notes that such signals may reflect user-behavior biases and that overlaps across snapshots complicate interpretation of observed changes (Cancellieri et al., 11 Mar 2025).
5. Retrieval methods studied on LongEval
The methodological landscape around LongEval is heterogeneous, but several recurrent patterns are clear. Multi-stage retrieval was common in 2024, typically using BM25 for first-stage retrieval followed by neural or LLM-based reranking (Cancellieri et al., 11 Mar 2025). Participant work for CLEF 2025 exemplifies this structure: the DS@GT team used Pyserini BM25 to retrieve the top 100 documents, optional LLM-based French query expansion, and a CamemBERT-based cross-encoder reranker; its best configuration, BM25 plus reranking, achieved an average NDCG@10 of 0.296 across the entire training and test dataset and reached 0.395 on the 2023-05 snapshot (Miyaguchi et al., 11 Jul 2025).
A major LongEval theme is the reuse of historical relevance feedback. “Counterfactual Query Rewriting to Use Historical Relevance Feedback” studied recurring queries in evolving Qwant snapshots and compared direct qrel-based boosting with two rewriting methods: tf-idf expansion from previously relevant documents and keyqueries that force historically relevant documents to the top of the current corpus. In that study, both rewriting methods substantially outperformed all baselines, including transformer models, on nDCG@10′, and keyqueries produced the best nDCG@10′ in most slices (Keller et al., 6 Feb 2025). The same paper also documented strong corpus evolution, reporting that across the period about 2.6 million documents were created and 1.7 million deleted, while many recurring URLs exhibited substantial content drift (Keller et al., 6 Feb 2025).
Neural reranking on LongEval has also been examined from the perspective of supervision depth. In experiments with MonoBERT trained on LongEval and MS MARCO, shallow-judged LongEval training sets offered broader query coverage, while deep-judged training sets offered many judgments for far fewer queries. On LongEval, BM25 remained difficult to beat at rank 10, and the results suggested that shallow judging was more data-efficient for temporal generalization, whereas deep judging required substantially more training instances and, plausibly, more negatives to mitigate overfitting (Iturra-Bocaz et al., 29 Jun 2025).
In SciRetrieval, later work emphasized full-text and temporal signals. A 2026 analysis of LongEval-Sci reported that full-text BM25 was the strongest single development retriever, while temporalized full-text runs were the strongest official submissions: FT BM25+temporal and FT BM25+temporal+citation achieved ARP values of 0.285, 0.267, and 0.180 across three snapshots and reduced snapshot-3 relative change from 0.481 for the BM25 pivot to 0.368 (Yang et al., 5 Jul 2026). That paper concluded that full-text retrieval was the strongest foundation, temporal integration improved longitudinal effectiveness when applied to that foundation, and citation features required cleaner ablation and calibration (Yang et al., 5 Jul 2026).
Some proposed methods are known chiefly through motivation rather than fully specified experiments. A CLEF 2024 working note on listwise reranking argued that LongEval exposes a gap between static knowledge-document evaluation and real-world temporal environments, and situated LongEval within broader work on positional invariance, Fusion-in-Decoder, and listwise rerankers such as ListT5, but did not itself provide a concrete experimental pipeline or results (Yoon et al., 2024).
6. Infrastructure, reproducibility, and scope expansion
LongEval has also driven tooling work for dynamic IR experimentation. A 2025 case study proposed a longitudinal extension of ir_datasets that models dynamic collections as hierarchies of snapshot datasets and exposes APIs such as get_datasets(), get_prior_datasets(), get_timestamp(), and document-level timestamps through docs_store(). The central idea is to replace bespoke, imperative snapshot logic with declarative dataset descriptions that encode time-aware availability and lineage directly in the dataset object (Keller et al., 22 Sep 2025).
This tooling matters because LongEval-style experimentation imposes constraints not present in static Cranfield evaluation: systems must index evolving corpora, exploit only relevance signals available at or before a snapshot’s timestamp, and be evaluated repeatedly across slices while avoiding future leakage (Keller et al., 22 Sep 2025). The same work notes that 2024 LongEval submissions were run under TIRA to ensure consistent environments and enforce time-causality, and argues that declarative access paths reduce code complexity, reproducibility problems, and maintenance overhead for shared-task submissions (Keller et al., 22 Sep 2025).
The lab’s dissemination infrastructure includes the LongEval website, dataset releases through repository integrations, and explicit support for software-oriented participation (Cancellieri et al., 11 Mar 2025, Cancellieri et al., 22 Sep 2025). The later 2025 overview reported 19 submitting teams and 68 runs across web and scientific retrieval, marking LongEval’s highest participation to date (Cancellieri et al., 22 Sep 2025).
LongEval’s scope has expanded beyond ad-hoc retrieval. In CLEF 2026, LongEval-RAG introduced a candidate-constrained retrieval-augmented generation task in which each query is associated with an organizer-provided candidate set and all retrieved evidence and citations must remain within that set. The reported system design enforced deterministic provenance tracking, passage-based retrieval, query expansion, PRF, RRF, lightweight reranking, and optional MiniLM sentence reranking; under the primary organizer evaluation, the strongest balanced submission was the rule-minilm variant (Yang et al., 4 Jul 2026). This later task is not a temporal web-search benchmark in the original sense, but it illustrates how the LongEval label has come to designate a broader CLEF evaluation ecosystem.
7. Other research artifacts named “LongEval”
Outside CLEF IR, the name has been reused for at least three distinct evaluation programs. In long-form summarization, “LongEval” denotes a set of guidelines for human evaluation of faithfulness. That work surveyed 162 long-form summarization papers, argued for clause-level rather than coarse summary-level judgments, and showed that switching to finer-grained judgments reduced inter-annotator standard deviation from 18.5 to 6.8 on SQuALITY; it also found that partial annotation of fine-grained units correlated strongly with full annotation, reaching Kendall’s of 0.89 with 50% judgments (Krishna et al., 2023).
In long-context language modeling, another work uses “LongEval” for a framework built around LongPPL and LongCE. Its core claim is that conventional perplexity averages over mostly short-context tokens and therefore fails to track true long-context capability. The proposed LongPPL focuses on key tokens identified through long-short context contrast, and the reported correlations between LongPPL and downstream long-context performance reached -0.96 on LongBench and -0.90 on LongEval-style long-context benchmarks, while LongCE fine-tuning yielded gains of up to +22% absolute accuracy on LongEval (Fang et al., 2024).
A further homonymous benchmark targets long-text generation through a plan-based paradigm. That LongEval evaluates arXiv papers, blogs, and Wikipedia articles under direct and section-wise plan-based generation, uses eight document- and section-level metrics, and reports that plan-based generation substantially outperforms direct generation on instruction following, length adherence, and redundancy control (Wu et al., 26 Feb 2025). The benchmark contains 166 human-authored documents across the three domains and is explicitly motivated by cognitive and linguistic models of writing (Wu et al., 26 Feb 2025).
These homonymous projects are unrelated in data and protocol to the CLEF IR lab, but they share a methodological orientation: evaluation should stress properties that standard short-horizon or single-shot setups tend to hide. In that limited sense, the repeated use of the name reflects a broader shift toward evaluation under temporal, contextual, or structural extension rather than static point estimates.