---
title: 'LongEval: Temporal & Extended Evaluation'
url: https://www.emergentmind.com/topics/longeval
type: topic
---

# LongEval: Temporal & Extended Evaluation

LongEval is a name used for several evaluation frameworks in contemporary NLP and IR, but it is most prominently associated with the CLEF lab on **longitudinal evaluation of IR systems**, where retrieval models are trained on earlier snapshots and tested on later, evolving collections in order to measure temporal persistence under document churn, query drift, and changing relevance signals [2503.08541]. The same name also appears in work on human evaluation of long-form summarization faithfulness, long-context language-model evaluation, and plan-based long-text generation, so the term is polysemous rather than designating a single benchmark family [2301.13298][2410.23771][2502.19103].

## 1. Multiple uses of the name

In current research usage, “LongEval” designates several distinct artifacts rather than a single canonical benchmark.

| LongEval usage | Domain | Representative paper |
|---|---|---|
| CLEF LongEval | Longitudinal IR evaluation | [2503.08541] |
| LongEval guidelines | Human evaluation of long-form summarization faithfulness | [2301.13298] |
| LongEval framework built on LongPPL/LongCE | Long-context language modeling | [2410.23771] |
| LongEval benchmark for plan-based generation | Long-text generation | [2502.19103] |

The CLEF usage is historically the most elaborate as a shared-task ecosystem. It studies how IR systems behave when the evaluation environment changes over time, rather than assuming a static Cranfield-style collection. That framing has generated a sequence of benchmark papers, lab overviews, participant reports, and methodological extensions, including web retrieval, scientific retrieval, classification, and later retrieval-augmented generation settings [2303.03229][2509.17469][2309.13562][2607.04008].

This multiplicity is itself a source of confusion. In IR, LongEval concerns **temporal persistence** under evolving corpora. In the other lines of work, the name refers to evaluation of long-form or long-context behavior. The shared label therefore signals a concern with longitudinal or extended-context evaluation, but not a shared dataset or metric stack.

## 2. LongEval as a longitudinal IR paradigm

The original IR benchmark, “LongEval-Retrieval,” defines a **dynamic test collection** composed of successive sub-collections, each representing the state of a web search environment at a given time step. In the formalization introduced for the 2023 web benchmark, a sub-collection at time $t$ is denoted $C_t=(D_t,Q_t,R_t)$, where topics remain fixed across time, while queries, documents, and relevance evolve [2303.03229].

The web benchmark was built from Qwant, a privacy-preserving web search engine with predominantly French traffic, and initially provided both a French “source” collection and an English “mirror” created through translation. Queries were extracted from logs through topic-based filtering; documents combined SERP-exposed pages with topic-conditioned random samples; and relevance assessments were derived from a simplified Dynamic Bayesian Network with session length $1$, equivalent to the Cascade Model. The click-based attractiveness proxy was estimated as
$$
\alpha_{q,d} = \frac{1}{|S_{q,d}|}\sum_{s\in S_{q,d}} c_s(d),
$$
and then mapped into graded labels $\{0,1,2\}$ for offline IR evaluation [2303.03229].

This design was intended to bridge offline and online evaluation. Rather than treating time as a nuisance variable, LongEval makes temporal divergence the object of study. The lab’s central claim is that conventional single-snapshot evaluation can obscure a basic fact of deployed IR: collections are updated continuously, users develop new information needs, and relevance perceptions change over time [2503.08541]. Earlier LongEval editions already showed that system rankings can vary across time and that the highest-scoring system on one snapshot is not necessarily the most temporally robust, so effectiveness and stability can diverge [2503.08541].

Although retrieval has been the core of the lab, the CLEF ecosystem was not limited to ad-hoc search. LongEval-Classification, for example, evaluated temporal robustness in sentiment analysis over timestamped Twitter data; a Bernice-based system using date-prefixed inputs and self-labeling ranked second with an overall score of 0.6923 and the best Relative Performance Drop of -0.0656 over the short evaluation set [2309.13562].

## 3. Tasks, datasets, and temporal splits in the CLEF lab

The 2025 edition was the third iteration of the lab and organized LongEval around two retrieval tasks sharing a single-training, evolving-test protocol: **LongEval-WebRetrieval** and **LongEval-SciRetrieval** [2503.08541]. In the web task, systems are trained on earlier Qwant data and evaluated on multiple later monthly snapshots. The call paper described a training set with 18 million French documents from June 2022 to February 2023, translated to English, together with 9,000 queries and click-model-derived relevance assessments, followed by seven monthly test lags from March 2023 to August 2023 [2503.08541]. A later overview reported post-processed 2025 WebRetrieval data with 19 million French documents and 119,341 queries for training, 14 million documents and 63,416 queries for test, and 33 million documents and 182,757 queries overall; it also noted that similar queries had been merged and identifiers unified, which improved continuity but made direct comparison with earlier releases unsafe despite overlapping periods [2509.17469].

The scientific task extended LongEval beyond web search into scholarly retrieval. Its data source is CORE, a large open-access aggregation of scholarly documents with logs capturing queries, returned results, and clicks. The task processes Apache server logs, removes bot traffic, and records anonymized session IDs, search queries, first-ten-result SERPs, clicked URLs, result positions, and timestamps [2503.08541]. In the later 2025 overview, the SciRetrieval training snapshot covered mid-November 2024 to mid-December 2024 and contained about 2 million documents, 393 queries, and 4,262 relevance assessments, while the test snapshot for January 2025 contained more than 1 million documents together with 492 new queries and 99 held-out queries from the training period [2509.17469].

The longitudinal structure is the defining feature. Training data always precede test data temporally; the time distance between datasets is the **lag**; and participants submit runs for each lag using the same trained system [2503.08541]. Retraining on test periods is not part of the core protocol in the 2025 call paper. In the web task, the lags are consecutive monthly snapshots, enabling measurement of degradation over increasing temporal distance. In the scientific task, the first year exposed fewer lags, but preserved the same principle of training on earlier data and evaluating on later snapshots [2503.08541].

Across editions, the lab has also had to manage substantial overlap across snapshots. Earlier releases revealed that large overlaps in documents and queries were difficult to analyze because of identifier conventions, and later iterations therefore improved document and query identifiers and collection descriptions to better quantify overlap and drift [2503.08541]. A later overview explicitly reported that consecutive monthly web snapshots shared the highest proportion of documents and that overlap declined across longer gaps, making document overlap itself a useful empirical indicator of temporal divergence [2509.17469].

## 4. Evaluation methodology, robustness measures, and empirical findings

LongEval’s basic effectiveness measure is **nDCG**, typically with emphasis on top ranks such as nDCG@10 for web search [2503.08541]. The early benchmark paper also introduced **Relative nDCG Drop (RnD)** as a longitudinal indicator:
$$
\mathrm{RnD} = \mathrm{nDCG}(\text{heldout, Train collection } t) - \mathrm{nDCG}(\text{test, Test collection at } t' \text{ or } t''),
$$
with the intended interpretation that systems combining high nDCG and small RnD are more robust to temporal evolution [2303.03229].

Subsequent work reframed temporal persistence as a **replicability** problem. In that view, a later time slice functions as a changed evaluation environment, and persistence is assessed by comparing a system’s effect relative to a pivot baseline such as BM25. The adapted measures include Relative Improvement (RI), Delta Relative Improvement (DRI), and Effect Ratio (ER), with the key conclusion that the most effective systems are not necessarily the ones with the most persistent performance [2409.05417]. This replicability framing was already explored in CLEF 2023 participation work, which showed that ARP-level stability can hide large per-topic volatility and that ER and $\Delta$RI reveal robustness properties missed by raw score deltas alone [2308.10549].

The empirical record reinforces the distinction between snapshot effectiveness and temporal robustness. Lessons from the 2024 lab, summarized in the 2025 call paper, reported high correlation between system rankings by nDCG at Lag6 and Lag8 but low correlation between rankings by Relative nDCG Drop, showing that systems preserving performance over time are not necessarily those with the highest snapshot effectiveness [2503.08541]. The completed 2025 overview reported similarly high rank correlations for WebRetrieval—Pearson 0.954 and Kendall 0.930 for the short-term comparison, Pearson 0.965 and Kendall 0.893 for the long-term comparison—while SciRetrieval showed weaker but still substantial cross-snapshot agreement, with Pearson 0.708 and Kendall 0.500 [2509.17469].

LongEval 2025 also moved toward cross-snapshot comparability through **empirical standardization** of nDCG@10, producing Emp(nDCG@10) values standardized relative to the ensemble of participating systems in each snapshot [2509.17469]. That later overview emphasized that high effectiveness does not imply low temporal change, that RI, DRI, and ER do not always agree, and that systems exploiting historical qrels or query variants often display strong standardized stability even when raw effectiveness trends differ [2509.17469].

A recurrent limitation is the nature of the judgments. WebRetrieval relies heavily on click-derived signals such as sDBN, while test sets may lack fresh relevance labels and instead reuse human annotations from earlier iterations or behavioral signals [2503.08541]. The lab explicitly notes that such signals may reflect user-behavior biases and that overlaps across snapshots complicate interpretation of observed changes [2503.08541].

## 5. Retrieval methods studied on LongEval

The methodological landscape around LongEval is heterogeneous, but several recurrent patterns are clear. Multi-stage retrieval was common in 2024, typically using BM25 for first-stage retrieval followed by neural or LLM-based reranking [2503.08541]. Participant work for CLEF 2025 exemplifies this structure: the DS@GT team used Pyserini BM25 to retrieve the top 100 documents, optional LLM-based French query expansion, and a CamemBERT-based cross-encoder reranker; its best configuration, BM25 plus reranking, achieved an average NDCG@10 of 0.296 across the entire training and test dataset and reached 0.395 on the 2023-05 snapshot [2507.08360].

A major LongEval theme is the reuse of **historical relevance feedback**. “Counterfactual Query Rewriting to Use Historical Relevance Feedback” studied recurring queries in evolving Qwant snapshots and compared direct qrel-based boosting with two rewriting methods: tf-idf expansion from previously relevant documents and keyqueries that force historically relevant documents to the top of the current corpus. In that study, both rewriting methods substantially outperformed all baselines, including transformer models, on nDCG@10′, and keyqueries produced the best nDCG@10′ in most slices [2502.03891]. The same paper also documented strong corpus evolution, reporting that across the period about 2.6 million documents were created and 1.7 million deleted, while many recurring URLs exhibited substantial content drift [2502.03891].

Neural reranking on LongEval has also been examined from the perspective of supervision depth. In experiments with MonoBERT trained on LongEval and MS MARCO, shallow-judged LongEval training sets offered broader query coverage, while deep-judged training sets offered many judgments for far fewer queries. On LongEval, BM25 remained difficult to beat at rank 10, and the results suggested that shallow judging was more data-efficient for temporal generalization, whereas deep judging required substantially more training instances and, plausibly, more negatives to mitigate overfitting [2506.23191].

In SciRetrieval, later work emphasized **full-text** and **temporal** signals. A 2026 analysis of LongEval-Sci reported that full-text BM25 was the strongest single development retriever, while temporalized full-text runs were the strongest official submissions: FT BM25+temporal and FT BM25+temporal+citation achieved ARP values of 0.285, 0.267, and 0.180 across three snapshots and reduced snapshot-3 relative change from 0.481 for the BM25 pivot to 0.368 [2607.04088]. That paper concluded that full-text retrieval was the strongest foundation, temporal integration improved longitudinal effectiveness when applied to that foundation, and citation features required cleaner ablation and calibration [2607.04088].

Some proposed methods are known chiefly through motivation rather than fully specified experiments. A CLEF 2024 working note on listwise reranking argued that LongEval exposes a gap between static knowledge-document evaluation and real-world temporal environments, and situated LongEval within broader work on positional invariance, Fusion-in-Decoder, and listwise rerankers such as ListT5, but did not itself provide a concrete experimental pipeline or results [2407.06716].

## 6. Infrastructure, reproducibility, and scope expansion

LongEval has also driven tooling work for dynamic IR experimentation. A 2025 case study proposed a longitudinal extension of `ir_datasets` that models dynamic collections as hierarchies of snapshot datasets and exposes APIs such as `get_datasets()`, `get_prior_datasets()`, `get_timestamp()`, and document-level timestamps through `docs_store()`. The central idea is to replace bespoke, imperative snapshot logic with declarative dataset descriptions that encode time-aware availability and lineage directly in the dataset object [2509.17440].

This tooling matters because LongEval-style experimentation imposes constraints not present in static Cranfield evaluation: systems must index evolving corpora, exploit only relevance signals available at or before a snapshot’s timestamp, and be evaluated repeatedly across slices while avoiding future leakage [2509.17440]. The same work notes that 2024 LongEval submissions were run under TIRA to ensure consistent environments and enforce time-causality, and argues that declarative access paths reduce code complexity, reproducibility problems, and maintenance overhead for shared-task submissions [2509.17440].

The lab’s dissemination infrastructure includes the LongEval website, dataset releases through repository integrations, and explicit support for software-oriented participation [2503.08541][2509.17469]. The later 2025 overview reported 19 submitting teams and 68 runs across web and scientific retrieval, marking LongEval’s highest participation to date [2509.17469].

LongEval’s scope has expanded beyond ad-hoc retrieval. In CLEF 2026, **LongEval-RAG** introduced a candidate-constrained retrieval-augmented generation task in which each query is associated with an organizer-provided candidate set and all retrieved evidence and citations must remain within that set. The reported system design enforced deterministic provenance tracking, passage-based retrieval, query expansion, PRF, RRF, lightweight reranking, and optional MiniLM sentence reranking; under the primary organizer evaluation, the strongest balanced submission was the rule-minilm variant [2607.04008]. This later task is not a temporal web-search benchmark in the original sense, but it illustrates how the LongEval label has come to designate a broader CLEF evaluation ecosystem.

## 7. Other research artifacts named “LongEval”

Outside CLEF IR, the name has been reused for at least three distinct evaluation programs. In long-form summarization, “LongEval” denotes a set of **guidelines for human evaluation of faithfulness**. That work surveyed 162 long-form summarization papers, argued for clause-level rather than coarse summary-level judgments, and showed that switching to finer-grained judgments reduced inter-annotator standard deviation from 18.5 to 6.8 on SQuALITY; it also found that partial annotation of fine-grained units correlated strongly with full annotation, reaching Kendall’s $\tau$ of 0.89 with 50% judgments [2301.13298].

In long-context language modeling, another work uses “LongEval” for a framework built around **LongPPL** and **LongCE**. Its core claim is that conventional perplexity averages over mostly short-context tokens and therefore fails to track true long-context capability. The proposed LongPPL focuses on key tokens identified through long-short context contrast, and the reported correlations between LongPPL and downstream long-context performance reached -0.96 on LongBench and -0.90 on LongEval-style long-context benchmarks, while LongCE fine-tuning yielded gains of up to +22% absolute accuracy on LongEval [2410.23771].

A further homonymous benchmark targets **long-text generation through a plan-based paradigm**. That LongEval evaluates arXiv papers, blogs, and Wikipedia articles under direct and section-wise plan-based generation, uses eight document- and section-level metrics, and reports that plan-based generation substantially outperforms direct generation on instruction following, length adherence, and redundancy control [2502.19103]. The benchmark contains 166 human-authored documents across the three domains and is explicitly motivated by cognitive and linguistic models of writing [2502.19103].

These homonymous projects are unrelated in data and protocol to the CLEF IR lab, but they share a methodological orientation: evaluation should stress properties that standard short-horizon or single-shot setups tend to hide. In that limited sense, the repeated use of the name reflects a broader shift toward evaluation under temporal, contextual, or structural extension rather than static point estimates.

Source: https://www.emergentmind.com/topics/longeval