LongEval: Longitudinal Evaluation of IR Systems
- LongEval is a framework for evaluating IR systems over time with temporally ordered snapshots that capture evolving documents, queries, and relevance judgments.
- It employs metrics like nDCG and Relative nDCG Drop to assess both absolute effectiveness and the robustness of systems to temporal drift.
- The framework is applied to both web and scientific retrieval settings, highlighting challenges in replicability and the impact of data drift on evaluation.
Longitudinal Evaluation of Model Performance, commonly abbreviated LongEval, is a benchmark and shared-task framework for evaluating information retrieval systems under temporal change rather than on a single static test collection. Its central premise is that retrieval systems operate in environments where documents are created, updated, or removed, queries evolve, and relevance judgments shift over time; evaluation must therefore measure not only pointwise effectiveness but also the persistence of that effectiveness as the test environment diverges temporally from training data. In CLEF, LongEval operationalizes this idea through temporally ordered snapshots, or lags, with systems trained on earlier snapshots and evaluated on later ones in web and scientific retrieval settings (Cancellieri et al., 11 Mar 2025).
1. Conceptual basis
LongEval is grounded in the observation that standard Cranfield-style evaluation freezes the corpus, topics, and relevance assessments, while deployed retrieval systems face a moving target. In the LongEval formulation, documents, query distributions, and relevance all evolve, so a system that performs well at time may degrade at later times or . The framework distinguishes short-term persistence from long-term persistence, with the latter corresponding to larger temporal gaps between development and evaluation (Deveaud et al., 2023).
The lab’s core question is therefore not merely which system is best at one date, but which system remains effective as the environment changes. CLEF 2025 describes this as the study of temporal persistence, temporal robustness, and temporal generalisability. The test protocol evaluates a system over several later snapshots and examines how retrieval quality changes across lags. In this setting, absolute retrieval quality and robustness to temporal drift are separate properties: a method can be strong on raw effectiveness yet less stable over time, or vice versa (Cancellieri et al., 11 Mar 2025).
This design addresses a recurrent misconception in offline IR evaluation: that a single held-out test set is sufficient to characterize future deployment behavior. LongEval instead treats temporal distance itself as an evaluation variable. A plausible implication is that leaderboard positions obtained on static benchmarks can overstate a system’s reliability in dynamic search environments.
2. Development of the benchmark
The retrieval benchmark underlying LongEval was introduced as LongEval-Retrieval, a dynamic test collection derived from the Qwant search engine. Rather than a single collection, it is organized as a sequence of temporally indexed sub-collections, each containing queries, documents, and soft relevance assessments built from click models. The initial 2023 benchmark used a June 2022 training collection, a July 2022 short-term test collection, and a September 2022 long-term test collection, thereby embedding temporal progression directly into offline evaluation (Deveaud et al., 2023).
A distinctive feature of LongEval-Retrieval is that it keeps a common set of topics across sub-collections while allowing the actual logged queries, document inventory, and relevance estimates to vary. The 2023 benchmark also released a French-English mirror collection: the resource was constructed in French to exploit Qwant traffic volume and then translated into English, making the benchmark usable for both French and English retrieval experiments (Deveaud et al., 2023).
By 2025, LongEval had become the third edition of the CLEF lab and expanded beyond web retrieval into a second domain, scientific retrieval. The 2025 overview paper states that the web collection was reprocessed to combine similar queries and unify identifiers, so direct year-to-year score comparison with earlier releases is not valid despite overlapping temporal coverage. The same overview also emphasizes that LongEval is no longer a web-only evaluation setting, but a broader laboratory for longitudinal IR in both web and scholarly search (Cancellieri et al., 22 Sep 2025).
3. Tasks, datasets, and temporal structure
LongEval 2025 comprises two tasks, both based on temporally evolving snapshots and both evaluated with a single system trained only on the historical training portion.
| Task | Source and temporal structure | Scale reported for 2025 |
|---|---|---|
| WebRetrieval | Qwant; train on June 2022–February 2023, test on monthly snapshots March 2023–August 2023 | 19 million training documents, 119,341 training queries; 14 million test documents, 63,416 test queries; 33 million documents and 182,757 queries total |
| SciRetrieval | CORE; first snapshot mid-November 2024–mid-December 2024, second snapshot January 2025 | about 2 million training documents, 393 training queries, 4,262 training relevance assessments; over 1 million test documents; 492 new test queries plus 99 held-out queries |
The web task continues the original Qwant-based scenario. Earlier task documentation described the 2025 web training set as covering June 2022 to February 2023 with 18 million French documents, 9,000 queries, and test data from March 2023 to August 2023, with one snapshot per month; the later overview reports the reprocessed larger query counts and explicitly notes changed identifiers and query merging (Cancellieri et al., 11 Mar 2025).
The scientific task, introduced in 2025, uses CORE search logs and click-derived supervision. Its two test sets serve distinct functions: the 99 held-out queries from the first snapshot provide a within-time comparison point, while the 492 new queries from January 2025 assess generalization to later information needs. This yields a smaller but structurally similar longitudinal setup to the web task (Cancellieri et al., 22 Sep 2025).
The underlying collection design follows the broader dynamic-test-collection principle first articulated for LongEval-Retrieval: each snapshot can be interpreted as a time-indexed triple of queries, documents, and relevance assessments, with changes in all three components across time (Deveaud et al., 2023).
4. Metrics and notions of persistence
The basic LongEval measures are nDCG scores on each lag and Relative nDCG Drop (RnD), which quantifies how much effectiveness changes between lags. The 2025 task description presents LongEval as measuring retrieval quality at each time point and robustness to temporal drift across time points; in this setting, high nDCG indicates effectiveness, whereas low RnD indicates temporal robustness (Cancellieri et al., 11 Mar 2025).
The 2025 overview extends this by reporting both ordinary and empirically standardized effectiveness and by analyzing rank stability across snapshots with Pearson correlation and Kendall’s Tau. For WebRetrieval, system ordering remained highly stable across later snapshots, with Pearson $0.954$ and Kendall’s tau $0.930$ for 2023-03 versus 2023-05, and Pearson $0.965$ and Kendall’s tau $0.893$ for 2023-03 versus 2023-08. SciRetrieval showed lower stability, with Pearson $0.708$ and Kendall’s tau $0.500$ between 2024-11 and 2025-01; on standardized scores these rose to $0.848$ and 0, respectively (Cancellieri et al., 22 Sep 2025).
A related methodological line treats longitudinal IR evaluation as a replicability problem rather than only a sequence of effectiveness measurements. In that view, the LongEval shared-task measure Result Delta is
1
where 2 is the system, 3 an effectiveness metric, and 4 and 5 the original and evolved evaluation environments. To factor out environmental change, the same work introduces pivot-relative measures against a baseline system 6, typically BM25:
7
with
8
It also defines Effect Ratio (ER) through topic-wise gains over the pivot:
9
0
Using the 124 core queries present in WT, ST, and LT, that study found that the most effective systems are not necessarily the most persistent, and that rankings vary across both time and retrieval metric (Keller et al., 2024).
5. Systems, empirical findings, and software infrastructure
Across LongEval editions, strong systems have typically been multi-stage retrieval pipelines. The 2024 web task summary reports that the strongest approaches combined BM25 first-stage retrieval with neural-based or LLM-based reranking. It also notes that some teams explicitly used temporal signals, including past relevance information for query reformulation, and that rankings based on raw nDCG at Lag6 and Lag8 were highly correlated whereas rankings based on RnD were only weakly correlated, reinforcing the distinction between effectiveness and robustness (Cancellieri et al., 11 Mar 2025).
The 2025 results preserve that pattern. In the WebRetrieval task, top later-snapshot systems included RISE, baseline-qrel-boost, and RACOON run1; on the 2023-08 snapshot, their nDCG@10 values were 1, 2, and 3, respectively. In SciRetrieval, by contrast, the best system changed across snapshots: OpenWebSearch variants led on the 2024-11 within-time set, while Academy Retrievals BM25 variants led on the 2025-01 future snapshot. The same overview notes that many strong web systems exploited historical qrels or prior snapshots, but no team trained or fine-tuned a retrieval model directly on prior snapshots (Cancellieri et al., 22 Sep 2025).
A representative lab notebook, "DS@GT at LongEval: Evaluating Temporal Performance in Web Search Systems and Topics with Two-Stage Retrieval" (Miyaguchi et al., 11 Jul 2025), illustrates the behavior of a practical two-stage pipeline across monthly Qwant snapshots. Its best system, BM25 followed by a French cross-encoder reranker, achieved mean nDCG@10 4 across training and test months and a best monthly score of 5 on 2023-05. The same notebook reports that reranking consistently improved BM25, while the particular LLM-based query expansion strategy used there reduced mean performance when applied without reranking. The month-by-month scores also exposed a clear temporal regime shift: all systems were markedly weaker in the earliest months and stronger from approximately 2022-09 or 2022-10 onward (Miyaguchi et al., 11 Jul 2025).
LongEval has also generated work on reproducible longitudinal experimentation infrastructure. "Simplified Longitudinal Retrieval Experiments: A Case Study on Query Expansion and Document Boosting" (Keller et al., 22 Sep 2025) extends ir_datasets with temporal abstractions such as get_datasets(), get_prior_datasets(), get_timestamp(), and get_snapshot(), representing a dynamic collection as a meta-dataset over snapshots. The paper re-implements LongEval 2024 submissions using this interface and reports reduced code complexity, including a drop in total NLOC for qrel_boost from 250 to 99. Its central methodological point is that temporally valid access to prior snapshots should be encoded in the dataset interface rather than left to ad hoc scripting.
6. Methodological significance and open problems
LongEval’s importance extends beyond its role as a CLEF lab. Related work in longitudinal NLP argues that once data become person-indexed and time-ordered, evaluation must move away from IID document assumptions toward explicit generalization over people and time, with splits aligned to cross-sectional and prospective questions and with metrics that separate between-person from within-person signal (Ganesan et al., 12 Jan 2026). Although this work is not part of the LongEval lab, it supports the same underlying principle: longitudinal validity depends on matching the evaluation protocol to the temporal structure of the data.
A similar point appears in longitudinal clinical-record evaluation. "TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records" (Cui et al., 6 Mar 2025) defines benchmark instances as instruction-response pairs grounded in explicit timestamps and varies the temporal distribution of evidence across a patient timeline. Its results show that temporally balanced supervision improves performance on both a human-generated benchmark and a synthetic time-aware benchmark, suggesting that longitudinal evaluation should not collapse all time positions into a single undifferentiated test set. This suggests an analogous design principle for LongEval: temporal location and temporal distance should be treated as first-class benchmark dimensions.
Within LongEval itself, several open problems remain explicit in the published analyses. Click-derived relevance judgments are scalable but inherit limitations from user behavior and presentation bias. Overlap across snapshots complicates interpretation, because part of the temporal signal may come from repeated content rather than substantive drift. The new SciRetrieval task currently has only two snapshots, limiting fine-grained temporal analysis. The 2025 overview also notes that different robustness measures can disagree and that, despite the lab’s focus, participating systems have generally optimized for effectiveness rather than explicitly for temporal stability (Cancellieri et al., 22 Sep 2025).
The broader implication is that LongEval is not merely a benchmark suite but a methodological agenda for time-aware evaluation. Its defining contribution is to treat effectiveness over time as a primary object of measurement. In that formulation, a retrieval system is characterized not only by how well it ranks at one snapshot, but by how its performance profile evolves as documents, queries, and relevance move forward in time.