Submitted and Diagnostic Analysis of Full-Text Temporal Retrieval for LongEval-Sci

Abstract: LongEval-Sci evaluates scientific retrieval under collection change, where a system should be effective on the current corpus and remain usable as documents accumulate over time. This paper reports both official Task 1 results and development diagnostics for LongEval-Sci 2026. We compare the official PyTerrier BM25 and Qwen3 dense baselines with full-text BM25, additive and router variants, temporal full-text retrieval, temporal+citation retrieval, RM3 query expansion, cross-encoder reranking, and reciprocal rank fusion (RRF). In the official DCTR evaluation, the temporalized full-text runs are our strongest submissions: FT BM25+temporal and FT BM25+temporal+citation obtain the best ARP on all three snapshots (0.285, 0.267, and 0.180 nDCG@10) and reduce snapshot-3 relative change from 0.481 for the BM25 pivot to 0.368. Citation features match the temporal-only variant but do not provide a measurable additional gain in the official summary. Our internal snapshot-1 diagnostics show a complementary pattern: full-text BM25 is the strongest single development retriever (DCTR nDCG@10 = 0.3302, MAP = 0.2853), RRF gives the best deep recall (Recall@1000 = 0.9667), and some uncalibrated overlays can sharply degrade top-rank quality. We therefore conclude that full-text retrieval is the strongest foundation, temporal integration can improve official longitudinal effectiveness when applied to that foundation, and citation evidence still requires cleaner ablation and calibration. Beyond ranking, we also report a qualitative weekly IR-system update-monitoring analysis based on ingestion velocity and stale-coverage drift.

• The paper demonstrates that full-text BM25 with temporal overlays yields sustained retrieval effectiveness across document snapshots.
• It employs a cascade architecture integrating lexical, dense, and fusion methods with temporal and citation-aware reranking techniques.
• Empirical results show that proper calibration and update policies are critical for maintaining robust performance under collection drift.

Analysis of Full-Text Temporal Retrieval for LongEval-Sci

Longitudinal Scientific Retrieval: Task Challenges and Methodological Framework

LongEval-Sci 2026 targets the longitudinal robustness of scientific information retrieval systems as collections experience ongoing document accumulation and metadata evolution. The paper formally evaluates submitted systems under the CLEF LongEval-Sci Task 1, focusing on explicit corpus snapshots and public qrels with document click-through rates (DCTR).

The evaluation follows a cascade architecture, separating retrieval foundations from temporal and citation-aware reranking overlays. Lexical (BM25), dense (Qwen3 embeddings), and fusion (RRF) systems establish candidate pools, upon which temporal and citation signals are optionally applied. Notably, full-text BM25 over the entire document is contrasted with compact title+abstract views to determine field-dependent retrieval effectiveness.

Empirical Results: Temporalized Full-Text Retrieval and Diagnostic Insights

The official longitudinal DCTR evaluation across three snapshots demonstrates that temporalized full-text BM25 submissions, with and without citation overlays, exhibit superior effectiveness and robustness. FT BM25+temporal and FT BM25+temporal+citation both achieve ARP (nDCG@10) scores of 0.285, 0.267, and 0.180 for snapshots 1–3 and reduce snapshot-3 relative degradation (RC) to 0.368, outperforming the baseline BM25 (RC=0.481). The temporal+citation variant does not further improve these metrics, indicating citation evidence as marginal under current aggregation.

The snapshot-1 development diagnostics reveal that BM25 retrieval over full text yields higher MAP, nDCG@10, and recall compared to title+abstract baselines or dense retrieval. RRF fusion combining FT BM25 with dense retrieval maximizes Recall@1000 (0.9667), underscoring its utility as a candidate generator. Local temporal/citation overlays, when naively calibrated, can severely degrade top-rank quality, highlighting the importance of feature weight normalization and per-query intent adaptation.

Architectural and Maintenance Considerations

Overlay designs compute temporal features from document publication dates, tailoring recency and foundational weights via a query-intent classifier. Citation overlays use OpenCitations-derived graphs for inbound/outbound velocity modeling, but sparse coverage and incomplete timestamps limit their reliability.

Index update monitoring policies are proposed based on ingestion velocity and stale coverage drift signals, implementing burst and structural triggers. The operational policy favors ingestion velocity as the primary update determinant, with secondary checks for indexed collection staleness and coverage gap.

Figure 1: Selected weekly system update timeline, demonstrating concentration of retrieval refreshes in the intermediate period (2/5/1 across snapshots).

Practical and Theoretical Implications

The results decisively establish full-text lexical retrieval as the preferable baseline for longitudinal scientific IR, with temporal integration providing measurable robustness gains in the official Task 1 protocol. Field choice (full-text vs. title+abstract) significantly affects absolute and relative retrieval performance, supporting broader adoption of FT BM25 as a first-stage retriever when full text is accessible.

Fusion strategies such as RRF facilitate robust candidate generation without imposing score normalization or expensive reindexing, especially for deep recall tasks. Cross-encoder reranking, while improving precision in certain cases, cannot compensate for inadequacies in the candidate pool.

Temporal overlays require careful calibration; the official pipeline demonstrates benefit, but local diagnostic overlays show substantial risk in misweighting. Citation evidence, though not harmful, needs more rigorous ablation and coverage audits to establish its incremental value.

Maintenance triggers substantially inform practical deployment: operational refresh decisions should rely on collection monitoring rather than direct effectiveness metrics, separating maintenance from model selection and ranking logic.

Limitations and Future Directions

The official evaluation exposes limitations in aggregate citation modeling and overlay calibration. MAP and recall are reported only for internal diagnostics, restricting generality. The citation overlay needs finer-grained timestamp coverage and per-query analysis. The current monthly protocol and judgement scope are best suited for development diagnostics, not longitudinal claims.

Future work should develop citation-only and temporal-only ablations, conduct paired significance testing, improve overlay calibration, and robustly package run artifacts and evaluation scripts for reproducibility. Explicit correlation between update triggers and downstream effectiveness should be substantiated.

Conclusion

This analysis confirms that temporal retrieval overlays applied to full-text BM25 pipelines achieve sustained retrieval effectiveness and robustness under collection drift, outperforming both sparse and dense baselines in longitudinal scientific IR. RRF fusion is recommended for deep recall optimization. Caution is warranted in overlay calibration, especially for sparse systems. Citation evidence remains inconclusive pending further audit. Maintenance decisions should be guided by ingestion velocity and stale coverage signals, orthogonal to ranking logic.