ArxivRoll: Dynamic LLM Evaluation Framework

• ArxivRoll is a dynamic evaluation framework that creates one-time private benchmarks from recent arXiv papers to minimize overfitting and contamination.
• The framework uses the SCP automated test-case generator to produce Sequencing, Cloze, and Prediction tasks, ensuring robust and diverse evaluation.
• It introduces Rugged Scores to quantify public benchmark leakage and biased overtraining, thereby enhancing assessment transparency and reproducibility.

ArxivRoll is a dynamic evaluation framework for LLMs that constructs private benchmark sets from recent arXiv papers and uses them exactly once, following an explicit one-time-pad analogy drawn from cryptography. In its core formulation, ArxivRoll combines an automated test-case generator, termed Sequencing, Cloze, and Prediction (SCP), with “Rugged Scores” (RS) that quantify public-benchmark contamination and biased overtraining. The framework is designed to preserve freshness, limit leakage, and retain transparency by releasing each benchmark only after its single evaluation round has expired. In adjacent proposals, the name ArxivRoll is also associated with ranking and curation mechanisms for arXiv-mediated content, including mitigation of positional bias in daily announcements and adaptation of living-review pipelines for continuously updated scholarly corpora (Liang et al., 25 Jul 2025).

1. Conceptual basis and system scope

ArxivRoll is explicitly motivated by the observation that evaluation on public LLM benchmarks can be overestimated because of contamination of public test sets or imbalanced model training. Its central analogy is to the One-Time Pad: a private benchmark is generated anew from fresh arXiv material, remains secret before evaluation, is used for exactly one evaluation round, and is then revealed but marked unusable for future training or evaluation. The benchmark rotation period is six months, and the intention is to prevent both memorization and overfitting to held-out data (Liang et al., 25 Jul 2025).

Within this design, ArxivRoll has two principal components. The first is SCP, an automated generator of private test cases. The second is Rugged Scores, a family of metrics intended to measure overestimation. The framework therefore combines benchmark construction with post hoc diagnosis of evaluation distortion, rather than treating secrecy of test data as sufficient in itself.

A recurring issue in LLM evaluation is the trade-off among secrecy, reproducibility, and operational efficiency. ArxivRoll addresses this by separating the benchmark lifecycle into a private phase and an archived phase. The benchmark remains sealed during evaluation, then becomes a public “expired” artifact afterward. This suggests a model in which reproducibility is preserved through archival release, while contamination is constrained through non-reuse.

2. SCP benchmark construction

SCP produces private benchmarks from arXiv uploads in the last six months across eight domains: CS, Math, Physics, Statistics, Biology, Economics, Quantitative Finance, and EE. The pipeline splits each paper at newlines, filters out fragments dominated by formulas or tables, and randomly samples text fragments of at least $N_f$ words. In practice, $N_f=80$ and $N=1$ fragment is drawn per paper. Fragments shorter than $N_f$ are discarded, near-duplicates are removed via simple heuristics, and two human annotators spot-check a small fraction to ensure quality (Liang et al., 25 Jul 2025).

The test cases are then instantiated in three formats. In Sequencing, a fragment is divided into four non-overlapping chunks, these chunks are permuted at random, and the model is prompted to select the correct order. In Cloze, four sentences are randomly masked in place and exactly those four masked sentences are supplied as multiple-choice candidates. In Prediction, the last sentence of the fragment is removed and three distractors are retrieved from elsewhere in the same article by TF–IDF similarity.

The final assembly repeats this process across sampled fragments, producing approximately $2$–$3$K samples per domain per format. Over a six-month cycle, ArxivRoll generates approximately $20$K total samples and packages them into ArxivRollBench. The benchmark is then sealed until an announced evaluation window, exemplified in the specification as one month.

These choices make the benchmark generation procedure highly automated while preserving direct dependence on recent scholarly text. Because the source distribution advances with arXiv itself, the benchmark is coupled to the current research frontier rather than to a fixed historical corpus.

3. Rugged Scores and evaluation protocol

ArxivRoll’s RS metrics are intended to separate two distinct failure modes. The contamination score, $\mathrm{RS_I}$, measures the gap between model performance on public benchmarks and matched or unmatched private benchmarks in the same domain. High $\mathrm{RS_I}$ is interpreted as evidence that a model is effectively “cheating” on public benchmarks in the sense that public-set performance substantially exceeds private-set performance. The paper further states that when the performance measure $\mathcal{M}$ is relative, such as rank among models, $N_f=80$0 becomes benchmark-independent across evaluation rounds; when $N_f=80$1 is absolute accuracy, $N_f=80$2 is model-independent under the same benchmarks (Liang et al., 25 Jul 2025).

The second family, $N_f=80$3 and its normalized form $N_f=80$4, measures unevenness across private domains. The normalized variant is defined as

$N_f=80$5

where $N_f=80$6 is the mean private-benchmark performance over the relevant domain set. High $N_f=80$7 indicates that a model is unevenly overtrained in some private domains at the expense of others.

The evaluation protocol includes both open-source and closed-API models. The open-source set includes GPT-J-6B, Phi-1, Phi-1.5, Phi-2, Phi-3-Mini, Phi-3.5-Mini, Phi-4, Llama2-7B-Chat, Llama3 variants, Qwen2-7B, Qwen2.5 variants, Qwen3 variants, Yi-1.5-34B, Kimi-K2, and Deepseek-Chat-V3. The closed-API set includes GPT-3.5-turbo, GPT-4, GPT-4o, Claude-3.5, Claude-3.7, Claude-4, Gemini-2.0, and Gemini-2.5. Public baselines are MMLU and MMLU-Pro. Open-source models are evaluated through LM Evaluation Harness with greedy decoding and a maximum of 50 tokens, and exact-match accuracy is used for Sequencing, Cloze, and Prediction under identical settings for public and private benchmarks.

A common misconception is that private benchmarking alone resolves comparability problems. ArxivRoll’s formalization implies a narrower claim: secrecy reduces leakage, but fair comparison also requires explicit quantification of public–private gaps and cross-domain unevenness.

4. Empirical results and overestimation patterns

The reported experiments emphasize stability, external validity, and systematic overestimation. For SCP stability, generating ArxivRollBench CS_Sequencing 32 times with different random seeds yields Llama3-8B accuracies with standard deviation below 1 point out of 100, which is presented as evidence of reproducibility. For external correspondence, Spearman correlations with ChatbotArena are 0.76 for Sequencing, 0.61 for Cloze, and 0.73 for Prediction, while internal correlations among the three SCP formats are reported as Spearman 0.86 or greater (Liang et al., 25 Jul 2025).

On private-benchmark performance, the Sequencing task illustrates the score range: open-source Qwen2-7B-Instruct reaches 27.6% on CS, Kimi-K2 reaches 35.7%, and GPT-4 reaches 42.9%. Cloze and Prediction are described as showing similar model ordering but uniformly lower absolute scores, indicating higher task difficulty.

The contamination results are more diagnostic. Qwen2.5-72B has the largest absolute $N_f=80$8, indicating a public-over-private gap greater than 100%. Phi-1 is also reported with high $N_f=80$9. The study further states that many newer models, especially within the Phi and Qwen series, display growing $N=1$0 values as they evolve.

For biased overtraining, Llama3.1-Nemotron-70B and Llama3.1-70B have the highest $N=1$1, although their normalized unevenness is less dramatic because of higher overall accuracies. Domain-radar plots are described as showing that models can appear artificially strong on public-set domains such as Economics, Quantitative Finance, Biology, and Physics, whereas private-set performance is more uniform.

Taken together, these results frame ArxivRoll not merely as a freshness mechanism, but as an instrument for measuring how much benchmark exposure and training imbalance inflate apparent capability.

5. Positional bias, ranking, and recommender design

A separate line of work relevant to ArxivRoll concerns the effect of article position in arXiv’s daily announcements on later readership and citation impact. In arXiv subcommunities studied over 2002–2004, articles appearing in position 1 received substantial citation and download boosts, and these gains were decomposed into intentional “self-promotion” and accidental “visibility” effects. For position 1, citation boosts were reported as +83% overall and +44% accidental for astro-ph, +50% overall and +38% accidental for hep-th, and +100% overall and +71% accidental for hep-ph. Early-readership boosts at position 1 were +82% overall and +53% accidental for astro-ph, +61% overall and +44% accidental for hep-th, and +58% overall and +46% accidental for hep-ph (0907.4740).

The analysis uses median-based statistics because citation and download counts are treated as heavy-tailed, with

$N=1$2

Position-based effects are summarized through

$N=1$3

where the baseline median is taken from positions 10–40. Differences in medians are tested with Mann–Whitney $N=1$4, and stochastic dominance is assessed with a one-sided Kolmogorov–Smirnov test.

For ArxivRoll ranking and interface design, the proposed response is a hybrid scoring rule

$N=1$5

where $N=1$6 is a recency score, $N=1$7 a quality score such as an early-readership signal, and $N=1$8 a personalization score. Randomized tie-breakers $N=1$9 are then added so that ranking is proportional to $N_f$0. Interface strategies include “Top-N randomized,” personalized feeds keyed to keywords or past-read profiles, and “Boost badges” that mark early high-quality self-promoted work without assigning it position 1.

The underlying concern is that recommender loops can amplify popularity artificially. A fair aggregator, in this view, should distinguish intentional self-promotion, which may carry genuine quality information, from accidental visibility effects, which are purely positional.

6. Continuous curation and living-review adaptation

ArxivRoll has also been discussed as a possible target for adaptation of a living-review pipeline developed for AI/ML applications in accelerator physics. That pipeline automatically harvests publications from arXiv, InspireHEP, HAL, OpenAlex, Crossref, and Springer, deduplicates records, semantically filters them, classifies them into thematic categories, and exports the curated corpus in JSON, HTML, PDF, and Bib\TeX\ formats. The authors state that the description is sufficient to reproduce or adapt the pipeline for ArxivRoll or similar living-review applications (Ghribi, 10 Oct 2025).

The architecture comprises six major components: Data Harvesting, Deduplication, Semantic Filtering, Thematic Classification, Output and Integration, and System Evaluation. Each source is handled by a fetcher module with a uniform interface, query parameters tuned to domain keywords and date ranges, pagination by cursor or offset, and rate-limit compliance such as 1 request per second for open APIs. Deduplication merges records first by DOI, then by arXiv ID, then by fuzzy title matching using a Levenshtein ratio threshold below 0.1, with preference for non-empty abstracts, unioned author lists, earliest dates, and aggregated URLs.

Semantic filtering uses the embedding model sentence-transformers/all-MiniLM-L6-v2, with output dimension $N_f$1, on concatenated title and abstract. Cosine similarity is computed against accelerator, ML, and noise anchors, and a paper is retained when

$N_f$2

with $N_f$3 and $N_f$4. Negative-keyword exclusions include terms such as “Higgs,” “dark matter,” “FPGA accelerator,” “beam welding,” and “earthquake.” Threshold calibration on a manually annotated sample of approximately 500 candidate papers selected operating points near 95% precision and 40% recall on validation; on a held-out month, measured precision is approximately 0.92 and recall approximately 0.45.

The system processes approximately 12,000 candidate papers per month and retains about 2%, or roughly 240 papers. Deduplication and embedding of 12k papers take 4–6 minutes on a 16-core CPU node, while GPU embedding on a Tesla T4 is reported at approximately 60 ms per document, yielding a full pass in about 12 seconds. Static-site integration uses Hugo, optional live queries can be served through Flask or FastAPI, and automated runs can be scheduled daily or monthly via cron, GitHub Actions, or Jenkins.

For ArxivRoll, this living-review pattern suggests an extensible outer layer around dynamic benchmarking: harvesting recent literature, maintaining a filtered corpus, and exposing structured outputs suitable for ranking, browsing, and archival snapshots. The stated limitations of the pipeline—fixed thresholds, incomplete metadata, and source bias between preprints and journal articles—also indicate the principal constraints on any such extension.