LLMEval-3: Dynamic LLM Benchmark

• LLMEval-3 is a dynamic evaluation framework that counteracts data contamination and leaderboard overfitting by using a private question bank and dynamic test sampling.
• It employs a two-layer anti-cheating protocol and calibrated LLM-as-a-judge scoring with 90% agreement to ensure fair and robust assessments.
• A 20-month study of nearly 50 models demonstrates LLMEval-3’s ability to achieve ranking stability and reveal domain-specific performance gaps.

LLMEval-3 is a framework for dynamic evaluation of LLMs that was introduced to address the vulnerability of static benchmarks to data contamination and leaderboard overfitting. It is built around a private bank of 220k or more graduate-level questions, dynamically samples unseen test sets for each evaluation run, enforces a two-layer anti-cheating protocol, uses a calibrated LLM-as-a-judge process achieving 90% agreement with human experts, and reports relative rankings intended to remain stable despite test-set variation. A 20-month longitudinal study of nearly 50 leading models positions LLMEval-3 as a methodology for assessing model capability beyond fixed public leaderboards (Zhang et al., 7 Aug 2025).

1. Motivation and problem formulation

LLMEval-3 was proposed in response to three problems identified in the evaluation of contemporary LLMs: data contamination, leaderboard overfitting, and misleading performance metrics. In the formulation given for the framework, static and public benchmarks can leak into training corpora, repeated evaluation on fixed test sets can induce overfitting or manipulation, and fixed datasets can obscure the distinction between memorization and genuine capability. LLMEval-3 therefore replaces static test sets with dynamic evaluation in which each run draws unseen questions from a private and continually updated bank (Zhang et al., 7 Aug 2025).

This emphasis marks a shift in focus within the broader LLMEval line. The earlier LLMEval study concentrated on “what, where, and how to evaluate,” comparing evaluation criteria, evaluator types, scoring methods, and ranking systems across LLMEval-1 and LLMEval-2; LLMEval-3 instead targets the reliability of the benchmark substrate itself, especially under contamination pressure and repeated public reuse (Zhang et al., 2023).

A common misconception is that higher scores on established public benchmarks necessarily indicate better generalization. LLMEval-3 directly contests that assumption by making benchmark freshness, access control, and ranking stability central parts of the evaluation protocol rather than peripheral engineering concerns. This suggests that benchmark design and benchmark governance are treated as part of the measurement problem, not merely as implementation details.

2. Contamination-resistant data construction

The core data resource in LLMEval-3 is a private question bank built from graduate and undergraduate exam questions spanning 13 academic domains and more than 50 subdisciplines. The examples named in the source description include engineering, medicine, and literature. Upwards of 38 subject specialists participate in filtering, removing errors and irrelevant or factual issues. The resulting bank is not only curated but expanded: original questions are systematically rewritten with LLM-based augmentation so that, for example, a multiple-choice item with $n$ options can be transformed into $n$ fill-in-the-blank or truth-value questions, and material-analysis questions can be broken down for greater diversity and depth (Zhang et al., 7 Aug 2025).

The framework also standardizes the bank operationally. All items receive unique UUIDs, metadata enrichment, consistency validation, and deduplication. Regular expansion is explicitly part of the design, both to keep the benchmark fresh and to reduce the risk that its contents enter public pretraining corpora. After LLM-driven expansion, the bank contains 220k or more questions (Zhang et al., 7 Aug 2025).

The construction procedure is significant because LLMEval-3 treats contamination resistance as a property that must be engineered upstream. The bank is private, dynamically sampled, and continuously updated; these are not ancillary safeguards but the primary mechanism by which memorization pressure is reduced. A plausible implication is that LLMEval-3 views dataset maintenance as an ongoing adversarial process rather than a one-time release event.

3. Dynamic testing and anti-cheating architecture

LLMEval-3 organizes testing around a secure and dynamic protocol. Each evaluation run samples a new set of 1,000 unseen questions. The testing process further requires that questions be randomly sampled, non-repeating, and ordered, and models cannot skip or shuffle them. This is intended to prevent cherry-picking, avoid selective exposure to easier samples, and reduce the utility of benchmark leakage (Zhang et al., 7 Aug 2025).

The anti-cheating system has two layers. The outer layer handles authentication and access control through JWT. It is described as operating in three phases—generation, transmission, and verification—and checks session validity, permissions, expiration, and concurrent attempts. Role-Based Access Control enforces strict session isolation and is designed to prevent cross-user or cross-session leaks. The inner layer governs the evaluation process itself: it tracks question quota, records allocation and completion, forbids over-requesting or re-answering, and strips away answers and explanations so that only the essential prompt is exposed to the model (Zhang et al., 7 Aug 2025).

These controls are technically important because they bind protocol compliance to evaluation validity. LLMEval-3 does not treat cheating resistance as a social norm or a post hoc audit; it embeds access restriction, sequence control, and information minimization into the execution environment. That design choice underwrites the paper’s claim that the framework is suited to fair testing under repeated longitudinal use.

4. Scoring, judge calibration, and relative ranking

LLMEval-3 combines automated scoring with explicit calibration. The judge assigns an integer score from 0 to 3, where 0 denotes a wrong answer with wrong explanation, 1 a wrong answer with partially reasonable explanation, 2 a correct answer with partially reasonable explanation, and 3 a correct answer with correct explanation. The primary judge model is GPT-4o, used with a standardized prompt and scoring criteria for reproducibility (Zhang et al., 7 Aug 2025).

Calibration is reported through Cohen’s $\kappa$ between the LLM judge and human evaluators across 13 models and two rounds. GPT-4o achieves $\kappa > 0.90$, described as “almost perfect” agreement, and is reported to outperform other candidate judge models such as Doubao and Gemini, whose agreement is more variable. In the abstracted description of the system, this is summarized as 90% agreement with human experts (Zhang et al., 7 Aug 2025).

LLMEval-3 distinguishes between absolute and relative scoring. The absolute score, $S_\text{model}$, averages the per-question scores over $N = 1000$ questions with a maximum score $s_{\text{max}} = 3$, then rescales to 100. The relative score, $R_\text{SOTA}^\text{model}$, normalizes a model’s absolute score by the best model in the same run. The motivation is explicit: absolute scores can vary with the sampled question set, whereas per-run relative comparison is intended to preserve fairness across dynamic test draws (Zhang et al., 7 Aug 2025).

This ranking design has a methodological consequence. LLMEval-3 does not interpret benchmark evaluation as the estimation of a single invariant score on a fixed corpus; instead, it treats comparison under repeated sampling as the more robust object of interest.

5. Longitudinal empirical results

The empirical study runs for 20 months and covers nearly 50 leading open and proprietary models from late 2023 to mid-2025. One headline finding is convergence near a performance ceiling on knowledge-intensive tasks: high-performing models cluster around $90\%$ to $93\%$. The concrete examples reported are DeepSeek-R1 at 91.23 and Doubao-1.5-Thinking-Pro at 93.67 (Zhang et al., 7 Aug 2025).

The study also reports marked domain variation. Models score highly in engineering, management, and economics, at or above $n$0, but remain weaker in literature, medicine, and military topics, with scores below $n$1. Prompting paradigm contributes comparatively little: Zero-Shot, Few-Shot, and Chain-of-Thought differ by no more than 1.6 points. By contrast, external web search augmentation, exemplified by GPT-4o-search, yields notable gains in knowledge-heavy subjects (Zhang et al., 7 Aug 2025).

The error analysis is likewise specific. The main limitations are domain knowledge gaps, accounting for 47.7% of errors, and comprehension failures, accounting for 39.8%; only a small share is attributed to reasoning failures. In context, this result is notable because it shifts attention away from generic “reasoning” narratives and toward coverage and understanding failures on expert-level knowledge tasks (Zhang et al., 7 Aug 2025).

Taken together, these findings suggest that the dynamic protocol is used not only to rank models but also to characterize where current systems are saturating and where they remain fragile. The reported ceiling is therefore not a general statement about all LLM competence, but about the knowledge-intensive evaluation regime instantiated by LLMEval-3.

6. Comparison with static benchmarks and robustness claims

LLMEval-3 reports that its dynamic rankings often diverge from static public benchmarks such as AGIEval and C-Eval, with rank correlations around $n$2 to $n$3. The interpretation given is that static benchmarks do not adequately capture true generalization or contamination risk. This claim is reinforced by the paper’s contamination probe: on static public benchmarks, even modestly sized LLMs can recall or complete more than 200 fill-in-the-blank items, whereas on LLMEval-3’s private set successful memorization is rare and mostly below 100 (Zhang et al., 7 Aug 2025).

The framework also emphasizes ranking stability under repeated resampling. Across multi-round experiments with different question sets and sizes $n$4, relative rankings are described as virtually unchanged, with variance below 2% even in the worst case. The relative ranking method is reported to be significantly more stable than absolute, Elo-style scoring (Zhang et al., 7 Aug 2025).

This robustness claim connects to broader concerns in the evaluation literature about unstable rank aggregation. Earlier LLMEval work had already argued that Elo can be sensitive to order and noise and that more stable alternatives are preferable for large-scale LLM ranking (Zhang et al., 2023). LLMEval-3 extends that concern into the dynamic-benchmark regime by coupling relative ranking with repeated resampling and private test-set rotation.

7. Position within the LLMEval ecosystem and future directions

Within the broader LLMEval ecosystem, LLMEval-3 occupies the role of a dynamic, contamination-resistant general evaluation framework. Other components of the ecosystem pursue different targets; for example, LLMEval-Logic is a solver-verified Chinese benchmark for natural-language logical reasoning with forward-authored items, formal verification via Z3, rubric-based formalization grading, and adversarial hardening (Zhang et al., 19 May 2026). The coexistence of these efforts indicates that “LLMEval” names a family of evaluation infrastructures rather than a single benchmark format.

The future directions explicitly named for LLMEval-3 are broader and deeper dataset expansion, improved LLM-judge methodologies, community engagement through open-sourcing of protocols and evaluation APIs, advanced robustness analysis under adversarial settings, and extension to multilingual and multimodal evaluation (Zhang et al., 7 Aug 2025). A plausible extension is suggested by later work on learned-rule-augmented evaluators, which argues that rule distillation, Chain-of-Rule prompting, and reinforcement-learned rule-augmented evaluators could supply more interpretable and data-aligned scoring criteria for frameworks such as LLMEval-3, though these are not part of the released LLMEval-3 pipeline itself (Meng et al., 1 Dec 2025).

In that sense, LLMEval-3 can be situated at the intersection of benchmark security, judge calibration, and longitudinal measurement. Its distinctive contribution is not a new task family alone, but a claim about how credible LLM evaluation must be organized when static public test sets are no longer reliable proxies for capability.