Papers
Topics
Authors
Recent
Search
2000 character limit reached

LLM Olympiad: Why Model Evaluation Needs a Sealed Exam

Published 24 Mar 2026 in cs.AI and cs.CL | (2603.23292v1)

Abstract: Benchmarks and leaderboards are how NLP most often communicates progress, but in the LLM era they are increasingly easy to misread. Scores can reflect benchmark-chasing, hidden evaluation choices, or accidental exposure to test content -- not just broad capability. Closed benchmarks delay some of these issues, but reduce transparency and make it harder for the community to learn from results. We argue for a complementary practice: an Olympiad-style evaluation event where problems are sealed until evaluation, submissions are frozen in advance, and all entries run through one standardized harness. After scoring, the full task set and evaluation code are released so results can be reproduced and audited. This design aims to make strong performance harder to ``manufacture'' and easier to trust.

Summary

  • The paper introduces an Olympiad-style protocol that uses sealed tasks and frozen submissions to prevent overfitting from web-scale training contamination.
  • It presents empirical evidence showing that benchmark instability can lead to up to a 13% drop in performance on certain tasks.
  • The protocol emphasizes post-hoc auditability and centralized evaluation to enhance transparency and reproducibility in LLM testing.

LLM Olympiad: Reframing Model Evaluation with Sealed Exams

Motivation and Evaluation Landscape

The paper "LLM Olympiad: Why Model Evaluation Needs a Sealed Exam" (2603.23292) asserts that current evaluation protocols for LLMs fail to provide robust, generalizable measures of capability due to three central issues: (1) test contamination from web-scale training, (2) protocol fragility stemming from multiple degrees of freedom in evaluation and prompt engineering, and (3) incentive misalignment leading to selective disclosure and benchmark-chasing. While open benchmarks (e.g., GLUE [Wang et al., 2018], MMLU [Hendrycks et al., 2021]) are transparent and reproducible, they are susceptible to overfitting and contamination. Closed benchmarks delay direct exposure but lack auditability and impede community learning. Shared tasks centralize scoring but often provide prior knowledge of task types, encouraging targeted system engineering rather than general preparedness.

Empirical evidence shows substantial instability: the ranking of leading LLMs fluctuates across versions of benchmarks and observed contamination can reduce generalization capability by significant margins (e.g., up to 13% drop in accuracy for certain models on arithmetical tasks [Zhang et al., 2024]). Furthermore, selective reporting amplifies leaderboard distortions—providers may privately test dozens of model variants, reporting only the best, as in the documented behavior on Chatbot Arena [Singh et al., 2025].

LLM Olympiad Protocol

The authors propose introducing an "Olympiad-style" evaluation protocol that combines the advantages of open, closed, and shared evaluation formats. Central features are:

  • Task Sealing: Evaluation problems—and task types themselves—remain strictly confidential until the scoring phase, eliminating targeted optimization.
  • Submission Freezing: Participants submit artifacts/endpoints prior to task disclosure, enforcing a genuine test of general preparedness and mitigating selective disclosure.
  • Centralized Harness: All entries are evaluated under a uniform, organizer-controlled pipeline, minimizing protocol variance and ensuring methodological comparability.
  • Post-hoc Auditability: After scoring, the entire bundle—task set, scoring code, evaluation harness, submission manifests—is released to the community, enabling transparent verification and longitudinal analysis.

This protocol does not replace existing benchmarks but augments them with a higher-assurance checkpoint. The Olympiad guarantees not only dataset freshness but procedural standardization and verifiable results.

Design and Operational Mechanics

The operational design emphasizes predictable rules, surprise task content, and rigorous governance. Core mechanics include:

  • Pre-event Syllabus: Public release of interfaces, budgets, allowed tool policies, submission contractual requirements, and scoring aggregation procedures.
  • Task Solicitation and Curation: Open calls for task proposals, mandated conflict-of-interest policies, overlap checks and ambiguity red-teaming as per best practices in robustness-oriented evaluation [Goel et al., 2021].
  • Sealed Problem Bundling: Restricted-access repositories, freeze dates, and public archive fingerprints minimize tampering and leakage risk.
  • Submission Types: Separate tracks for models (base plus minimal glue) versus systems (model plus retrieval/tools/orchestration). Supporting open-weights (containerized artifacts) and closed-weights (endpoint with version commitment) modes, with assurance tiers clearly labeled.
  • Harness Execution: Fixed evaluation periods, deterministic decoding or declared stochasticity policy, standardized retry/timeout handling, budget enforcement, option for lightweight stability probes, and central logging.
  • Result Reporting: Layered release with leaderboards by track/type, budget reporting, per-task breakdowns, and full reproducibility bundles.

The protocol is aligned with MLPerf’s compliance requirements [Reddi et al., 2020], HELM’s multi-metric transparent reporting [Liang et al., 2022], and recent efforts in dynamic benchmarking and contamination-free evaluation (Dynabench [Kiela et al., 2021], LiveBench [White et al., 2024]).

Threat Model, Risks, and Mitigations

The protocol is designed against strong optimization incentives, anticipating both rational and adversarial behaviors. Key mitigations include:

  • Contamination Reduction: Preference for newly curated datasets, similarity screening, and post-hoc public release for community cross-checking.
  • Integrity Enforcement: Freeze-and-commit window, endpoint versioning, separate reporting by assurance tier. Stability probes supplement detection of endpoint drift or stochasticity.
  • Harness Pre-registration: Decoding, aggregation, error handling choices are fixed and disclosed prior to evaluation; bug-triggered reruns are required.
  • Interpretational Transparency: Robust reporting prevents aggregation-induced misinterpretation, detailed per-task results mitigate overgeneralization.

The authors acknowledge residual risks: contamination cannot be eliminated, closed-endpoint assurance is inherently lower, harness bugs can affect results, and sparse task sets may create upstream benchmark lottery effects [Dehghani et al., 2021]. Explicit filtration, coverage enforcement in curation, and iterative expansion of task diversity are recommended.

Practical and Ethical Implications

Practically, the Olympiad imposes non-trivial infrastructure and governance requirements, including restricted-access repositories, clear conflict-of-interest policies, centralized evaluation resources, and comprehensive logging for auditability. Task authors should be incentivized via recognition and citation; accessibility concerns are addressed via clear budget classes and academic subsidies.

Ethically, the procedure shapes incentives and credible claims. The event must avoid privileging resource-rich teams, ensure provenance and privacy in data handling, manage dual-use risks in system-oriented tasks (e.g., prompt injection stress tests), and communicate results responsibly, focusing on capability profiles rather than winner-takes-all rankings.

Conclusion

The paper’s position is clear: existing benchmark protocols are insufficient for high-assurance, generalizable evaluation in the LLM era. The Olympiad-style protocol directly addresses contamination, protocol fragility, and incentive distortion by sealing tasks, freezing submissions, centralizing evaluation, and enabling community audit. While not a replacement for other benchmarks, this approach establishes a reproducible, transparent checkpoint for the field, better suited to evaluating broad preparedness and supporting high-stakes decision-making.

Future developments in AI evaluation can build on this framework by expanding task diversity, refining governance structures, and integrating robust auditing infrastructure, thereby strengthening the epistemic foundation of claims regarding LLM capabilities. The practical challenges—governance, infrastructure, scalability—are surmountable and essential for maintaining scientific integrity as the field evolves.

References

See (2603.23292) for complete references.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

What this paper is about

This paper argues that the way we “test” LLMs today can be confusing and sometimes unfair. The authors propose a new kind of test, like an academic Olympiad, where the questions are kept secret until test day, all teams follow the same rules and setup, and then everything (questions and grading code) is shared afterward so everyone can double-check the results.

What questions the paper asks

The paper tries to answer, in simple terms:

  • How can we make LLM tests harder to game and easier to trust?
  • How do we avoid models “studying the answer key” by seeing test questions on the internet before the test?
  • How can we make scores more comparable across different teams and models?
  • How do we keep the process transparent so the community can learn from the results?

How the authors approach the problem

Instead of running another open or closed benchmark, the authors propose an “LLM Olympiad,” inspired by school Olympiads (like the International Math Olympiad):

  • Problems are sealed: The exact tasks stay secret until the evaluation starts. This reduces “teaching to the test.”
  • Submissions are frozen: Teams must lock in their model version before seeing any tasks, so they can’t tune at the last minute.
  • One standardized harness: All models are run in the same environment with the same rules (like taking the exam in the same room, with the same time limit, and the same calculator rule).
  • After the test: The task set, scoring code, and logs are released, so anyone can reproduce and audit the results.

To avoid “gotcha” surprises, rules are published in advance (the “syllabus”): what inputs/outputs look like, time and token limits, allowed tools, and how scores are calculated and reported.

To keep things fair and clear, they also suggest splitting submissions into tracks:

  • Model track: Just the model (no outside tools).
  • System track: Model plus tools (like search), with strict limits and logging.

They also outline how to collect and “seal” tasks (limited access, freeze dates, and a digital fingerprint to prove nothing changed later), how to run the evaluation (budgets, retry rules, logging), and how to release results in two stages (leaderboard first, then full public release).

What they found and why it matters

This is a position paper, not an experiment with results. So the “findings” are arguments backed by examples from recent research:

  • Today’s benchmarks can be fragile:
    • Small, seemingly harmless choices (like prompt order) can flip who looks best.
    • Models trained on web data may accidentally see test questions (or close copies) beforehand.
    • Teams can try many private runs and only report their best score, making progress look bigger than it is.
  • A sealed, one-shot, standardized test helps because:
    • Sealed tasks limit “training on the test.”
    • A single, shared evaluation harness removes hidden differences in how teams run their models.
    • Freezing submissions stops “best-of-many-tries” score shopping.
    • Releasing tasks and code afterward preserves transparency and learning.

In short, the proposal combines the good parts of open benchmarks (transparency), closed benchmarks (test secrecy), and shared tasks (centralized evaluation), while trying to avoid their biggest weaknesses.

What this could change going forward

If adopted even once a year, an LLM Olympiad could:

  • Give the community a trusted “checkpoint” that shows what models can really do on fresh, unseen problems.
  • Make leaderboard rankings more meaningful for decisions (like deployments, safety claims, and funding).
  • Encourage broader, more general preparation instead of narrow, benchmark-specific tuning.
  • Still let everyone audit and learn afterward, because all tasks and code would be released.

The authors are clear about limits: this isn’t a magic fix. Some contamination risk remains, closed API models are harder to audit than open ones, and the evaluation harness itself can have bugs. Also, running this kind of event takes careful governance and effort. That’s why they frame it as a complement—an extra “exam” for high-assurance testing—rather than a replacement for existing benchmarks.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a single, concrete list of what remains missing, uncertain, or unexplored in the paper’s proposal so future researchers can act on it.

  • Empirical validation: Design and run pilot Olympiad(s) to quantify whether sealed tasks, submission freezing, and a standardized harness materially improve ranking stability, reduce contamination effects, and curb selective disclosure relative to existing benchmarks.
  • Statistical reliability: Develop and preregister methods for uncertainty quantification (confidence intervals, bootstrap), statistical significance testing, and score stability analyses for per-task and overall rankings.
  • Psychometrics and equating: Apply item response theory or related methods to assess task difficulty/discrimination, and design “equating” procedures (e.g., anchor items) that make scores comparable across Olympiad rounds.
  • Sample size planning: Determine the minimum number and mix of tasks needed to achieve target reliability (e.g., standard error of measurement), including power analyses for detecting meaningful model differences.
  • Capability coverage: Formalize a capability taxonomy and machine-checkable coverage metrics (e.g., extraction, reasoning, generation, robustness) to guide task selection and to report coverage gaps.
  • Task diversity and bias: Establish procedures (quotas, reviewer guidelines, automated checks) to prevent over-concentration in English, specific domains, or cultural contexts; extend to multilingual and multimodal tasks with fairness guarantees.
  • Task “freshness” measurement: Create scalable, standardized pre- and post-hoc overlap screening (exact, near-duplicate, paraphrase-level) and report a contamination score with each task/model result.
  • Contamination audit after release: Provide a reproducible pipeline and public reports quantifying overlap between released tasks and major training corpora to contextualize results ex post.
  • Security of task sealing: Specify and validate technical controls beyond hashing (e.g., secure enclaves, HSMs, threshold encryption, access logging, audits) to mitigate insider leaks or tampering.
  • Governance and COI management: Prototype and evaluate governance structures (role separation, conflict-of-interest enforcement, appeals process) and assess their effectiveness and failure modes in practice.
  • Task author incentives: Test incentive schemes (recognition, awards, coauthorship, grants) to ensure a steady supply of high-quality, confidential task submissions.
  • Harness determinism and reproducibility: Define and evaluate policies for decoding settings, random seeds, hardware classes, library/tokenizer versions, and quantization to minimize hardware/software nondeterminism.
  • Stochastic generation handling: Compare alternatives (deterministic decoding vs. multiple sampled runs with aggregation) and quantify their impact on ranking robustness.
  • Endpoint auditing (closed-weights): Investigate mechanisms (remote attestation, cryptographic version pinning, signed logs, differential testing) to detect endpoint drift, hidden routing/tool use, or unlogged changes during evaluation.
  • Open- vs closed-weights comparability: Study how assurance tiers diverge in practice and develop reporting that makes cross-tier comparisons interpretable without overstating certainty.
  • System-track design: Specify organizer-provided retrieval corpora (provenance, licensing, contamination checks), standard tool APIs, and logging; measure the fairness and representativeness of these resources.
  • Prompt-injection and tool security: Develop standardized, auditable defenses (proxies, sandboxing, input sanitization) and evaluation protocols for prompt-injection and data exfiltration risks during system-track runs.
  • Sandbox and container security: Define threat models and implement isolation (network egress controls, filesystem restrictions, secrets management) to prevent model-side attacks on the harness or task leakage during evaluation.
  • Retry and timeout policy effects: Quantify how different timeout/retry policies alter scores, and standardize a policy with demonstrated minimal bias across model types.
  • Metric normalization and aggregation: Establish principled metric normalization and weighting schemes across heterogeneous tasks; publish sensitivity analyses showing how leaderboard order changes under alternative aggregations.
  • Human/LLM judging reliability: When qualitative judging is unavoidable, create protocols for rater training, inter-annotator agreement, adjudication, and controlled use of LLM judges (bias auditing, calibration).
  • Budget classes and accessibility: Empirically assess how token/time/tool budgets affect ranking and accessibility for smaller teams; design budget-class leaderboards and subsidy mechanisms that preserve comparability.
  • Multilingual and multimodal expansion: Specify how interfaces, budgets, and scoring extend to non-English and multimodal tasks; measure cross-language fairness and report per-language performance.
  • RAG contamination control: Devise methods to ensure retrieval corpora are “fresh” and not already memorized by models; document curation and licensing; report overlap metrics for corpora as well as tasks.
  • Longitudinal sustainability: Determine feasible event frequency, funding, and staffing models; measure community uptake and the administrative burden of repeated sealed evaluations.
  • Incentive impact studies: Empirically evaluate how sealed evaluations change participant behavior (e.g., reduction in benchmark-chasing, possible discouragement of task-specific innovation) and refine rules accordingly.
  • External validity: Test whether Olympiad scores predict real-world deployment performance and reliability (predictive validity), including robustness to data drift and user-defined tasks.
  • Leakage post-release: Develop policies and tooling to prevent released tasks from compromising future rounds (e.g., rotation schedules, partial redaction for security-sensitive items, delayed release strategies).
  • Fairness auditing of tasks: Build pipelines to check tasks for harmful content, stereotyping, or demographic biases; require and release task-level datasheets detailing provenance, licenses, and ethical considerations.
  • Cross-venue coordination: Explore ways to harmonize Olympiad outputs with existing benchmarks (model cards, HELM-style reports), enabling meta-analyses and consistent public communication.
  • Preparedness metrics beyond accuracy: Define and validate auxiliary metrics (stability under perturbations, calibration, refusal handling, tool-use safety) that better capture “general preparedness.”
  • Tie-breaking and missingness: Specify principled policies for handling missing outputs, tool failures, and ties; quantify how these choices affect rankings.
  • Geographic/latency fairness (endpoints): Measure and control for infrastructure-induced disparities (rate limits, latency variability across regions) in endpoint evaluations.
  • Session memory and adaptivity control: Create checks that endpoint-based models do not adapt during the evaluation window (e.g., stateless sessions, memory reset tests).
  • Legal/IP frameworks: Provide standard licensing templates and guidance for sealed tasks and retrieval corpora to minimize legal risk and streamline post-event release.
  • Community auditing workflows: Define minimal artifacts (run manifests, logs, seeds, hardware specs) required for third parties to replay evaluations and dispute resolution timelines when discrepancies are found.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below are actionable use cases that can be deployed now, drawing directly on the paper’s sealed-task, freeze-and-commit, and standardized-harness protocol.

  • Model vendor selection and procurement checkpoints [Industry, Policy; software, healthcare, finance]
    • Use sealed “exam” evaluations as a pre-launch or vendor-comparison gate for LLMs (e.g., require an Olympiad-style scorecard with per-task breakdowns and budget reporting in RFPs).
    • Potential tools/workflows: standardized evaluation harness with budget enforcement; assurance-tier labels (open-weights vs. closed-endpoints); reproducible submission badges.
    • Assumptions/dependencies: trusted third-party organizers; clear syllabus and budget classes; adequate compute or endpoint access; willingness to adopt per-task reporting.
  • Internal “sealed eval sprint” for product readiness [Industry; software, education, support ops]
    • Run quarterly in-house Olympiad-style sprints before shipping major features (surprise tasks, frozen artifacts, central harness) to reduce cherry-picking and overfitting to public benchmarks.
    • Potential tools/workflows: containerized inference interface; freeze window automation; retry/timeout policy; consistency probes (duplicate items) to detect instability.
    • Assumptions/dependencies: internal governance separating task authors and submitters; task confidentiality; change-control and rerun procedures.
  • Safety and red-team validation with system track rules [Industry, Policy; security, software]
    • Stress-test tool-augmented systems (RAG, function-calling) via organizer-controlled tool proxies, budgets, and full logging; include prompt-injection probes.
    • Potential tools/workflows: tool gateway/proxy with audit logs; injection challenge set; stability/statistics dashboards for tool-call failures.
    • Assumptions/dependencies: curated, license-compliant task sources; responsible release for security-sensitive tasks; separation of model vs. system tracks.
  • Academic shared tasks upgraded to “sealed exam” format [Academia; NLP, IR, MT]
    • Adapt existing competitions (e.g., SemEval, WMT, TREC) with sealed task types, centralized harness, and post-hoc release to increase auditability and reduce target engineering.
    • Potential tools/workflows: public fingerprinting (hash of encrypted task bundle); preregistered decoding/aggregation policies; per-task capability profiles.
    • Assumptions/dependencies: program committee governance; conflict-of-interest policies; participant buy-in.
  • Conference/journal reproducibility requirements [Academia, Policy]
    • Require that claims relying on benchmarks include one sealed-evaluation score under a standardized harness, plus release of the run manifest once tasks are public.
    • Potential tools/workflows: “assurance tier” labels in papers; artifact review checklists mapped to harness metadata (versions, settings, logs).
    • Assumptions/dependencies: editorial policy alignment; accessible evaluation windows; exceptions for restricted data domains.
  • Sector-specific pilot evaluations (domain Olympiads) [Industry, Academia, Policy; healthcare, finance, legal]
    • Organize small, domain-focused sealed events using newly curated, license-cleared corpora (e.g., de-identified clinical Q&A, regulatory compliance scenarios).
    • Potential tools/workflows: domain syllabus (budgets, interfaces); overlap screening/deduplication tools; per-task error taxonomy for domain risks.
    • Assumptions/dependencies: data licensing and privacy vetting; domain experts as task setters; limited qualitative judging with blinded protocols.
  • Government procurement and grant language [Policy; public sector, defense]
    • Add boilerplate requiring sealed-task evaluations, disclosure of budget classes, and separate reporting for endpoints vs. organizer-run artifacts.
    • Potential tools/workflows: “evaluation governance kit” (templates for COI, freeze windows, bug-fix reruns); standardized scorecard format attached to contracts.
    • Assumptions/dependencies: regulatory authority endorsement (e.g., NIST-like guidance); funding for independent evaluators.
  • Customer-facing “nutrition labels” for LLM products [Industry, Daily life]
    • Publish per-task performance, stability indicators, and budgets alongside a single score to enable informed choices by enterprise buyers and consumers.
    • Potential tools/workflows: auto-generated, citable report bundles; capability radar charts; stability/timeout rates.
    • Assumptions/dependencies: marketing/legal alignment; clear communication to avoid overinterpretation of a single number.
  • MLOps/LLMOps integration of the harness [Industry; software]
    • Embed budget enforcement, logging, and deterministic rerun policies into CI/CD for models and systems; auto-trigger sealed evals at release candidates.
    • Potential tools/workflows: GitOps templates for freeze-and-commit; container interfaces; replayable run manifests.
    • Assumptions/dependencies: platform team capacity; cost ceilings; deterministic seeds or multi-run aggregation policy.
  • Third-party evaluation-as-a-service pilots [Industry; cross-sector]
    • Offer timeboxed sealed evaluations with post-hoc release, using public fingerprints and open harnesses to preserve auditability.
    • Potential tools/workflows: endpoint notarization/version attestation; queueing and rate-limit enforcement; billing aligned to budget classes.
    • Assumptions/dependencies: legal agreements with model providers; fair access across open/closed models; transparent change-control.

Long-Term Applications

These applications require further standardization, scaling, ecosystem coordination, or regulatory development before widespread deployment.

  • Annual “LLM Olympiad” as a field-wide checkpoint [Academia, Industry, Policy]
    • A recurring, multi-track sealed event with rotating capability coverage (reasoning, extraction, robustness, tool-use) serving as a high-assurance progress measure.
    • Potential tools/products: community-governed task repository; rolling task-set diversification targets; public leaderboards with assurance tiers.
    • Assumptions/dependencies: sustainable funding; neutral governance; task author incentives and bounties.
  • Regulatory assurance tiers for AI systems [Policy; healthcare, finance, critical infrastructure]
    • Codify open-weights (organizer-run) vs. endpoint (closed-weights) as distinct assurance tiers in compliance frameworks and audits.
    • Potential tools/products: certification badges; conformity assessments referencing sealed-eval protocols; insurer-aligned risk categories.
    • Assumptions/dependencies: regulator consensus; mechanisms to handle endpoint drift; harmonization across jurisdictions.
  • Insurance and risk underwriting for AI deployments [Industry, Policy; insurance, enterprise IT]
    • Use sealed-evaluation results and stability metrics to price operational and compliance risk for LLM-powered products.
    • Potential tools/products: risk scoring models tied to per-task failure modes; premium discounts for reproducible submissions and stability thresholds.
    • Assumptions/dependencies: actuarial data linking eval outcomes to incidents; standardized reporting across vendors.
  • Secure task-exchange marketplaces with incentives [Industry, Academia]
    • Establish marketplaces where vetted task authors submit fresh, licensed tasks with diagnostics, compensated via prizes/royalties, and tracked for provenance.
    • Potential tools/products: provenance tracking and similarity-screening pipelines; contributor recognition systems; exploit-redaction workflows for security tasks.
    • Assumptions/dependencies: IP frameworks; anti-leakage protocols; sustainable economics for contributors.
  • Sectoral consortia for domain-specific sealed testing [Healthcare, Finance, Legal, Education]
    • Consortia curate domain tasks (with privacy controls) and coordinate sealed evaluations tied to sector benchmarks and deployment thresholds.
    • Potential tools/products: de-identification pipelines; consented corpora; sector-specific aggregation and harm-weighted metrics.
    • Assumptions/dependencies: data access agreements; cross-institution governance; ethical review processes.
  • Continuous sealed-eval networks (rolling exams) [Industry, Academia]
    • Evolve from periodic events to continuous, rolling sealed evaluations drawing from recent sources (contamination-resistant) while retaining standardized harness and freeze-commit rules.
    • Potential tools/products: scheduling and attestation services; freshness-driven task ingestion; dynamic budget adaptation.
    • Assumptions/dependencies: compute and ops scalability; robust freshness checks; guardrails against participant overfitting to the “style” of tasks.
  • Education and credentialing for LLM competencies [Academia, Daily life; education, hiring]
    • Use Olympiad-like tasks to certify LLM-based tutoring/assessment tools and to develop “capability profiles” for AI-assisted learning environments.
    • Potential tools/products: educator-facing scorecards; classroom-safe task banks; alignment with academic integrity policies.
    • Assumptions/dependencies: fairness and accessibility constraints; accommodation of multilingual and domain diversity; careful human-in-the-loop evaluation.
  • Robotics and embodied AI scenario exams [Industry; robotics, manufacturing, logistics]
    • Apply sealed protocols to plan-and-act evaluations (simulators, task randomization) with standardized interfaces and logs.
    • Potential tools/products: simulator adapters in the harness; tool budgets mapped to actuator limits; safety-case documentation tied to eval outcomes.
    • Assumptions/dependencies: high-fidelity simulators; scenario licensing; cross-hardware comparability.
  • National/International standards referencing sealed-eval protocols [Policy; standards bodies]
    • Incorporate sealed-task, standardized-harness, and post-hoc release principles into ISO/IEC/NIST guidance and sector standards.
    • Potential tools/products: publicly maintained reference harness; test method standards; compliance test suites.
    • Assumptions/dependencies: multi-stakeholder alignment; maintenance funding; versioning and deprecation policies.
  • Endpoint attestation and notarization infrastructure [Industry; cloud, platforms]
    • Build cryptographic attestation for endpoint versions and routing transparency during sealed evaluations to increase assurance for closed models.
    • Potential tools/products: signed model manifests; verifiable logging; third-party observers for evaluation windows.
    • Assumptions/dependencies: platform support (cloud providers, API gateways); privacy-preserving telemetry; standard schemas.
  • Harmonized reporting and “capability nutrition labels” for consumers [Industry, Daily life]
    • Mature a standardized, comprehensible label for AI tools that aggregates sealed-eval results across editions, highlighting stability, budgets, and known failure modes.
    • Potential tools/products: UX patterns for labels; update policies as tasks become public and enter training corpora; media and consumer-advocacy adoption.
    • Assumptions/dependencies: avoidance of single-number overreach; periodic refresh; clear caveats about contamination limits.
  • Cross-model reliability research leveraging released artifacts [Academia]
    • Use post-hoc releases (tasks, harness, logs) to study rank instability, contamination, and aggregation effects, improving evaluation science and methods.
    • Potential tools/products: meta-benchmark analyses; new aggregation and fairness metrics; robustness probes integrated into future Olympiads.
    • Assumptions/dependencies: persistent access to artifacts; open licensing; community incentives to reproduce and critique results.
View Paper Prompt  View All Prompts 

Glossary

  • Aggregation choices: The selection of how to combine per-task results into overall scores, which can change leaderboard rankings. "Leaderboard ordering can depend on aggregation choices"
  • Aggregation policy: A predefined rule for combining metrics across tasks to produce an overall score. "report per-task and per-track results with an explicit aggregation policy"
  • Audit mindset: An approach to evaluation that anticipates verification and accountability through transparent procedures and artifacts. "align with an audit mindset"
  • Auditability: The ability for others to inspect and verify how an evaluation was conducted and what led to the results. "auditability (the community can later inspect what happened)"
  • Benchmark-chasing: Over-optimizing systems to perform well on known benchmarks rather than improving general capability. "Scores can reflect benchmark-chasing"
  • Budgets: Resource limits set for evaluation, such as token counts, time, or tool usage. "Budgets: context/output limits, latency, number of calls, and tool rules (if any)."
  • Centralized evaluation: Running all submissions under one controlled environment and process to ensure comparability. "Evaluation is centralized: one harness, one environment (as far as possible), one reporting pipeline."
  • Change-control policy: A predefined procedure for handling discovered bugs or changes during evaluation to maintain fairness. "follow a published change-control policy"
  • Closed benchmarks: Test sets that are not publicly available prior to evaluation to reduce overfitting or leakage. "Closed benchmarks can delay direct test-set overfitting"
  • Closed endpoints: Proprietary API-based model submissions where internal components are not visible to evaluators. "Closed endpoints enable broader participation, but they are intrinsically harder to audit."
  • Closed-weights submissions: Submissions where model weights are not shared and are accessed via endpoints with fixed versions. "closed-weights submissions (endpoints with version commitments)"
  • Consistency probe: A small diagnostic test that repeats or perturbs inputs to detect instability or drift. "A lightweight consistency probe can detect instability or endpoint drift"
  • Contamination risk: The danger that evaluation data has been seen during training, inflating performance. "especially around contamination risk"
  • Contest syllabus: The publicly posted rules and constraints of the evaluation event, excluding task content. "the event must publish a contest syllabus in advance"
  • Deduplication practices: Procedures to remove duplicate or near-duplicate data to reduce leakage and bias. "deduplication practices"
  • Decoding settings: Inference-time parameters (e.g., temperature, top-k) that affect model outputs. "decoding settings (when applicable)"
  • Deterministic decoding: Fixing randomness in generation so identical inputs produce identical outputs for fair comparison. "require deterministic decoding"
  • Endpoint commitments: Declaring and locking endpoint versions and behavior during the evaluation window. "endpoint commitments (version identifiers and timeboxed evaluation)"
  • Endpoint drift: Changes in endpoint behavior over time that can affect evaluation reliability. "endpoint drift"
  • Evaluation harness: The standardized execution framework that runs all submissions and enforces rules. "Phase 3: Evaluation harness and tool use."
  • Few-shot settings: Evaluation setups where models are given a small number of examples in the prompt. "In few-shot settings, even demonstration order can swing performance"
  • Freeze-and-commit requirement: A rule that participants must submit a fixed model/version before seeing tasks. "The Olympiad's freeze-and-commit requirement forces participants to submit a single artifact"
  • Generalization claims: Assertions that performance reflects ability to handle unseen data rather than memorized content. "complicating generalization claims"
  • Human-judged components: Parts of evaluation that rely on human assessors rather than automatic metrics. "reduce bias in any human-judged components"
  • Inference pipeline: The end-to-end process and settings used to run model inference during evaluation. "each team still runs its own inference pipeline"
  • Leakage: Unintended exposure of test content that models might memorize or exploit. "expose tasks to optimization and leakage"
  • Leaderboards: Ranked lists of model performance on benchmarks used to communicate progress. "Benchmarks and leaderboards are how NLP most often communicates progress"
  • Macro-average: An averaging method that gives equal weight to each task when computing overall scores. "overall score is macro-average across tasks."
  • MLPerf: A standardized benchmarking suite and ruleset emphasizing fair and comparable evaluation. "MLPerf's emphasis on shared rules and compliance"
  • Open benchmarks: Publicly available test sets that promote transparency and reproducibility. "Open benchmarks are transparent and easy to reproduce"
  • Open-weights submissions: Submissions where model artifacts (e.g., weights, containers) are provided for organizer-run evaluation. "open-weights submissions (containerized artifacts run centrally)"
  • Olympiad-style evaluation protocol: An event where tasks are sealed until scoring, submissions are frozen, and a standardized harness is used. "sealed, Olympiad-style evaluation protocol"
  • Post-hoc auditability: The capacity to verify evaluations after they occur through released artifacts. "post-hoc auditability after release"
  • Post-hoc release: Publishing tasks, code, and logs after evaluation for community review. "mandatory post-hoc release"
  • Preregistration: Declaring evaluation procedures and settings before running tests to prevent ad hoc changes. "Mitigation here is preregistration"
  • Prompt-based evaluation: Assessing models via carefully designed prompts, often sensitive to prompt details. "amplified by prompt-based evaluation"
  • Prompt injection: An attack where inputs manipulate an LLM’s instructions or tools to produce unintended behavior. "prompt injection is a concrete system risk"
  • Public fingerprint: A published cryptographic hash of the sealed task archive to verify integrity. "Public fingerprint: organizers publish a hash of an encrypted archive"
  • Qualitative judging: Human assessment of outputs using subjective criteria rather than automated metrics. "If qualitative judging is included, it should be small, blinded"
  • Red-team: Actively probing tasks or systems to find ambiguities, loopholes, or vulnerabilities before evaluation. "organizers should red-team tasks"
  • Robustness-oriented evaluation tools: Tools focused on testing model behavior under perturbations or adversarial conditions. "robustness-oriented evaluation tools"
  • Sealed evaluation set: A hidden bundle of tasks and scoring code that is not revealed until after submissions are frozen. "sealed evaluation set (instances, scoring code, and metadata)"
  • Separation mechanisms: Procedural separations (e.g., sealing tasks, freezing submissions) to prevent gaming. "two separation mechanisms"
  • Shared tasks: Community evaluations with centralized scoring and hidden test labels, often announced in advance. "Shared tasks improve comparability by running centralized evaluation on hidden test labels."
  • Standardized harness: A uniform execution framework applied to all submissions to reduce variability. "one standardized harness"
  • Standardized reporting: Consistent, transparent disclosure of evaluation details and results. "transparent, standardized reporting"
  • Stochasticity: Randomness in model generation that can affect repeatability and scoring. "define how stochasticity is handled"
  • Submission contract: The explicit definition of what is fixed and committed in a submission. "Submission contract: what counts as 'frozen'"
  • Submission freeze window: A time period during which submissions are locked before evaluation begins. "enforce a submission freeze window"
  • System track: An evaluation track allowing models with tools, retrieval, or orchestration under specified rules. "System track: model + retrieval/tools/orchestration under explicit tool rules"
  • Timeboxed evaluation: Constraining evaluation to a fixed window to prevent changes during testing. "timeboxed evaluation"
  • Tool calls: Invocations of external tools (e.g., retrieval) by the system during evaluation. "tool calls logged"
  • Tool proxies: Organizer-controlled gateways that mediate and log tool use during evaluation. "tool proxies (organizer-controlled gateways)"
  • Version commitments: Declaring a fixed model or endpoint version for the duration of evaluation. "version commitments"
View Paper Prompt  View All Prompts 

Open Problems

We found no open problems mentioned in this paper.

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 61 likes about this paper.

Sign Up for Free