Eval Factsheets: Standardizing AI Evaluation

• Eval Factsheets are structured documentation artifacts that standardize the capture and reporting of AI evaluation metadata for clearer, reproducible assessments.
• They organize evaluation information along five dimensions—Context, Scope, Structure, Method, and Alignment—to improve transparency and comparability.
• Applications in benchmark evaluations and federated learning demonstrate their role in ensuring accountability through cryptographic assurance and tamper-evident documentation.

Eval Factsheets are structured, descriptive documentation artifacts developed to systematically capture, organize, and communicate the essential metadata of AI evaluation methodologies. Originating in response to the proliferation and fragmentation of benchmarks and evaluation protocols in machine learning, Eval Factsheets aim to improve reproducibility, comparability, and transparency of model assessment by providing a standardized questionnaire-based framework that parallels established metadata practices for datasets (Datasheets) and models (Model Cards) (Bordes et al., 3 Dec 2025). Variants and extensions, such as those introduced in federated learning settings, instrument the factsheet paradigm with strong accountability mechanisms, including verifiable claims and tamper‐evident facts (Baracaldo et al., 2022).

1. Rationale and Standardization Drivers

The lack of consistent evaluation documentation has led to significant challenges: proliferation of benchmarks with incompatible assumptions, hidden protocols leading to irreproducible results, and opacity for stakeholders making decisions about capability claims. Eval Factsheets address these core issues by standardizing the way evaluation metadata are collected and reported (Bordes et al., 3 Dec 2025). The framework is inspired by Datasheets for Datasets and Model Cards, but is distinct in focusing on evaluation methodologies themselves rather than their inputs or outputs. This distinction is crucial, as evaluation strategies have become increasingly complex and integral to model deployment and comparison.

2. Structural Taxonomy: Five Dimensions

Eval Factsheets follow a five-dimensional organizational taxonomy:

Dimension	Formal Definition	Example Mandatory Elements (▣)
Context	Provenance and intent: authors, dates, purposes	Title, authors, date, stated purpose
Scope	Capabilities, properties, and modalities under test	Targeted capabilities, primary metrics, input/output modalities
Structure	Data sourcing, splits, organization, dynamics	Data sources, reference sources, set sizes
Method	Protocol steps, judge type, system access	Judge type, protocol description, access level
Alignment	Reliability, validity, robustness, limitations	Validation methods, baselines, limitations

Each section of the questionnaire includes both mandatory (▣) and recommended (◻) fields, ensuring a baseline level of reproducibility while enabling depth for advanced users (Bordes et al., 3 Dec 2025).

3. Questionnaire Implementation and Fields

The Eval Factsheets questionnaire operationalizes the taxonomy with precise fields:

• Context: Captures evaluation name, versioning, authors/maintainers, and explicit statements of intent (e.g., Development, Research, Deployment).
• Scope: Enumerates the specific abilities or properties probed, primary evaluation metrics (e.g., BLEU, pass@k, accuracy, Elo), and modalities under study.
• Structure: Specifies sources for input and output data, split statistics, and whether the data are static, dynamic, or periodically refreshed; annotation protocols can be documented here for clarity.
• Method: Requires explicit description (or pseudocode) of evaluation flows, judge types (human, automatic, hybrid), and access levels (output-only, partial, full).
• Alignment: Details how reliability is assessed, including measurement validation strategies (correlation against established benchmarks or human review), inclusion of statistical uncertainty quantification (e.g., confidence intervals), baselines for calibration, and known limitations or biases.

Illustrative mathematical metrics include Cohen’s κ for inter-rater reliability:

$\kappa = \frac{p_0 - p_e}{1-p_e}$

and confidence interval estimation for mean scores:

$\mu \pm z \cdot \frac{\sigma}{\sqrt{N}}$

These formulas explicitly support transparent reporting of evaluation uncertainty.

4. Applications and Case Studies

Eval Factsheets have been applied to canonical benchmarks, demonstrating their generality (Bordes et al., 3 Dec 2025):

• ImageNet: Documents dataset provenance (Princeton, 2009), categorical coverage (1,000 classes), static structure (1.2M train, 50K val, 100K test splits), and reference-automatic judging. Alignment includes annotation quality analyses and explicit listing of human performance ceilings and known failure modes (label noise, domain shift).
• HumanEval: Captures Python code synthesis task structure, unit test execution-based judging, prompt protocol transparency, and acknowledges prompt sensitivity and task contamination.
• MT-Bench: Details conversational LLM evaluation with GPT-4 judge, Elo aggregation, refresh protocols for dynamic prompt pools, and explicit agreement metrics between human and LLM judges (~80%).

In federated learning, AF² extends factsheets with cryptographically signed, tamper-evident claims and nested facts for full auditability (Baracaldo et al., 2022). For example, in a multi-city citizen-participation scenario, parties’ data preprocessing and update steps are signed, hashed, recorded on a ledger, and verified by automated predicate-checkers. The factsheet produced documents every step, supporting full bit-for-bit reproducibility.

5. Advanced Features in Distributed and Accountable Settings

The AF² framework applies Eval Factsheets to federated learning by integrating:

• Claims: Signed tuples of information, issuer, timestamp, and cryptographic signature
• Facts: Atomic claims, e.g., preprocessing hashes, model update hashes
• Predicate Evaluation: Logical checks (e.g., “all parties used same data handler”) with binary results
• Ledger Storage: All exchanges (claims/facts) are append-only and tamper-evident, typically implemented via blockchains (e.g., Hyperledger Fabric)
• Assurance Tree: High-level claims justified by subclaims, forming a tree structure supporting complicity audits

Reproducibility is operationalized as the distance

$\rho(M^*, M) = \| M^* - M \|_2,$

where $M^*$ is an auditor-reconstructed model and $M$ is the original, with “$\epsilon$-reproducibility” defined as $\rho(M^*, M) \leq \epsilon$.

6. Benefits, Limitations, and Best Practices

Eval Factsheets facilitate:

• Standardization of terminology and reporting structure, enabling automated meta-analyses and benchmark indexing
• Explicit surfacing of evaluation assumptions and potential validity threats via the alignment dimension
• Strong reproducibility protocols, particularly under AF², where cryptographic guarantees support third-party auditability

Caveats include computational and storage overhead in distributed ledger–backed factsheet variants, the need for universal adoption of the documentation schema, and the requirement that all steps be as deterministic as possible for provable reproducibility (Baracaldo et al., 2022).

Recommended practices:

1. Integrate documentation during evaluation suite design, not post hoc
2. Reuse dataset/model metadata from existing Datasheets or Model Cards to avoid redundancy
3. Distinguish mandatory versus recommended documentation for efficiency and comprehensiveness
4. Link factsheets directly in repositories and model cards for transparency
5. In federated settings, minimize predicates to core concerns and summarize data via cryptographic proofs (Merkle roots) rather than full payloads

7. Impact and Future Directions

Adoption of Eval Factsheets provides a foundation for transparent, reproducible, and accountable AI evaluation. The framework is extensible to emerging domains, including LLM-as-judge settings, dynamic and evolving evaluation suites, and regulated AI environments. Automated toolchains leveraging Eval Factsheet schemas can facilitate metadata collection, completeness validation, and cross-reference construction between evaluation, model, and data artifacts. Extensions such as AF² illustrate evolving demands around cryptographic assurance and runtime auditability in collaborative and federated learning contexts, suggesting continued expansion of the Eval Factsheet paradigm to address the full lifecycle and assurance needs of modern AI systems (Bordes et al., 3 Dec 2025, Baracaldo et al., 2022).