EvalCards: Unified AI Evaluation Reporting

• EvalCards is a unified reporting system that integrates benchmark metadata, evaluation run data, and model information to provide traceable AI evaluation records.
• It organizes scores through a five-level rollout hierarchy and computes integrity signals like reproducibility, completeness, and comparability for precise assessment.
• The framework supports dual reader modes, catering to technical researchers and policy makers, which enhances both detailed analysis and executive summary reporting.

EvalCards is an operational reporting layer for AI evaluation reporting that composes benchmark metadata, evaluation run data, and model metadata into a unified record. It was introduced to address fragmented formats across leaderboards, model cards, benchmark papers, and company blogs; missing context around reported scores; poor traceability from aggregate claims to underlying evidence; and inconsistent cross-source results for nominally similar model–benchmark pairs. The framework derives a reporting schema from a structured review of 52 papers and 10 stakeholder interviews, computes four interpretive signals over reported results, and deploys a monitoring tool intended to surface systematic gaps in reporting practice at scale (Ghosh et al., 8 Jun 2026).

1. Reporting problem and design objectives

The motivating claim behind EvalCards is that the central difficulty in contemporary AI evaluation is not merely the production of scores, but their interpretation. Existing reporting artifacts use incompatible schemas, often omit key metadata such as prompt-engineering parameters, dataset provenance, or scoring rules, and typically flatten evaluation evidence into $(\text{model}, \text{benchmark}, \text{score})$ triples that obscure sub-tasks, versions, and configurations. EvalCards is designed to replace that flattening with a canonical record in which evidence can be traced unambiguously and omissions can be surfaced explicitly (Ghosh et al., 8 Jun 2026).

The framework specifies four design goals. First, it introduces a unified schema that joins benchmark metadata, evaluation-run logs, and model metadata. Second, it computes four “integrity checks”—Reproducibility, Completeness, Provenance & Risk, and Comparability—over every reported result. Third, it organizes benchmark reporting through a five-level rollout hierarchy, $$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$, so that each score can be located within a structured benchmark decomposition. Fourth, it renders the same underlying evidence in two reader modes, one oriented toward research use and one toward summary or policy use.

These objectives distinguish EvalCards from a leaderboard-centric view of evaluation. The framework does not treat scores as self-sufficient. Instead, it treats every score as an artifact that must be embedded in benchmark structure, model provenance, and evaluation configuration before comparison becomes meaningful. A plausible implication is that EvalCards shifts evaluation reporting from simple result display to evidence-accounting.

2. Unified schema and rollout hierarchy

EvalCards ingests and unifies three core entities. Benchmark Metadata is sourced from Auto-BenchmarkCards. Evaluation Run Data is sourced from Every Eval Ever (EEE). Model Metadata is sourced from Hugging Face hub-stats and models.dev. These sources are linked through an Entity Registry using alias tables and fuzzy matching, after which the pipeline joins run records to benchmark cards and emits a unified Parquet warehouse (Ghosh et al., 8 Jun 2026).

Entity	Source	Key fields
Benchmark Metadata	Auto-BenchmarkCards	name, overview, domains, tasks, languages, data details, methodology, risk tags, category tags
Evaluation Run Data	Every Eval Ever (EEE)	evaluation_id, source_metadata, model_info, evaluation_results
Model Metadata	Hugging Face hub-stats; models.dev	release_date, parameter_count, weight_accessibility

Within Benchmark Metadata, the key fields include name, overview, domains, tasks, languages, data details such as size, provenance, and annotation, methodology such as metrics, calculation, and baselines, risk tags from IBM AI Atlas Nexus, and category tags such as agentic, safety, and reasoning. Evaluation Run Data provides evaluation_id; source_metadata, including source_type and evaluator_relationship; model_info, including name, id, developer, and inference_platform; and evaluation_results, whose items include evaluation_name, source_data, metric_config, score_details, and optional generation_config.

A sample JSON schema snippet makes the normalization logic explicit. evaluation_id, source_metadata, model_info, and evaluation_results are required top-level fields. Within source_metadata, source_type is enumerated over "evaluation_run" and "documentation", while evaluator_relationship is enumerated over "first_party", "third_party", "collaborative", and "other". Within evaluation_results, evaluation_name, metric_config, and score_details are required; generation_config is optional and may contain temperature and max_tokens.

The rollout hierarchy is central to EvalCards’ notion of traceability. Each evaluation_result is linked to one model and one benchmark “leaf,” and that leaf is located within the five-level path $$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$. This hierarchy is intended to prevent headline claims from floating free of the specific benchmark component to which they refer.

3. Interpretive signals

EvalCards computes four interpretive signals over result triples $r=(m,b,\mu)$, with field presence represented by $\mathrm{pop}(f,r)\in\{0,1\}$ (Ghosh et al., 8 Jun 2026).

Signal	Central question	Core output
Reproducibility	“Can I re-run this evaluation exactly?”	Missing-field gap flag
Reporting Completeness	“How much of the full 28-field schema do we have for this benchmark?”	Benchmark completeness score
Provenance & Risk	“Who reported this, and what benchmark risks apply?”	Party relationship flags and risk badges
Score Comparability	“Do reported scores diverge across setups or parties by >5%?”	Divergence flags

For reproducibility, the required sub-schema is

$$F_{\mathrm{repr}}=\{\text{temperature},\ \text{max\_tokens}\}.$$

For agentic benchmarks, the required fields also include $$\{\text{harness}, \text{eval\_plan}, \text{eval\_limits}\}$$. The gap flag is

$$G_{\mathrm{repr}}(r)=1-\prod_{f\in F_{\mathrm{repr}}}\mathrm{pop}(f,r),$$

so $G_{\mathrm{repr}}(r)=1$ whenever any required field is missing. The user interface lists the missing fields explicitly.

For reporting completeness, EvalCards defines, for each benchmark $b$,

$$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$0

Here, $$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$1 is a binary population indicator for scalar fields or the fraction of sub-items populated for partial fields. The resulting score quantifies how much of the full 28-field schema is available for that benchmark.

For provenance and risk, the framework records the evaluator relationship $$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$2. It defines a first-party-only flag,

$$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$3

and a multi-party flag when more than one relationship type appears. Risk categories $$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$4 are inherited from Auto-BenchmarkCards and surfaced as attention badges.

For score comparability, EvalCards asks whether reported scores diverge across setup variants or across reporting parties by more than a fixed threshold $$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$5, normalized by the metric range $$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$6. The variant-divergence flag is

$$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$7

the cross-party divergence flag is

$$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$8

and the overall comparability flag is

$$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$9

The interface then highlights which fields differ, such as temperature or shot count. The framework is therefore interpretive rather than merely archival: it does not just store metadata, but computes structured warnings from it.

4. Reader modes and rendering logic

A defining feature of EvalCards is that the same underlying data and signals are rendered in two distinct reader modes: Research Mode and Summary/Policy Mode (Ghosh et al., 8 Jun 2026).

Research Mode is oriented toward technical evaluators. It displays the full metric_config, including metric_kind, min/max, and judge settings; lists missing reproducibility fields explicitly; shows setup differences relevant to comparability; and provides drill-down into the rollout hierarchy path. The mode is configuration-focused and assumes that the reader wants to inspect the mechanics of score production.

Summary/Policy Mode is oriented toward non-technical readers. It hides raw JSON and replaces configuration detail with plain-language summaries. For reproducibility, it may render the warning “How this model was prompted is not documented.” For provenance, it states who reported the result and whether that report is first-party. For comparability, it states that “Different evaluators report divergent scores; interpret with caution.” It also presents three executive summary blocks: What it measures, Main caveat, and Intended for.

The existence of these modes reflects a claim about evaluation audiences: the same evidence record must support both technical reproducibility and accountability-oriented reading. EvalCards therefore does not solve reporting inconsistency by forcing a single presentation style. Instead, it layers interpretation on top of a shared canonical record.

5. Deployment pipeline and empirical monitoring

The deployment architecture is described as a five-stage pipeline (Ghosh et al., 8 Jun 2026).

1. Ingest (A–B): load EEE, Auto-BenchmarkCards, and model catalogs, then explode to one row per result.
2. Standardize (C–D): resolve canonical model, benchmark, and metric IDs via the Entity Registry and join benchmark metadata.
3. Compute signals (E–F): compute per-row reproducibility and completeness flags and group-level provenance and comparability.
4. Build rollout hierarchy (G): construct the hierarchy and emit six Parquet files (I).
5. Render frontend views (J): render Next.js frontend views in Research and Summary modes.

At the reported scale of application, as of June 4, 2026, the system covered 5,816 models, 635 benchmarks, 62 families, 10 composites, 101,955 reported results, and 30 contributing organizations. Entity Registry resolution accuracy was reported as 98.3% for models, 77.4% for benchmarks, and 86.7% for metrics.

The monitoring results are central to the paper’s empirical argument. The corpus-level reproducibility-gap rate is approximately 96.5%, meaning that 96.5% of $$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$0 triples lack at least one minimal reproducibility field. Median benchmark completeness is 10.7% of the 28-field schema. Provenance is highly concentrated: 98.2% of $$\text{Family} \rightarrow \text{Composite} \rightarrow \text{Benchmark} \rightarrow \text{Split} \rightarrow \text{Metric}$$1 pairs are reported by only one party. Among the 1.8% with multi-party reports, 51.9% exceed the 5% comparability threshold.

These figures are used to argue that reporting failure is systematic rather than anecdotal. In particular, the combination of missing reproducibility fields, sparse benchmark documentation, and frequent divergence in the comparatively small multi-party subset suggests that many apparent benchmark comparisons are underdetermined by the public record.

6. Relation to other evaluation cards, limitations, and extensions

EvalCards belongs to a broader family of card-based reporting frameworks, but its scope is distinct. “Scorecards for Synthetic Medical Data Evaluation and Reporting” introduces an SMD Card for synthetic medical data, structured around seven “C” dimensions—Correctness, Coverage, Constraint, Completeness, Compliance, Comprehension, and Consistency—and divided into Descriptive Information and Quantitative Scores (Zamzmi et al., 2024). “Evaluation Cards for XAI Metrics” proposes a lightweight, four-section XAI Evaluation Card covering Identity, Scope and Context, Implementation and Validation, and Relationships and Limitations, with emphasis on target properties, grounding levels, metric assumptions, validation evidence, gaming risks, and known failure cases (Gipiškis et al., 6 May 2026).

This suggests that EvalCards extends the “card” idiom from domain-specific dataset evaluation and metric-specific documentation to a cross-source reporting layer for AI evaluation evidence. Unlike those narrower templates, it emphasizes composition across benchmark metadata, run data, and model metadata, together with extraction infrastructure and at-scale monitoring.

The framework also states several limitations directly. It is source dependent and inherits incompleteness or errors from EEE and Auto-BenchmarkCards. It is artifact-side only and does not cover process documentation such as preregistration or pilot runs. Its reproducibility signal omits random seeds and infrastructure details, since there is no seed/hardware/determinism coverage. Its upstream extraction sources produce a bias toward English and frontier models. Finally, the signals are descriptive, not prescriptive: EvalCards defines no normative pass/fail regime (Ghosh et al., 8 Jun 2026).

The future extensions identified in the framework are correspondingly specific. These include adding structured fields for contamination control through data-overlap detection, automating categorization of non-English benchmarks, integrating with preregistration tools such as PREP-Eval and audit records such as Audit Cards for fuller lifecycle coverage, introducing metric-specific comparability thresholds beyond the uniform 5% rule, and expanding reader modes to include, for example, deployer mode and general public mode. Together, these proposals situate EvalCards as a reporting and monitoring layer whose present contribution is interpretive transparency rather than final standardization.