Evidently AI Benchmark Registry

• Evidently AI Benchmark Registry is a structured, continuously maintained catalog that indexes and governs AI benchmarks through standardized metadata and quality assessments.
• It integrates descriptive, operational, and evaluative metadata layers to support reproducibility, interpretability, and lifecycle governance.
• The registry accommodates diverse metric regimes and domain-specific patterns, enhancing transparency and reliability in AI evaluation.

In current benchmark-registry literature, an “Evidently AI Benchmark Registry” is best interpreted as a public, structured, and continuously maintained catalog layer for AI evaluation: it records what a benchmark measures, which tasks and datasets it covers, how scores should be interpreted, which software and hardware stacks are assumed, and whether the benchmark is interpretable, usable, reproducible, and fit for purpose. Recent work realizes this layer through living repositories of benchmark assessments, trustworthiness-oriented benchmark maps, and modular benchmark suites with explicit specifications and released artifacts (Reuel et al., 2024, Ivanov et al., 2024, Li et al., 2022).

1. Conceptual definition and registry scope

BetterBench defines a benchmark as a combination of datasets plus a metric representing one or more tasks or abilities, adopted by a research community for comparing methods. A registry, in turn, emerges when such benchmarks are not merely published as isolated papers, but are organized as shared comparison frameworks with standardized metadata, explicit evaluation criteria, and public artifact access (Reuel et al., 2024). This suggests that a benchmark registry is not itself a single benchmark; it is an indexing, assessment, and governance layer over benchmarks.

The literature shows several concrete registry-like realizations. HPC AI500 is framed as a “new yardstick” for HPC AI evaluation and is positioned in an emerging benchmark registry or catalog context by providing a publicly specified, modular suite with reference implementations and explicit metric definitions (Jiang et al., 2019). AIBench treats end-to-end, component, and micro benchmarks as an integrated suite and releases specifications, source code, testbed, and results publicly, which gives it the characteristics of a reusable benchmark catalog rather than a one-off experiment (Gao et al., 2020). Milabench is described as a curated, versioned, extensible catalog of workloads encoding community importance and operational evaluation rules, while SAIBench is explicitly a scientific benchmarking infrastructure that stores, discovers, composes, and evaluates benchmarks across disciplines rather than a single fixed benchmark (Delaunay et al., 2024, Li et al., 2022). AI-NativeBench extends the same logic into agentic systems by packaging applications, traces, datasets, and evaluation setup as an open-source white-box suite (Wang et al., 14 Jan 2026).

Resource	Registry function	Organizing principle
BetterBench	Living repository of benchmark assessments	46 best practices across a benchmark lifecycle
HPC AI500	Publicly specified modular suite	Scientific DL workloads, metrics, and reference implementations
Milabench	Curated, versioned, extensible catalog	Domain, model family, dataset or environment, parallelization, library stack, weight
SAIBench	Modular registry/framework	Problems, models, metrics, ranking, software, hardware via SAIL
AI-NativeBench	Open-source white-box suite	Applications, traces, protocols, datasets, system variants

A recurring theme is that registry scope depends on the problem class. Some registries organize end-to-end workloads for procurement or system ranking; others catalog public benchmarks for quality assessment, policy interpretation, or scientific reuse. The unifying property is standardization of benchmark identity, scope, and interpretability.

2. Metadata schema and benchmark description layers

A benchmark registry becomes practically useful only when it stores rich metadata rather than names alone. The survey of LLM evaluation benchmarks implies a registry schema with fields such as benchmark name, citation, short description, task type, data scale, modality, intended capability or risks covered, tags corresponding to the seven EU Trustworthy AI requirements, and sometimes dataset splits or number of examples, evaluation methods, or metrics (Ivanov et al., 2024). This is already more expressive than a leaderboard entry, because it supports benchmark selection by purpose rather than by popularity.

BetterBench adds a second description layer: benchmark quality metadata. Its framework organizes 46 criteria over design, implementation, documentation, maintenance, and retirement, although only the first four stages are scored for active benchmarks. The scored stages yield benchmark-level design, implementation, documentation, and maintenance assessments, plus a usability score derived from implementation, documentation, and maintenance (Reuel et al., 2024). This turns the registry from a catalog into an evaluative infrastructure.

Other systems enrich the schema in domain-specific ways. Milabench attaches metadata about domain, model family, dataset or environment, size, parallelization, library stack, unit of work, and weight, thereby making workload composition and subset selection part of the registry itself (Delaunay et al., 2024). AgentSearchBench standardizes heterogeneous agent metadata into a unified schema containing metadata, capability description, usage guidance, and availability or constraints, which is necessary because real agent repositories span multiple providers and interface styles (Wu et al., 24 Apr 2026). AI-NativeBench further shows that a registry entry for agentic systems may need trace-level structure: application domain, agent count, tool count, protocol configuration, evaluated model set, system variants, and released traces or datasets (Wang et al., 14 Jan 2026).

The literature therefore implies at least three metadata strata. The first is descriptive metadata: what the benchmark is, which task family it targets, and which modality or domain it covers. The second is operational metadata: software framework, deployment assumptions, hardware compatibility, scaling mode, and released artifacts. The third is evaluative metadata: interpretability constraints, replication support, contamination controls, maintenance signals, and scope-of-use warnings. A registry that omits one of these layers remains searchable, but it is less useful for rigorous benchmark selection.

3. Lifecycle governance, quality assurance, and maintenance

The strongest formal treatment of registry governance appears in BetterBench, which models the benchmark lifecycle in five stages: design, implementation, documentation, maintenance, and retirement (Reuel et al., 2024). Design addresses tested capability, task-capability mapping, real-world importance, use cases, metric choice, baselines, and interpretation. Implementation addresses evaluation code, accessible data or environments, replication scripts, contamination protections, and working-code indicators. Documentation addresses construction process, metric definition, limitations, provenance, requirements, licensing, and standardized metadata. Maintenance addresses code usability checks, feedback channels, and responsible contact information.

This lifecycle perspective matters because many benchmark failures are not conceptual but infrastructural. BetterBench reports that 14 out of 24 assessed benchmarks did not perform multiple evaluations of the same model or report statistical significance or uncertainty, 17 out of 24 did not provide easy-to-run scripts to replicate results from the original paper, only 3 out of 24 implemented a build status or equivalent, and standardized metadata was nearly absent in practice (Reuel et al., 2024). These findings recast the registry problem: the objective is not merely discovering benchmarks, but discovering whether a benchmark is runnable, auditable, and interpretable.

Public artifact release is therefore a core registry function. HPC AI500 states that the specification and source code are publicly available; AIBench Scenario releases specifications, source code, testbed, results, running and profiling scripts, and deployment templates; AIBench Training makes specification, source code, and performance numbers available; and Milabench and AI-NativeBench are open source with released repositories and datasets (Jiang et al., 2019, Gao et al., 2020, Tang et al., 2020, Delaunay et al., 2024, Wang et al., 14 Jan 2026). A plausible implication is that a registry entry should track not only citation and benchmark definition, but also artifact maturity.

The “living repository” model deepens this governance role. BetterBench explicitly supports community feedback, corrections if benchmarks update or stakeholders disagree, and continuous expansion with newly assessed benchmarks (Reuel et al., 2024). That moves benchmark registries away from static benchmark lists and toward maintained infrastructures analogous to software package indexes or standards registries.

4. Metric regimes and ranking philosophies

A central lesson of the literature is that benchmark registries cannot assume a universal scoring semantics. Different benchmark families encode different notions of performance, and a registry must preserve those metric philosophies rather than flatten them into a single scoreboard.

HPC AI500 uses a multi-axis regime. For component benchmarks it includes training accuracy, validation accuracy, target accuracy, time-to-accuracy, cost-to-accuracy, and power-to-accuracy; for micro benchmarks it uses FLOPS, images/s where applicable, and also power- and cost-related metrics (Jiang et al., 2019). AIBench Training similarly uses time-to-quality as the wall-clock time required to train a model to a specified target quality, with subsets designed separately for repeatable performance ranking and workload characterization (Tang et al., 2020). These suites therefore treat quality-constrained efficiency as primary.

AIPerf takes the opposite design choice: a single major metric, OPS, intended to quantify AI computing power in a way analogous to LINPACK while remaining representative of end-to-end AI workflows. It also introduces a regulated score, defined as $Regulated~Score = -\ln(Error) \times OPS$, to combine throughput with model quality (Ren et al., 2020). Milabench adopts a weighted geometric mean over benchmark performances adjusted by success rate, explicitly penalizing incomplete support so that missing workloads hurt the global score (Delaunay et al., 2024). BetterBench’s scoring is different again: the object being ranked is the benchmark itself, not the model, and the aggregated quantities are design, implementation, documentation, maintenance, and usability scores (Reuel et al., 2024).

Agentic-system benchmarks add process-sensitive metrics. AI-NativeBench measures outcome correctness for deterministic tasks, pass rate for open-ended tasks, retry rate, trace alignment against a golden trace, tool precision and recall, latency decomposition, and token usage across direct success, retry success, and failure states (Wang et al., 14 Jan 2026). AgentSearchBench evaluates retrieval and reranking quality using execution-grounded relevance, reporting Precision, Recall, NDCG, and Completeness at cutoffs, with execution-derived labels rather than textual similarity alone (Wu et al., 24 Apr 2026).

This diversity creates a common misconception: that “benchmark score” is a uniform object. The literature rejects that view. Some benchmarks score hardware throughput, some score time-to-solution under quality constraints, some score workflow integrity, and some score the benchmark’s own usability. A registry is therefore most informative when it records not only metric names, but also whether the benchmark is quality-aware, throughput-oriented, trace-based, execution-grounded, or quality-assurance-oriented.

5. Domain-specific registry patterns

Registry design is strongly shaped by domain structure. In HPC AI, benchmark registries arise from the mismatch between conventional commercial AI benchmarks and scientific DL workloads. HPC AI500 organizes its suite around Extreme Weather Analysis, High Energy Physics, and Cosmology; distinguishes 2D sparse matrix, 2D dense matrix, and 3D matrix data diversity; and combines end-to-end component benchmarks with micro benchmarks for hotspot operators (Jiang et al., 2019). The registry value here lies in explicit scientific domain taxonomy, scalable reference software, and metrics beyond FLOPS.

In Internet-service benchmarking, AIBench and AIBench Scenario use scenario distillation. AIBench models end-to-end services as permutations of essential AI and non-AI components, while AIBench Scenario formalizes real-world applications as DAGs and reduces them by six rules that preserve essential branches, dominant behaviors, and representative models (Gao et al., 2020, Gao et al., 2020). Here the registry pattern is hierarchical: scenario benchmarks preserve critical paths, component benchmarks isolate AI modules, and micro benchmarks expose hotspot functions.

In accelerator procurement and hardware-software maturity analysis, Milabench uses a community-derived workload catalog. The suite’s main and optional benchmarks are selected to reflect research domain, model architecture, model size, parallelization pattern, and library stack, with YAML-based composition and per-benchmark package isolation (Delaunay et al., 2024). The registry function is operational: it supports procurement evaluation, diagnostics, and cross-vendor comparison.

In scientific AI more broadly, SAIBench treats registry composition itself as the central problem. SAIL decouples problem definitions, AI models, metric definitions, ranking rules, and software and hardware configuration into reusable modules; a planner discovers compatible tuples and executes them in isolated benchmarking pods (Li et al., 2022). This pattern generalizes beyond fixed benchmark suites, effectively making the registry a combinatorial benchmark-generation system.

In trustworthiness and compliance evaluation, the LLM benchmark survey organizes entries against the seven EU Trustworthy AI requirements: Human Agency and Oversight, Technical Robustness and Safety, Privacy and Data Governance, Transparency, Diversity Non-discrimination and Fairness, Societal and Environmental Well-being, and Accountability (Ivanov et al., 2024). COMPL-AI is described there as an open-source, compliance-centered evaluation framework and the first regulation-oriented LLM benchmarking suite based on EU Trustworthy AI requirements. The registry pattern is therefore policy-centered rather than workload-centered.

Agentic systems introduce two additional patterns. AI-NativeBench is application-centric and white-box, treating agentic spans as first-class citizens in distributed traces and organizing systems around MCP, A2A, and heterogeneous A2A variants (Wang et al., 14 Jan 2026). AgentSearchBench, by contrast, is a benchmark-and-registry for “AI agents in the wild,” built from nearly 10,000 real-world agents and formalized as retrieval and reranking under executable task queries and high-level task descriptions (Wu et al., 24 Apr 2026). The former catalogs workflows; the latter catalogs agents.

6. Limitations, misconceptions, and open problems

The literature identifies substantial unresolved issues for any benchmark registry. BetterBench shows that benchmark quality varies widely and that benchmark popularity does not imply benchmark usability or interpretability. Its comparison between MMLU, which received a weighted average of 5.5, and GPQA, which received 11.0, is used to show that model release reports often cite benchmarks side by side without clarifying that the benchmarks differ greatly in quality (Reuel et al., 2024). A benchmark registry that lists names without quality metadata can therefore reinforce misleading comparison practices.

A second limitation is coverage incompleteness. The LLM benchmark survey is explicit that explainability remains hard to assess with existing tools, privacy evaluation is underdeveloped and benchmark methodology is inadequate, some EU AI Act requirements are difficult to map cleanly to technical tests, and current coverage of corrigibility, human oversight, lifecycle monitoring, incident detection, post-deployment drift, causal compliance, auditability, and redress mechanisms is limited (Ivanov et al., 2024). This suggests that even a well-structured registry may index gaps as much as established practice.

A third issue is construct validity under system evolution. AI-NativeBench notes LLM nondeterminism, closed-source model drift, external validity limits, and the insufficiency of final-answer metrics alone for AI-native systems (Wang et al., 14 Jan 2026). AgentSearchBench adds that semantic similarity is an unreliable proxy for actual agent performance, that execution labeling still relies on LLM-as-a-judge, that evaluation is limited to the top-20 retrieved agents per query, and that parts of the task-construction pipeline are synthetic even though grounded in real agents and documentation (Wu et al., 24 Apr 2026). Registries for agentic systems therefore need to encode observability assumptions and evaluation provenance.

A fourth issue is the tension between standardization and scalability. AIPerf was proposed partly because LINPACK does not reflect AI workloads and MLPerf’s fixed problem sizes limit automatic scalability; its answer is AutoML-based weak scaling with a single OPS metric (Ren et al., 2020). Yet this design choice also illustrates the broader trade-off: a registry can privilege comparability through fixed definitions, or adaptability through scalable workloads, but not without affecting what its scores mean.

Taken together, these works imply that a mature AI benchmark registry must do more than enumerate benchmarks. It must specify scope, encode metric semantics, expose implementation and maintenance quality, preserve domain context, and make incompleteness explicit. In that sense, the benchmark registry becomes part of the epistemic infrastructure of AI evaluation: not a substitute for technical judgment, but the mechanism by which technical judgment becomes auditable, comparable, and reusable.