CapabilityBench: Benchmarking Models & Hardware

• CapabilityBench is a suite of formal frameworks defining capability functions to evaluate models and hardware against executable requirements.
• It replaces single-number metrics with parameterized, context-specific evaluations, using registries, predictive modeling, and latent embeddings.
• The framework ensures transparent, auditable benchmarking by combining machine-readable policies, detailed reporting, and community-driven policy packs.

CapabilityBench is a suite of formal frameworks, registries, and methodologies for capability-oriented benchmarking, enabling systematic, transparent, and predictive evaluation of models against executable requirements or capability functions. Across AI model evaluation, quantum computing, and foundation model transfer tasks, CapabilityBench demotes single-number aggregate metrics in favor of parameterized, context-specific measurement tied directly to deployment-critical requirements or performance predictions. Its instantiations span public registries for LLM requirements satisfaction (Ball, 15 Dec 2025), predictive modeling in quantum hardware (Hothem et al., 2023), and relative capability encoding for remote sensing foundation models (Adorni et al., 6 May 2025).

1. Conceptual Foundations and Definitions

CapabilityBench, across all contexts, formalizes the notion of a “capability function” $s$—mapping an input object (e.g., string prompt, quantum circuit, downstream task) to a well-defined performance metric, such as success probability or requirement adherence. Benchmarking with CapabilityBench proceeds by (i) specifying explicit, machine-readable requirements or performance metrics, (ii) evaluating models or hardware against these specifications, and (iii) providing granular, auditable reports at the level of individual requirements or capability dimensions.

In LLM evaluation, CapabilityBench operationalizes capability measurement as the degree to which models satisfy context-dependent, executable policies (expressed in CPL) over specified test prompts, shifting the focus from “intelligent” aggregates to actionable capability verdicts (Ball, 15 Dec 2025). In quantum computing, CapabilityBench denotes a general methodology for constructing predictive models $\epsilon(c;\theta)$ tied to benchmarking data ($\hat s(c)$), enabling extrapolation beyond observed circuits (Hothem et al., 2023). In remote sensing, it provides a low-cost predictive framework via latent-space encoding of models and tasks, predicting relative performance gaps without exhaustive fine-tuning (Adorni et al., 6 May 2025).

2. Architectural Frameworks and Core Components

LLM Registry

For LLMs, CapabilityBench is implemented as a three-tier registry and evaluation service (Ball, 15 Dec 2025):

• Frontend (UI/UX): Catalog and browse policy packs (manifest, core/extended policies, test cases), trigger model evaluations, and visualize results—adherence profiles, per-policy violation rates, and downloadable evaluation reports.
• Backend and Orchestration: Backend services manage policy pack storage, execution of evaluations (model output extraction, PredicateGraph structuring, CPL verification), and runtime aggregation of adherence metrics. Submission and CI automation pipelines ensure correctness and reliability of policy definitions and corresponding test suites.
• Data Model: Central tables track policy packs, policies (with tiers and scope), test cases, models, and granular evaluation results—facilitating reproducibility and versioned auditing.

Example REST API Endpoints

Verb	Endpoint	Description
GET	/packs	List all policy packs
GET	/packs/{pack_id}	Retrieve details for a policy pack
POST	/packs	Submit a new policy pack
GET	/models	List all registered models
POST	/evaluate {model_id, pack_id}	Trigger evaluation for model and policy pack
GET	/models/{model_id}/packs/{pack_id}/results	Retrieve per-case, per-policy results

Quantum Capability Modeling

In quantum benchmarking, CapabilityBench is instantiated through model fitting pipelines—linking empirical benchmark outputs to interpretable or expressive capability models, e.g., parameterized error rate models or neural predictors (Hothem et al., 2023). The architecture encapsulates:

• Benchmark Execution: Collect performance metrics on ensembles of quantum circuits.
• Model Fitting: Select a predictive form (ERM or neural network), optimize parameters $\theta$ to approximate the empirical capability function.
• Prediction and Generalization: Use the trained model to extrapolate benchmark results onto unseen circuits, supporting robust prediction for deployment scenarios.

Shared Latent Space for Foundation Models

In remote sensing foundation models, CapabilityBench structures all models and tasks as points in a common latent space, using distance to encode relative capability gaps. The components include (Adorni et al., 6 May 2025):

• Data Preprocessing: Normalize observed model-task performance into relative gaps.
• Geometric Embedding: Simultaneously learn latent vectors for models and tasks by minimizing deviation between predicted (distance) and actual normalized gaps.
• Prediction: Embed new models/tasks with a small calibration budget for performance forecasting across untested settings.

3. Specification Languages and Policy Formalisms

PredicateGraph Schema and CPL

In LLM evaluation, model outputs are first structured as PredicateGraphs: JSON objects representing entities, claims, operations, tool calls, citations, and code blocks. Capabilities are then specified in the Capability Policy Language (CPL), which defines:

• Tiering: T1 (objective correctness), T2 (safety/governance), T3 (structural preferences), with different override semantics.
• Scope and Assertion: Each policy targets a subset of PredicateGraph elements and asserts an expression over them, with deterministic, terminating logic (arithmetic, comparison, aggregation, collection operations).
• Violation Handling: Configurable actions such as CORRECT (with correction hint) or FLAG are invoked on policy failures.

Example Policy (core, LLMs):

{ "id": "policy.tool.calc_matches", "tier": "T1", "scope": { "kind": "tool_call", "filter": { "name": "calc" } }, "assert": [ { "expr": "tool_call.arguments.value == last(operations).output" } ], "on_violation": { "action": "CORRECT", "correction_hint": "Use exact computed value" } }

4. Evaluation Methodologies and Metrics

All CapabilityBench variants define explicit, fine-grained evaluation metrics tied to specified requirements or capability models:

• LLM Adherence Metrics (Ball, 15 Dec 2025):
  • Violation Rate: $$VR = \frac{1}{N P} \sum_{i=1}^{N}\sum_{p=1}^P v_{i,p}$$
  • Core Adherence: $$A_{core} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}( \forall p \in \mathcal{P}_{core}, v_{i,p} = 0 )$$
  • Extended Adherence: Similar, with $\mathcal{P}_{core} \cup \mathcal{P}_{ext}$
  • Capability Score: $C = w_{core} A_{core} + w_{ext} A_{ext}$
  • Violation Distribution: $f_p = \frac{1}{N}\sum_{i=1}^N v_{i,p}$
• Quantum Capability Modeling (Hothem et al., 2023):
  • Error Rates Model (ERM): Predicts capability as a function of elementary operation error rates, fitted to observed benchmark outcomes.
  • Neural Predictors: Leverage transfer learning (ResNet50) with custom circuit-to-image encodings.
• Foundation Model Gap Encoding (Adorni et al., 6 May 2025):
  • Latent Embedding Distance: $f(x_m, x_t) = \| x_m - x_t \|_2$ predicts the normalized performance gap $\Delta_{m,t}$
  • Objective: Minimize $$L(\{x_i\}) = \frac{1}{|M||T|}\sum_{m \in M, t \in T} (\| x_m - x_t \|_2 - \Delta_{m,t})^2$$

All routines are backed by empirically validated metrics (e.g., $\mathrm{RMSE} \approx 0.15$ for relative performance gap prediction in remote sensing), deterministic verification (LLM policies), or interpretable parameterization (quantum ERMs).

5. Policy Packs, Community Practices, and Workflows

In the LLM instantiation, CapabilityBench supports open, versioned submission of policy packs—bundles of policies, test cases, and manifests—by community experts (Ball, 15 Dec 2025). Submission involves:

1. Authoring and validation of CPL definitions and tests.
2. Automated CI checks: schema validation, termination/linting, dry-runs against baselines.
3. Maintainer review and semantic validation.
4. Versioned publishing and immediate availablity for evaluation.
5. Automatic backfilling of results on prior model-pack pairs upon pack updates.

Initial policy packs include arithmetic/tool-use, code-safety, citation compliance, and customer support, each with core and extended policies.

Workflow supports maintenance, auditing, and reproducibility analogous to established software engineering practices, replacing opaque aggregate scores with actionable, traceable verdicts.

6. Comparative Context and Extension to Predictive and Relative Benchmarking

CapabilityBench is positioned as a departure from legacy intelligence benchmarks (aggregate accuracy or “intelligence” proxies) toward operational requirement satisfaction. Key comparative results:

• For LLMs, standard DPO training yielded plateaued violation rates (~13%), while CAPE methods (evaluated via CapabilityBench) achieved 2.5% (Ball, 15 Dec 2025).
• Core adherence reached 99.7% versus 96% for code-safety policies.
• Explicit, contextually grounded verification (contextual objectivity) yielded inter-annotator agreement (kappa ≈ 0.98) versus subjective preference alignment (kappa ≈ 0.4).
• Community-driven policy specification decreased annotation costs by 5–20×.

In quantum computing, CapabilityBench closes the gap between descriptive (historical summarization) and predictive (out-of-distribution generalization) benchmarks, demonstrating scalability from interpretable error-rate models to hybrid neural architectures (Hothem et al., 2023).

In foundation model evaluation for remote sensing, CapabilityBench formalizes shared latent representations for models/tasks, producing practical, low-cost predictions for model selection and new-task forecasting, closing ~70% of the gap to oracle (fine-tuned) performance at a fraction of the computational cost (Adorni et al., 6 May 2025).

7. Limitations, Open Problems, and Future Directions

CapabilityBench frameworks are limited by:

• For relative performance encodings, predictions are inherently bounded within the span of observed results; extrapolation beyond the literature or outside observed model-task pairs is not supported (Adorni et al., 6 May 2025).
• Accuracy of fine-grained policy verification is gated by extraction and verifier fidelity (Ball, 15 Dec 2025).
• Embedding-based predictions degrade with sparse calibration (recommended ≥5–10 finetuning observations per model or task).

A plausible implication is that future research may focus on integrating task and model metadata, hybridizing interpretable and subsymbolic prediction methods, and expanding coverage across modalities and deployment contexts. Active selection for calibration data and incorporation of higher-order model-task interactions represent promising extensions (Adorni et al., 6 May 2025).

CapabilityBench establishes an evidence-based, requirements-first paradigm across multiple domains, prescribing infrastructure, languages, and metrics to align benchmarking practice with operational objectives and context-specific model requirements.