Capabilities Score: Foundations & Applications

• Capabilities Score is a metric that quantifies how well a process or system meets target objectives by comparing performance against defined tolerances and benchmarks.
• It is applied in diverse fields—from manufacturing and computer systems to AI evaluation—using methods like z-score normalization, geometric means, and entropy measures.
• Its design incorporates local operational constraints and statistical adjustments to enable practical performance optimization and transparent decision-making.

A capabilities score is a scalar or vector-valued metric designed to quantify the degree to which a process, agent, system, or model can realize relevant objectives over a target space of conditions, inputs, or requirements. Its construction, interpretation, and significance differ across domains, reflecting local operational constraints, statistical properties, and theoretical motivations. The concept has been systematically formalized in manufacturing quality assurance, computational system optimization, autonomous agent evaluation, educational measurement, high-stakes human decision pipelines, advanced LLM assessment, and emergent complex AI behaviors.

1. Foundations and Purposes of Capabilities Scores

A capabilities score characterizes the extent to which a system or agent produces outputs within an acceptable region given intrinsic variability or environmental complexity. In traditional process engineering, this quantifies how well a process delivers outputs within engineering specification limits, providing a benchmark for quality improvement, supplier selection, and operational risk (Saha et al., 2015). In computational system management, it reflects how closely resource utilization aligns with optimal targets for given workloads, guiding adaptive scheduling and deployment (&&&1&&&). In AI agent evaluation, a capabilities score operationalizes multidimensional generalization and task competence, often under conditions of sparse or hierarchical achievement (Hafner, 2021). The definition is thus deeply contextual: the score’s design is bound to the specific performance envelope and failure semantics central to the domain.

2. Classical and Generalized Capability Indices in Process Engineering

In manufacturing and process control, the "capability score" (process capability index, PCI) formalizes the relationship between process spread/centering and specified tolerances (Saha et al., 2015). For a quality variable $X \sim N(\mu, \sigma^2)$ within limits $[L, U]$, the canonical indices are:

• Potential Capability: $C_p = \frac{U-L}{6\sigma}$ captures the ratio of allowed spread to actual process spread; $C_p = 1$ corresponds to 99.73% of the process output falling within specifications.
• Performance with Centering: $$C_{pk} = \min\{\frac{U-\mu}{3\sigma}, \frac{\mu-L}{3\sigma}\}$$ quantifies process shift toward either limit.
• Empirical counterparts ($P_p$, $P_{pk}$) replace population parameters with corresponding sample estimates.

For non-normal or discrete distributions, percentile-based generalizations (Clements, Mukherjee, Borges & Ho, Perakis & Xekalaki) replace $6\sigma$ with percentile intervals, defect counts, or directly observed conformance rates. Multivariate capability is addressed by transforming $X \in \mathbb{R}^v$ to a univariate score using structural functions (e.g., $N(x) = \sum a_ix_i$), or by computing ellipsoidal coverage ratios. Table 1 summarizes representative indices:

Name	Formula	Use Case
$C_p$	$\frac{U-L}{6\sigma}$	Potential capability
$C_{pk}$	$$\min \{\frac{U-\mu}{3\sigma}, \frac{\mu-L}{3\sigma}\}$$	Centering impact
$C'_{p}$	$\frac{U-L}{\xi_{1-\alpha} - \xi_\alpha}$	Non-normal, percentiles
$C_{gp}$	$\frac{F(U) - F(L)}{F(UDL) - F(LDL)}$	Generalized (arbitrary $F$)

Statistical inference for these scores employs $\chi^2$ confidence intervals, delta-method expansions, or bootstrapping for empirical indices.

3. Capabilities Scores in Computer Systems and Resource Management

Within computer systems, the WISE framework provides a parametric, resource-centric capabilities score for workload–machine pairs (Luciano et al., 2020). For $n$ resources, each with observed utilization $x_i$, target $\mu_i$, tolerance $\sigma_i$, hard limit $r_i^{max}$, and weight $w_i$, the per-resource normalized score uses either $\tanh$ or $\exp$ scaling of the z-score $z_i = (x_i - \mu_i)/\sigma_i$. Four aggregation schemes ($\mathcal{S}_1$–$\mathcal{S}_4$) combine L1/L2 norms and normalization modes, with penalties for hard-limit violations to reflect resource saturation or critical constraint breaches.

Optimal capabilities occur when all resources track their targets without exceeding bounds, as $\mathcal{S}_3$, $\mathcal{S}_4 \to 1$ (best) and $\mathcal{S}_1$, $\mathcal{S}_2 \to 0$ (best). Empirical studies demonstrate alignment with cost/performance-optimal machine choices.

4. Evaluation in AI, Education, and Holistic Decision Contexts

4.1 Agent Capabilities in Open-World RL.

Crafter operationalizes agent capabilities by enumerating $N$ discrete achievements, each denoting a nontrivial skill, exploration, or planning milestone (Hafner, 2021). The overall capabilities score $S$ is computed as a geometric mean of per-achievement success rates $s_i$:

$$S = \exp \left( \frac{1}{N} \sum_{i=1}^N \ln(1 + s_i) \right) - 1$$

This scoring property accentuates breadth and penalizes neglect of rare skills, making it preferable to arithmetic means in sparse or compositional domains.

4.2 Differential Index for Assessment Rater Capability.

For educational raters, the normalized differential index $\bar\kappa_r$ measures the expected sensitivity of a rater’s pass/fail rate to student ability, integrating over the ability distribution and normalizing against the supremum achieved by a "perfect" rater (Wang et al., 13 Feb 2025). In the generalized multifacet Rasch model,

$$\bar\kappa_r = \Delta^{-1} \rho_r \int_{-\infty}^\infty \mu_{nr}(1-\mu_{nr})\phi(\theta)d\theta$$

where $\rho_r$ is rater discrimination, $\eta_r$ severity, and $\phi$ is the standard normal density. This construction jointly captures discrimination and alignment.

4.3 Composite Multimodal Scoring for Human Decision Pipelines.

The CAPS score fuses standardized academic, essay, and extracurricular sub-scores via convex combination of weights derived from expert, regression, and tree ensemble methods (Zeng et al., 12 Jul 2025). Each component is fully decomposable and interpretable, and the final holistic score is additive up to an equity adjustment to enable transparent, fair admissions evaluation.

5. Modern LLM and Multimodal Model Capability Scoring

5.1 Multi-View, Multi-Dimensional Evaluation (CAPability).

Modern evaluation of multimodal LLMs uses the CAPability benchmark, where correctness (Precision) and thoroughness (Hit) are quantified for each of twelve visual and semantic dimensions (Liu et al., 19 Feb 2025). The macro-averages over these dimensions define the overall capabilities score, and the harmonic mean (H-score) reflects balanced performance. Additionally, the "know but cannot tell" ($K\bar{T}$) metric exposes gaps between latent model knowledge as probed by QA and its free-form caption output.

Metric	Formula
Precision	$$|S(\mathrm{COR})| / ( |S(\mathrm{COR})| + |S(\mathrm{INC})| )$$
Hit	$|S(\mathrm{COR})| / |S(\mathrm{ALL})|$
$K\bar{T}$	$$|S_{qa}(\mathrm{COR}) \cap [S(\mathrm{INC})\cup S(\mathrm{MIS})]| / |S_{qa}(\mathrm{COR})|$$

Empirical results indicate strong model performance on global/object-centric tasks and systematic weakness on dynamic, thoroughness, or relational dimensions.

5.2 Cognition–Domain–Task (CDT) for LLMs.

For instruction-tuned LLMs, CDT tags each example as a triplet $(c,d,t)$ for cognition, domain, and task (Mo et al., 29 Sep 2025). Dataset-level capability is quantified by:

• Coverage: Fraction of composite skill–domain–task tuples realized.
• Balance: Shannon entropy of the distribution over observed triples.

There is a demonstrated strong positive correlation between these composite measures and downstream performance, and greedy dataset construction strategies exploiting these metrics efficiently optimize for model capability.

6. Task- and Risk-Oriented Capability Metrics in Contemporary AI Safety

RepliBench assesses autonomous replication capability through pass@$k$ metrics, with hierarchical aggregation across domains—obtaining resources, acquiring model weights, deploying on compute, and persistent operation—reflecting threat-relevant capabilitiy vectors (Black et al., 21 Apr 2025). For task family $i$ and variant $j$:

$$\text{pass@}k_i^j = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

Domain, family, and overall mean pass@10 scores provide empirical visibility into current agent strengths and bottlenecks, and recursive AND/OR tree compositions yield conservative overall risk-centric capability scores.

7. Methodological Considerations, Limitations, and Domain-Specificity

Capabilities scores share common statistical, methodological, and interpretability themes:

• Normalization and Aggregation: Geometric means, macro-averages, and entropy-based normalizations mitigate bias toward frequent or easy achievements.
• Unidimensional vs. Multidimensionality: Domain context dictates whether a single index suffices (as in $C_p$) or high-dimensional, view-specific profiles are essential (as in CAPability or CDT).
• Statistical Inference and Confidence: Confidence intervals, hypothesis tests, and parameter estimation (e.g., Laplace approximations in educational indices) substantiate the robustness of scores.
• Transparency and Interpretability: Contemporary systems, notably in high-stakes human decision pipelines, integrate SHAP analyses and modular decomposability into scoring procedures (Zeng et al., 12 Jul 2025) to ensure auditability and fairness.

Limitations include the sensitivity to rare-task weighting, difficulty of capturing inter-task or inter-view dependencies, the indirectness of some capability proxies (as in geometric means or composite coverage), and the challenge of measuring emergent behaviors entailing cross-domain compositionality.

---

References:

(Saha et al., 2015) (Luciano et al., 2020) (Hafner, 2021) (Wang et al., 13 Feb 2025) (Zeng et al., 12 Jul 2025) (Liu et al., 19 Feb 2025) (Mo et al., 29 Sep 2025) (Black et al., 21 Apr 2025)