Capability-Oriented Metric

• Capability-oriented metrics are quantitative measures designed to assess system abilities by evaluating coverage, thoroughness, and balance across diverse capability domains.
• They employ methodologies such as multi-dimensional tagging, simulation-based assessments, and adaptive testing to reveal strengths and weaknesses.
• These metrics provide actionable insights for improving model robustness, guiding targeted interventions, and aligning performance with downstream objectives.

A capability-oriented metric is a quantitative measure explicitly designed to capture the extent to which a system, model, or collection of agents possesses, manifests, or enables specific functional abilities aligned with downstream tasks. Through multidimensional tagging, structured aggregation, or simulation-based assessment, capability-oriented metrics depart from conventional scalar performance measures by tracking coverage, thoroughness, adaptability, and robustness of relevant capabilities. This approach is applied in domains such as machine learning, software engineering, behavioral agent modeling, and safety-critical perception, with methodologies tailored to the context and the nature of the observed system.

1. Conceptual Foundations

Capability-oriented metrics are motivated by the inadequacy of traditional scalar performance metrics (e.g., accuracy, BLEU, F₁) to capture the fine-structured ability distribution in complex systems. Unlike aggregate measures on standard benchmarks, which may overfit to superficial patterns or fail to reveal failure modes, capability-oriented metrics are constructed to diagnose what an agent or system can do, cannot do, or fails to express, often in a compositional or multi-view fashion. Key properties include:

• Disaggregation across capability axes (e.g., cognitive, domain, task, perceptual dimension).
• Explicit assessment of coverage (breadth), thoroughness (completeness), and balance (uniformity) over the targeted capability space.
• Alignment with system-level or downstream objectives, often via simulation or ontology-driven approaches.
• Support for cross-system comparison, robustness analysis, and targeted interventions.

2. Formal Structures and Methodologies

2.1 Orthogonal Tagging and Coverage Metrics

Capability-oriented metrics often employ a multi-dimensional tagging framework. For example, in the CDT framework for LLM evaluation (Mo et al., 29 Sep 2025), any data instance is annotated by a tuple (cognition, domain, task):

• $\mathcal{C}$: cognitive abilities ($N_c=18$, e.g., pattern recognition, quantitative reasoning).
• $\mathcal{D}$: domains ($N_d=33$, e.g., biology, law).
• $\mathcal{T}$: tasks ($N_t=16$, e.g., generation, summarization).

Capabilities are indexed as the set $\mathcal{F} = \{(c,d,t)\}$, and metrics include:

• Coverage:

$\mathrm{Coverage}(D) = \frac{|T_D|}{|\mathcal{F}|}$

where $T_D$ is the set of unique (c,d,t) tuples in dataset $D$.

• Balance (Entropy):

$$\mathrm{Balance}(D) = -\sum_{f \in T_D} p(f) \log p(f)$$

where $p(f)$ is the empirical frequency of $f$ in $D$.

Higher values signal greater breadth and uniform presence of composite capabilities, empirically correlated with improved downstream model performance (Mo et al., 29 Sep 2025).

2.2 Correctness, Thoroughness, and Expressivity

In vision-language evaluation, CAPability (Liu et al., 19 Feb 2025) introduces two complementary metrics:

• Precision (Correctness of what is said):

$$\mathrm{Precision} = \frac{|S(\mathrm{COR})|}{|S(\mathrm{COR})| + |S(\mathrm{INC})|}$$

• Hit (Thoroughness of coverage):

$$\mathrm{Hit} = \frac{|S(\mathrm{COR})|}{|S(\mathrm{ALL})|}$$

Where $S(\mathrm{COR})$ is the set of dimensions mentioned correctly, $S(\mathrm{INC})$ incorrectly, $S(\mathrm{MIS})$ missed, and $S(\mathrm{ALL})$ their union.

Additionally, the $K\bar{T}$ metric quantifies latent but unexpressed knowledge by comparing QA and free-form caption outputs:

$$K\bar{T} = \frac{|S_{qa}(\mathrm{COR}) \cap [S(\mathrm{INC}) \cup S(\mathrm{MIS})]|}{|S_{qa}(\mathrm{COR})|}$$

2.3 Difficulty-Conditional Capability Metrics

For supervised classification, the Machine Learning Capability (MLC) metric (Kline et al., 2023) integrates case difficulty via Item Response Theory (IRT) and Computer Adaptive Testing (CAT):

• Each case is assigned a difficulty index (CDI, $\theta_s$) by fitting a 2PL or graded-response model to feature–outcome relationships.
• CAT is used to probe the upper bound of model reliability as difficulty increases, sampling minimal test instances to reach statistical confidence.
• The MLC for each class is:

$$\mathrm{MLC} = \frac{H}{L} + \ln\left(\frac{R}{W}\right)$$

where $H$ is cumulative CDI, $L$ number of iterations, $R/W$ correct/incorrect, with modifications to avoid singularities.

3. Capability-Oriented Metrics in System Evolution and Adaptation

Software evolution studies leverage capability-oriented metrics to quantify and localize change (S et al., 2020). For aspect-oriented systems, the maturity index and complementary change metric per artifact type $X$ (Aspect, Pointcut, Advice, Class, Method):

• $$\mathrm{MI}_X = \frac{X_c - (X_a + X_m)}{X_c}, \quad C_X = 1 - \mathrm{MI}_X$$ with $X_c$ total in current version, $X_a$ added, $X_m$ modified. High $C_X$ flags heavy churn and evolutionary stress, while low values signal stabilization in system capability.

4. Simulation and Downstream-Oriented Evaluation

In safety-critical autonomous driving, frame-based accuracy is insufficient for operational safety assessment (Sato et al., 2022). Instead, capability is measured by the system’s ability to maintain proper trajectory under closed-loop control:

• End-to-End Lateral Deviation (E2E-LD):

$$\mathrm{E2E-LD} = \max_{0 \leq t \leq T_E} |L_t - C_t|$$

quantifies the maximum deviation from lane center over simulated driving with perception–planning–control in the loop.

• Per-frame Simulated Lateral Deviation (PSLD) is a lightweight proxy measuring ability to recover from perception errors, correlating strongly with E2E-LD and reflecting true control capability.

5. Ontological and Agent-Based Capability Modelling

Agent-oriented approaches formalize capability via behavioral ontologies and structured attribute aggregation (Greer, 2014):

• Each behavior $B$ is scored along:
  • Ability $(B^A)$, Flexibility $(B^F)$, Coordination $(C^{OR})$, Cooperation $(C^{OP})$, Communication $(C^{OM})$, all $\in [0,1]$.
• Entity Complexity:

$EC(B) = \frac{I(B) + COL(B)}{2}$

where $I(B) = \frac{B^A + B^F}{2}$, $COL(B) = \frac{C^{OR} + C^{OP} + C^{OM}}{3}$.

• Problem Success Likelihood:

$PSL = \frac{\sum_{B \in \mathrm{PBS}} EC(B)/n}{PC}$

Nested/compound behaviors and decision rules are incorporated via And/Or combinators, accumulating bounds.

6. Comparative Analysis and Key Empirical Results

Capability-oriented metrics exhibit superior diagnostic power compared to traditional metrics across contexts:

Domain	Traditional Metrics	Capability-Oriented Metric	Empirical Findings
LLM evaluation	Scalar accuracy, BLEU	Multidim. Coverage, Balance, QA-Caption Hit	Higher correlation with downstream performance (Mo et al., 29 Sep 2025)
Visual captioning	BLEU, CIDEr	Precision, Hit, $K\bar{T}$	Models are precise yet non-thorough; $K\bar{T}$ exposes latent knowledge (Liu et al., 19 Feb 2025)
ML classification	Accuracy, AUC	MLC (CAT+IRT based)	MLC needs <1% data, robust to case difficulty (Kline et al., 2023)
Lane detection for AD	Frame pixel accuracy, F₁	E2E-LD, PSLD	Traditional metrics anti-correlate with true driving quality (Sato et al., 2022)
AO software evolution	Churn rate (diffs)	Maturity and change indices per artifact	Localizes evolutionary hotspots (S et al., 2020)

In all cases, capability-oriented metrics reveal weaknesses, bottlenecks, and actionable priorities absent from aggregate traditional scoring.

7. Implications and Future Directions

Capability-oriented metrics enable principled evaluation, model selection, and targeted improvement by:

• Guiding fine-grained data augmentation and pretraining focused on underrepresented or poorly covered capability dimensions (Liu et al., 19 Feb 2025, Mo et al., 29 Sep 2025).
• Enabling efficient, individualized decision support and confidence estimation in deployed systems (Kline et al., 2023).
• Supporting robustness analysis over evolutionary or adversarial scenarios (Sato et al., 2022, S et al., 2020).
• Stimulating development of interpretable modular architectures and benchmarks with explicitly compositional capability taxonomies.

These metrics are now integral to the evaluation pipeline for LLMs, perception systems, agent teams, and safety-critical control, and drive ongoing research in multidimensional performance quantification, simulation-based assessment, and adaptive data selection. Extension to further domains may involve deeper integration with causal inference, explicit representation of capability dependencies, and real-time monitoring in dynamic operational contexts.