---
title: Taxonomy for Evaluation Criteria
url: https://www.emergentmind.com/topics/taxonomy-for-evaluation-criteria
type: topic
---

# Taxonomy for Evaluation Criteria

A taxonomy for evaluation criteria is a structured framework for systematically categorizing and describing the properties, processes, or metrics by which systems, models, or artifacts are evaluated. Such taxonomies promote rigor, comparability, and reproducibility in scientific and engineering domains, and facilitate the synthesis, communication, and advancement of evaluation practices across diverse fields. The specific schemes, dimensions, and terminology vary widely according to application area, but leading research establishes several converging principles on the organization and utility of evaluation taxonomies.

## 1. Fundamental Concepts and Objectives

Evaluation criteria taxonomies are developed to address the fragmentation and ambiguity that arise when diverse laboratories, research groups, or industrial actors introduce non-standardized or context-dependent quality metrics. The overarching aims are:
- To achieve comparability of evaluation results across studies;
- To guide practitioners in designing robust, repeatable, and interpretable evaluation procedures;
- To support regulatory and compliance activities by ensuring transparency of reported metrics.

Key properties of a well-constructed evaluation criteria taxonomy include unambiguous definitions, coverage of both qualitative and quantitative aspects, hierarchical structuring by frame of reference, and adaptability across tasks or domains [2509.22064, 2403.00111, 2410.17281].

## 2. Taxonomic Structures: Hierarchies, Dimensions, and Scoping

Several leading taxonomies organize evaluation criteria along principled axes or hierarchies:

| Taxonomy           | Primary Dimensions                                                               | Description                                            |
|--------------------|----------------------------------------------------------------------------------|--------------------------------------------------------|
| QCET [2509.22064]  | Frame of reference (output in its own right, relative to input, target, external), Quality type (correctness, goodness, feature), Aspect (form/content/whole) | Explicit hierarchical mapping for NLP evaluation       |
| TAI [2410.17281]   | TAI principle: fairness, robustness, privacy, transparency, agency, well-being, accountability | Seven-principle framework tied to EU guidelines        |
| SlideAudit [2508.03630] | High-level domain area (layout, typography, color, imagery, animation) → specific flaw category | Graphic design flaw detection                          |
| Cloud Eval. [1302.1957] | Performance Feature (property/capacity), Experimental Scene (environmental/operational) | Empirical decomposition for cloud benchmarking         |

Hierarchical taxonomies often assign unique node IDs to each criterion and provide named, domain-grounded definitions at each level. For example, QCET uses a triply nested structure (frame of reference → quality type → aspect) to map the numerous, sometimes ambiguous, “quality criterion names” in the NLP literature to precise meanings [2509.22064], thus resolving issues where “fluency” or “naturalness” as reported in different papers are non-comparable.

In design evaluation (e.g., SlideAudit), taxonomy development follows domain theory (e.g., Gestalt principles) with expert-driven iteration to divide subjective flaws into mutually exclusive, actionable categories [2508.03630].

In domains requiring multi-stakeholder or system-level assessment, taxonomy structuring may also integrate actor roles or system lifecycle mappings (e.g., AI safety frameworks mapping evaluation criteria to roles such as developer, deployer, or user, and to lifecycle stages) [2404.05388, 2410.17281].

## 3. Criteria, Metrics, and Measurement Approaches

Taxonomies not only name evaluation criteria but often specify measurement techniques, whether these are direct quantitative metrics, qualitative judgments, or composite indicators. Leading approaches distinguish between:

- **Internal vs. External Measurements**: Internal measures are derivable from the taxonomy or artifact itself (e.g., count of categories), while external measures are observed when the taxonomy is applied to real data [2403.00111].
- **Subjective vs. Objective**: Some measures require expert or user judgment (e.g., annotator fluency ratings in NLP or design flaw recognition), others are computationally defined (e.g., classification accuracy, average response time, semantic proximity via embeddings).
- **Direct vs. Derived**: Direct measures count features or apply tests directly, whereas derived measures calculate ratios, correlations, or statistics from observed data or internal properties.

Some frameworks, such as GANTEE, also blend discriminative metrics (effectiveness, e.g., mean rank, hit@k) with efficiency metrics (time to process) and correctness/precision metrics (accuracy, F1) [2303.14480].

Mathematical formulas underpin some criteria:
- For example, **robustness** in taxonomy assessment is measured as
  $$
  R(T) = \frac{\sum_{i=1}^{n_{groups}} \left(1 - \frac{n_{ic}}{n_{gc} \times (n_{ac} - n_{gc})}\right)}{n_{groups}}
  $$
  where $n_{ic}$ is the number of intruder characteristics, $n_{gc}$ is the number of characteristics in group $i$, and $n_{ac}$ is the total number of characteristics [2403.00111].
- Hierarchical precision and recall for taxonomy-aware evaluation in vision-language models are given by
  $$
  hP = \frac{1}{N} \sum_{n=1}^{N} \frac{|(p_n) \cap (t_n)|}{|(p_n)|}, \quad
  hR = \frac{1}{N} \sum_{n=1}^{N} \frac{|(p_n) \cap (t_n)|}{|(t_n)|}
  $$
  where $(p_n)$ and $(t_n)$ denote the path from root to predicted/true node [2504.05457].

## 4. Applications and Case Studies

Evaluation taxonomies perform foundational roles across scientific and engineering domains:

- **NLP and NLG**: The QCET taxonomy enables researchers to disambiguate “fluency,” “accuracy,” and related QCs, thereby supporting directly comparable results across independent experiment suites. When mapping a given study’s definition of “naturalness” to the QCET node for “Native Speaker Likeness,” ambiguities can be resolved, enabling meaningful cross-study synthesis and meta-evaluation [2509.22064].
- **AI Safety and Trustworthiness**: The seven-principle taxonomy from TAI frameworks allows regulatory or organizational actors to select relevant, reportable, and auditable metrics spanning fairness, robustness, privacy, transparency, human agency, societal impact, and accountability. It also captures trade-offs and interdependencies between criteria—such as how improvements in fairness sometimes reduce robustness [2410.17281, 2404.05388].
- **Design Evaluation**: SlideAudit’s taxonomy-driven detection of presentation slide flaws demonstrated that LLMs provided more precise remediation plans and higher flaw detection F1 scores (by 33%) when explicitly prompted using the taxonomy’s categories, underscoring the practical value of structured, standard criteria [2508.03630].
- **Performance Benchmarking**: Cloud evaluation taxonomies (e.g., property/capacity × env/operation scene) have standardized the decomposition and reporting of empirical results by defining all relevant configuration variables and what is being measured, e.g., availability, latency, scalability, variability [1302.1957].
- **Mental Health AI**: Psychotherapy AI risk taxonomies integrate clinical standards (e.g., DSM-5) into AI evaluation via immediate/potential risk trees, explicit cognitive model factor tracking (e.g., change in “hopelessness” by ≥1 on a 1–10 scale), and dynamic monitoring over conversational sessions [2505.15108].

## 5. Impact, Standardization, and Open Challenges

Standardized taxonomies for evaluation criteria have demonstrable impacts:
- **Repeatability and Comparability**: Clear mappings of criteria to uniquely defined nodes permit reliable meta-analysis and scientific reproducibility. Nonstandard or ambiguous QCs (e.g., “fluency” in NLP) have hampered progress due to lack of comparability; taxonomy-driven naming and definition mapping provide a direct solution [2509.22064].
- **Design Guidance and Auditability**: Taxonomies serve as checklists for new evaluation designs and as frameworks for audit and compliance, ensuring that system evaluations are exhaustive, transparent, and interpretable by third parties [2410.17281].
- **Domain Adaptability and Expansion**: Taxonomies must be extensible to new use-cases. For example, the taxonomy for cloud service evaluation supports smooth expansion (e.g., by appending security or cost dimensions as research demands), and AI safety frameworks integrate evolving lifecycle mapping to stakeholders [1302.1957, 2404.05388].
- **Trade-offs and Interdependencies**: A central challenge for taxonomy-based evaluation is the tension among competing dimensions (e.g., trade-offs between fairness and accuracy, extensibility and stability, conciseness and comprehensiveness). A robust taxonomy explicitly marks such interdependencies [2410.17281, 2403.00111].

Persistent barriers include uneven metric maturity across dimensions, lack of interdependency metrics in general use, and difficulties in quantifying certain qualitative, context-sensitive criteria (e.g., societal or environmental well-being in TAI, or personalization in NLE explanations) [2410.17281, 2507.10585].

## 6. Methodological Innovations and Future Prospects

Recent research introduces methodological advances:
- **Automatic, Label-Free Scoring**: RaTE leverages masked language models to produce label-free, reproducible evaluation of taxonomic structure, sidestepping the cost and inconsistency of manual evaluation and achieving strong correlation with human ranking [2307.09706].
- **LLM-Driven Hierarchical Evaluation**: LiTe segments large taxonomies into manageable subtrees, employs cross-validation and penalty mechanisms for extreme cases, and delivers robust, quantitative and qualitative assessment aligned with human expert annotation, supporting scalable, automated evaluation pipelines [2504.01369].
- **Multi-facet Assessment**: Modern taxonomies often split criteria across content, presentation, and user-centered properties to better capture real-world desiderata (e.g., correctness, compactness, actionability, personalization) [2507.10585].

A plausible implication is that evaluation criteria taxonomies will increasingly integrate automated measures with human-in-the-loop oversight, domain-theoretic grounding, dynamic extensibility, and computational traceability as sociotechnical systems grow in scale and impact.

## 7. Summary Table: Exemplar Taxonomy Structures

| Field/Task             | Taxonomy Reference         | Primary Axes/Branches                                                                       |
|------------------------|---------------------------|---------------------------------------------------------------------------------------------|
| NLP/NLG                | QCET [2509.22064]         | Four frames (own right, input-relative, target-relative, external) × Quality type × Aspect  |
| TAI                    | [2410.17281]              | Seven EU principles (fairness, robustness, etc.), subdivided into metric-based and qualitative |
| Slide Design           | SlideAudit [2508.03630]   | Dimensions (layout, typography, color, imagery, animation) → 19–27 categorized flaws         |
| Cloud Service Eval     | [1302.1957]               | Performance feature (property/capacity) × Experimental scene (environmental/operational)    |
| Pedagogical LLMs       | [2412.09416]              | Eight pedagogical dimensions (identification, location, guidance, actionability, etc.)       |
| AI Trust/Safety        | [2404.05388]              | Component and system-level attributes; safety guardrails; lifecycle mapping to stakeholders  |

These architectures exemplify the centrality of taxonomy for evaluation criteria: providing a common language, facilitating reproducibility and comparability, supporting audit and governance, and serving as a foundation for responsible technological innovation.

Source: https://www.emergentmind.com/topics/taxonomy-for-evaluation-criteria