Fine-Grained Outcome Taxonomy
- Fine-Grained Outcome Taxonomy is a structured representation that decomposes outcomes into constituent, interpretable dimensions, preserving distinctions masked by aggregate scores.
- It applies across domains like sports analytics, LLM diagnostics, and clinical AI, using hierarchical, role-based, and multi-label frameworks to capture nuanced performance data.
- The taxonomy enables actionable insights by mapping observable events to intermediate variables, improving targeted diagnostics, benchmarking, and intervention strategies.
A fine-grained outcome taxonomy is a structured, interpretable decomposition of performance, events, or labels into constituent dimensions that preserve distinctions suppressed by aggregate scoring or coarse classification. Across recent work, the term denotes closely related but domain-specific constructions: per-player rally-ending outcomes in table tennis video, ability dimensions for diagnostic evaluation of LLMs, safety-and-tool-routing intents for clinical agents, multi-label sexism subtypes in social media videos, hierarchy-aware correctness and specificity in vision-language classification, span-level hallucination types in knowledge-grounded generation, aspect trees for multimodal evaluation, and part-of-speech-driven error categories in learner writing (Zhang et al., 14 Apr 2026, Abdul et al., 22 Dec 2025, Seo et al., 26 Sep 2025, Grazia et al., 17 Feb 2026, Snæbjarnarson et al., 7 Apr 2025, Mishra et al., 2024, Hong et al., 19 May 2025, Ye et al., 28 Nov 2025). In all of these settings, the taxonomy functions as an explicit representational layer between raw observations and downstream judgment, enabling diagnosis, prediction, benchmarking, or intervention at a granularity finer than a single scalar outcome.
1. Conceptual definition and motivation
The most explicit formal definition appears in diagnostic LLM evaluation: a fine-grained ability/outcome taxonomy is “a structured, interpretable set of dimensions that decomposes task performance into constituent abilities spanning domain knowledge and cognitive processes,” with each benchmark item mapped to the abilities it engages via an item–ability association matrix, or Q-matrix (Zhang et al., 14 Apr 2026). The same underlying motive recurs elsewhere. Aggregate benchmark scores “compress diverse task demands into a single scalar,” thereby obscuring where a model succeeds or fails, limiting targeted improvement, ability-guided model selection, and benchmark design (Zhang et al., 14 Apr 2026). In multimodal evaluation, overall or loosely specified criteria similarly produce low consistency because annotators and automated judges implicitly weight different criteria such as creativity, logical consistency, factuality, grounding, style, and safety (Hong et al., 19 May 2025).
In computer vision and multimodal classification, the motivating problem is not only coarse evaluation but also loss of hierarchical semantic structure. In fine-grained visual classification, predictions such as “conifer” for “Norway spruce” are under-specific rather than simply wrong, so an evaluation scheme must capture both correctness and specificity with respect to a taxonomy (Snæbjarnarson et al., 7 Apr 2025). In knowledge-grounded text generation, hallucinations are not uniform; they range from entity-level contradictions to invented, subjective, and unverifiable content, each with different verification demands and different edit operations (Mishra et al., 2024).
Other domains use the taxonomy to make operational decisions rather than only retrospective evaluation. TACOS maps a user query to one of 21 classes that simultaneously encode safety level and tool requirements, thereby unifying guardrails and tool selection in a single intent-classification step for clinical agents (Seo et al., 26 Sep 2025). The Extended OpenTT Games dataset uses a per-player, side-aware outcome taxonomy at point end, tightly coupled to shot type and posture labels, so that models can move beyond event spotting toward tactical understanding of how strokes, posture, and context relate to rally endings (Abdul et al., 22 Dec 2025).
A plausible implication is that “fine-grained outcome taxonomy” now names a family of representational devices rather than a single formalism. Some variants decompose latent competence, some partition observable outcomes, some structure error types, and some define evaluation criteria. What they share is explicit granularity, interpretable label semantics, and a design objective of preserving distinctions that matter for diagnosis or action.
2. Structural patterns and design principles
Despite domain heterogeneity, recent taxonomies exhibit recurrent structural motifs. The first is hierarchy. TACOS begins with Unsafe versus Safe, then subdivides into Clinical versus Non-clinical, then into non-information-seeking and information-seeking branches, with the latter further refining tool dependencies (Seo et al., 26 Sep 2025). FineMuSe is three-level: binary Sexist versus Non-sexist at Level 1, overlapping subtype labels at Level 2, and rhetorical devices such as Irony and Humor at Level 3 (Grazia et al., 17 Feb 2026). FRABench and GenEval organize 112 aspects under an overall root split into Universal Aspects and Task-specific Aspects (Hong et al., 19 May 2025). FEANEL uses a priority-ordered multi-level hierarchy spanning single-word, inter-word, and discourse-level errors, with Punctuation Error specially prioritized between Contraction Error and Determiner Error (Ye et al., 28 Nov 2025).
The second motif is explicit encoding of role, perspective, or scope. In the table-tennis outcome taxonomy, rally-ending labels concatenate a player-side prefix and an outcome class, with the player side defined from the camera’s perspective as “left_” or “right_,” yielding tags such as left_winner or right_net (Abdul et al., 22 Dec 2025). In taxonomy-aware VLM evaluation, specificity is encoded by depth in a rooted tree, where exact match, correct ancestor, sibling near-miss, and wrong-subtree predictions are distinct outcome types relative to the gold label’s path (Snæbjarnarson et al., 7 Apr 2025). In diagnostic MIRT, scope is determined by the binary Q-matrix entry , which specifies whether item requires ability (Zhang et al., 14 Apr 2026).
The third motif is separation of overlapping but non-identical dimensions. FineMuSe makes Level 2 subtypes multi-label rather than mutually exclusive and annotates Irony and Humor independently as rhetorical overlays that can accompany either Sexist or Non-sexist content (Grazia et al., 17 Feb 2026). FavaBench separates contradictory errors by unit of error—entity, relation, or sentence—from non-contradictory but unsafe content—invented, subjective, or unverifiable—because these categories differ in verification effort and editability (Mishra et al., 2024). FRABench distinguishes universal quality dimensions such as Coherence, Clarity, Fluency, Fidelity, and Harmfulness from task-conditioned aspects such as Coverage for summarization, Terminology for medical VQA, or Spatial Alignment for text-to-image generation (Hong et al., 19 May 2025).
The fourth motif is interpretability through controlled naming and compact code systems. The OpenTT extension introduces shorthand codes such as lfsv for left_forehand_serve, n for neutral, and ln for left_net, explicitly to ensure consistency and speed during annotation (Abdul et al., 22 Dec 2025). FEANEL requires one primary type per edit chosen by a global priority order, even when multiple issues co-occur, while the explanatory feedback must enumerate all identifiable facets (Ye et al., 28 Nov 2025).
The following table summarizes representative structural instantiations.
| Domain | Unit decomposed | Structural form |
|---|---|---|
| Table tennis video (Abdul et al., 22 Dec 2025) | Rally endings, shot types, posture | Hierarchical, per-player, side-aware labels |
| LLM diagnostic evaluation (Zhang et al., 14 Apr 2026) | Knowledge and cognitive abilities | 35-dimensional taxonomy + Q-matrix |
| Clinical agents (Seo et al., 26 Sep 2025) | Safety-aware user intents | 21-class hierarchy encoding risk and tool needs |
| Sexism detection (Grazia et al., 17 Feb 2026) | Binary outcome, subtypes, rhetoric | Three-level hierarchical multi-label taxonomy |
| VLM evaluation (Snæbjarnarson et al., 7 Apr 2025) | Correctness and specificity | Rooted taxonomy with ancestor-path scoring |
| Hallucination analysis (Mishra et al., 2024) | Hallucination types | Two top-level classes split into six fine-grained types |
| Multimodal evaluation (Hong et al., 19 May 2025) | Quality aspects | Hierarchical aspect tree with 112 aspects |
| K–12 writing (Ye et al., 28 Nov 2025) | Error types and severity | POS-driven hierarchy with 29 mutually exclusive types |
This suggests that hierarchy is not merely a presentational convenience. It is often the mechanism by which the taxonomy supports abstraction, fallback, partial credit, or label aggregation.
3. Annotation, coding, and alignment to evidence
Fine-grained taxonomies are operationalized through annotation protocols that tightly align labels to observable units. In the Extended OpenTT Games dataset, shot annotations mark the single frame of racket–ball impact; if the exact contact instant is not captured at 120 fps, the subsequent frame is chosen, consistent with OpenTTGames (Abdul et al., 22 Dec 2025). Rally endings are annotated on the frame that semantically captures the decisive event within the interval bounded by the last stroke of the rally and the first serve of the next rally, leveraging existing event markers such as bounce, net, ball_above_net, and empty_event (Abdul et al., 22 Dec 2025). The coding workflow is two-step: stroke frames and temporary point markers are placed first, then revisited and expanded with full stroke, posture, and rally-ending codes (Abdul et al., 22 Dec 2025).
In FEANEL, the annotation unit is the edit extracted from minimally corrected learner essays. Experts first perform minimal-edit correction using GEC principles, then aligned edit units are produced with CLEME, and finally each edit receives a type, a 1–5 severity score, and a detailed explanation (Ye et al., 28 Nov 2025). The explanation format is itself standardized: evidence words from the error sentence are enclosed in 〈 〉, corrections in [ ], and multiple issues are numbered as ①, ② … (Ye et al., 28 Nov 2025). This is a notably stronger form of annotation schema than label-only tagging, because it constrains both diagnostic category and pedagogical feedback.
FavaBench similarly uses span-level annotation. Annotators highlight exact erroneous spans, assign one of six fine-grained hallucination types, and, for entity and relation errors, provide minimal corrective edits; subjective and unverifiable spans are marked for deletion, while sentence-level contradictions may require whole-sentence deletion or rewrite (Mishra et al., 2024). Evidence acquisition is explicit: annotators use Wikipedia and top search results, record URLs consulted, and for unverifiable labels record the top 10 URLs used to establish unverifiability (Mishra et al., 2024).
FineMuSe departs from single-span text annotation by requiring span-based, multimodal annotation. Annotators mark the temporal spans in text, audio, and video where each subtype appears, enabling co-occurrence labeling and modality-specific grounding (Grazia et al., 17 Feb 2026). The annotation process is three-step—text-only, audio-only, then full video—and uses two teams in a cross-check structure (Grazia et al., 17 Feb 2026). FRABench uses a different regime: aspect-level pairwise comparisons between response A and response B, with aspect-specific feedback and two integer scores in per response, strictly bound to the given aspect (Hong et al., 19 May 2025).
The Q-matrix in diagnostic ability taxonomies is an annotation artifact of another kind. It is constructed by an expert-supervised LLM pipeline: ability pool generation, hierarchical ability refinement with full expert review, and item tagging to final granularity, followed by 10% random spot checks and propagation of corrections (Zhang et al., 14 Apr 2026). In VLM taxonomy-aware evaluation, the analogous step is mapping free-form text predictions to taxonomy nodes through normalization, string checks, n-gram overlap, semantic similarity, and ambiguity resolution via ancestor voting (Snæbjarnarson et al., 7 Apr 2025).
These workflows show that fine granularity depends not only on rich label sets but also on precise temporal, span-level, or item-level anchoring. A plausible implication is that the annotation protocol often constitutes part of the taxonomy’s semantics: without consistent anchoring, label definitions can drift or become incomparable across instances.
4. Modeling formalisms and evaluation regimes
Fine-grained taxonomies support distinct modeling paradigms, but each paradigm uses the taxonomy as a structured intermediate variable. In diagnostic LLM evaluation, the Q-matrix constrains a multidimensional IRT model. The classical reference writes
with whenever (Zhang et al., 14 Apr 2026). The instantiated NeuralCD-style parametrization defines
feeds it to a non-negative MLP to preserve monotonicity, and estimates parameters by binary cross-entropy,
This produces continuous ability vectors that can predict unseen items within and across benchmarks (Zhang et al., 14 Apr 2026).
In table-tennis outcome prediction, the formulation is more conventional supervised classification. The dataset supports probabilistic models of rally endings conditioned on stroke and posture context, for example
with multi-class cross-entropy loss and evaluation by Accuracy, Precision, Recall, F1, macro/micro averages, confusion matrices, and optionally mAP for detection-style formulations (Abdul et al., 22 Dec 2025). The taxonomy thus defines both the target space and the conditioning variables.
In taxonomy-aware VLM evaluation, the core formalism is hierarchical precision and recall over ancestor sets. With gold node 0 and predicted node 1, define 2 and 3. Then
4
and
5
These measures explicitly separate correctness of included path content from specificity or coverage of the gold path (Snæbjarnarson et al., 7 Apr 2025). The same paper also discusses Wu–Palmer-type similarity, path-length similarity, and information-content-weighted variants (Snæbjarnarson et al., 7 Apr 2025).
FineMuSe and FEANEL exemplify multi-label and structured-generation evaluation, respectively. FineMuSe reports Accuracy for binary sexism detection and Precision, Recall, F1, Macro-F1, Valid Macro F1, Failure Rate, and Format Error Rate for fine-grained multi-label prediction (Grazia et al., 17 Feb 2026). FEANEL treats the task as ordered generation of severity, type, and explanation:
6
and evaluates type classification with Accuracy and Macro-F1, severity with MAE, and explanations with BLEU, METEOR, and ROUGE-L (Ye et al., 28 Nov 2025).
FRABench uses pairwise aspect evaluation. Each pair is labeled on multiple selected aspects, and evaluators return aspect-specific feedback and two scores in 7. The principal metric is “accuracy with ties,” with “diff” excluding ties; model ranking is assessed with Kendall’s 8 (Hong et al., 19 May 2025). FavaBench evaluates per-type sentence-level hallucination detection via precision, recall, and F1, averages F1 across six types, and also considers a binary variant that collapses hallucination types (Mishra et al., 2024).
A common misconception is that fine granularity necessarily implies one canonical metric. The literature instead shows metric pluralism: probabilistic latent-trait models, hierarchical set-overlap scores, pairwise preference alignment, macro-F1 for rare classes, and structured explanation metrics all coexist. What is common is not the estimator but the use of taxonomy-defined dimensions as the unit of judgment.
5. Representative domain instantiations
In sports analytics, the Extended OpenTT Games dataset operationalizes fine-grained outcomes at rally end through six coarse outcome classes—out, net, winner, not_hitting_ball, double_bounce, and miss_on_own_side—combined with left/right player prefixes, and links them to side-aware stroke types and posture labels such as front_heavy, back_heavy, both_feet_planted, and left_foot_lifted (Abdul et al., 22 Dec 2025). The dataset comprises 12 videos at 1920×1080 and 120 fps, with 1,457 strokes, 1,432 lean labels, 1,319 leg labels, and 281–282 rally endings, and the outcome distribution is explicitly imbalanced (Abdul et al., 22 Dec 2025). The taxonomy’s function here is tactical: it encodes not only what stroke happened but who did it, how the body was positioned at impact, and how the rally ended (Abdul et al., 22 Dec 2025).
In LLM diagnostics, the taxonomy is ability-centered rather than event-centered. The mathematics taxonomy contains 35 dimensions, divided into 22 Knowledge dimensions and 13 Cognitive dimensions, ranging from Properties of Numbers and Euclidean Geometry to Error Diagnosis, Robustness Analysis, Procedure Construction, and Conventional Modeling (Zhang et al., 14 Apr 2026). The framework generalizes to physics with 27 dimensions, chemistry with 58, and computer science with 12 (Zhang et al., 14 Apr 2026). Across 41 models, unseen-item prediction achieves within-benchmark AUC 0.80–0.89 and across-benchmark AUC 0.77–0.86, while cross-domain predictive AUC reaches 0.7664 in chemistry, 0.7925 in physics, and 0.7824 in computer science (Zhang et al., 14 Apr 2026). Here the taxonomy’s role is diagnostic and predictive: ability profiles explain performance variation and support targeted training or ability-guided model selection.
In clinical AI, TACOS defines 21 classes that map user queries to safety thresholds and tool dependencies. Unsafe classes include Adversary, Prompt Leakage, Private Information Leakage, High Medical Risk, Medical Crime, and Self Harm; safe clinical information-seeking classes include General Inquiry, Patient Inquiry, Medical Inquiry, App Inquiry, and their combinations such as Patient Medical App Inquiry (Seo et al., 26 Sep 2025). Unsafe classes are to be blocked, warned, and logged; safe non-information-seeking clinical classes prompt empathic or redirective responses; safe information-seeking clinical classes determine whether patient data access, medical reference RAG, app APIs, or combinations thereof are required (Seo et al., 26 Sep 2025). The taxonomy here is neither merely descriptive nor evaluative; it is a control policy interface.
In social media moderation, FineMuSe defines Sexist and Non-sexist top-level labels, Level 2 subtypes such as Stereotypes, Denial of inequality and rejection of feminism, Discrimination, Objectification, Counter-speech, and Reported sexism, and Level 3 rhetorical devices of Irony and Humor (Grazia et al., 17 Feb 2026). The dataset includes 828 Spanish videos from TikTok, BitChute, and YouTube Shorts, with multimodal annotation over transcripts, audio, and frames (Grazia et al., 17 Feb 2026). The taxonomy is explicitly multi-label because sexist expressions frequently overlap, and visual cues can carry meanings not recoverable from text alone (Grazia et al., 17 Feb 2026).
In vision-language evaluation, the taxonomy is a semantic class hierarchy rather than a label ontology of errors or intents. The key distinction is between exact answers, correct ancestors, sibling near-misses, and wrong-subtree predictions, all relative to the gold node’s ancestor path (Snæbjarnarson et al., 7 Apr 2025). On a human-annotated iNaturalist21 subset, CLIP-t2t plus hybrid mapping yields HP 0.79, HR 0.82, HF 0.80, and exact node accuracy 47.1%, while CLIP-i2t plus hybrid mapping yields HP 0.80, HR 0.81, HF 0.80, and exact node accuracy 44.0% (Snæbjarnarson et al., 7 Apr 2025). The taxonomy thus enables principled partial credit.
In factuality analysis, FavaBench splits hallucinations into contradictory statements and non-contradictory but unverifiable content. Contradictions are localized as entity-level, relation-level, or sentence-level; non-contradictory problematic content is Invented Information, Subjective Statement, or Unverifiable Statement (Mishra et al., 2024). The benchmark reports that 59.8% of ChatGPT outputs, 70.2% of Llama2-Chat 7B outputs, and 64.9% of Llama2-Chat 70B outputs contain at least one hallucination in the sampled information-seeking setting (Mishra et al., 2024). The taxonomy here is closely tied to verification effort and edit strategy.
In multimodal evaluation, FRABench’s 112-aspect hierarchy spans text quality, safety and fairness, image quality, multi-image consistency, text-image relationship, advanced alignment categories such as Comparison and Negation, and numerous task-specific aspects such as Coverage, Relevance, Terminology, Feasibility, and Safety (Hong et al., 19 May 2025). FRABench comprises 60.4k pairwise samples with 325k aspect-level labels across 28 sub-tasks and four settings: Natural Language Generation, Image Understanding, Image Generation, and Interleaved Text-and-Image Generation (Hong et al., 19 May 2025). The taxonomy’s contribution is to make evaluation criteria explicit and portable across tasks and modalities.
In educational assessment, FEANEL defines 29 mutually exclusive error types, each edit receiving one primary type by priority, a severity from 1 to 5, and structured explanatory feedback (Ye et al., 28 Nov 2025). The benchmark contains 1,000 essays and 8,676 fine-grained analyses, with 3,005 edits in the elementary subset and 5,671 in the secondary subset (Ye et al., 28 Nov 2025). Here the taxonomy supports both error analysis and pedagogy.
6. Limitations, controversies, and methodological tensions
A recurrent limitation is imbalance or sparse coverage. The OpenTT extension reports strongly imbalanced rally-ending distributions, with double_bounce and miss_on_own_side especially rare, and explicitly advises stratified splits, re-weighting, focal loss, or label aggregation (Abdul et al., 22 Dec 2025). Diagnostic LLM evaluation notes that weak validity signals often arise from sparse coverage of abilities with fewer than 10 items, motivating benchmark redesign with balanced Q-matrix coverage (Zhang et al., 14 Apr 2026). FineMuSe reports that Objectification is scarce, likely due to platform moderation of sexual content, and that Irony and Humor are underrepresented (Grazia et al., 17 Feb 2026). FEANEL likewise shows lower Macro-F1 than accuracy because long-tail categories such as Contraction, Number, Auxiliary, PoS Confusion, Sentence Structure, and Format are difficult for models (Ye et al., 28 Nov 2025).
A second tension concerns annotation subjectivity. TACOS does not provide inter-annotator agreement statistics, and some class boundaries—General Inquiry versus Medical Inquiry, Patient Inquiry versus Patient Medical Inquiry, Redirective or Symptomic versus Patient Inquiry—are described as genuinely difficult rather than mere model failures (Seo et al., 26 Sep 2025). FineMuSe reports stronger agreement for binary labels than fine-grained ones, with irony and humor showing poor to moderate agreement because of subjectivity (Grazia et al., 17 Feb 2026). FavaBench reports 75.1% agreement on whether a sentence contains any hallucination but only 60.3% agreement for exact error-type detection at the sentence level (Mishra et al., 2024). FEANEL does not report formal IAA metrics despite its expert-driven process (Ye et al., 28 Nov 2025).
A third issue is taxonomy completeness and representational bias. FavaBench explicitly excludes common-sense, numerical, and logical reasoning errors from its hallucination taxonomy, restricting scope to factual errors given external world knowledge (Mishra et al., 2024). The OpenTT extension does not explicitly label forced versus unforced errors or separate service faults and lets into dedicated outcome classes (Abdul et al., 22 Dec 2025). TACOS uses categorical safe/unsafe thresholds rather than numeric risk scores, and the dataset is not publicly released because it contains private data (Seo et al., 26 Sep 2025). FRABench acknowledges that reliance on GPT-4o-generated labels for many sub-tasks may propagate source-model biases (Hong et al., 19 May 2025).
A fourth issue is the trade-off between granularity and usability. TACOS argues that under-specification merges clinically distinct tool requirements, while over-specification can degrade performance on unrelated intents; its toxic-subtype case study found that collapsing nine toxic subtypes into a single class improved overall 21-class performance (Seo et al., 26 Sep 2025). FEANEL resolves multi-categorial ambiguity through a single-label priority order, but this simplification can move secondary phenomena from label space into explanation space (Ye et al., 28 Nov 2025). VLM taxonomy-aware evaluation shows that exact match alone underestimates useful under-specific predictions, yet hierarchical metrics can also reward overly generic answers with high hierarchical precision but low hierarchical recall (Snæbjarnarson et al., 7 Apr 2025).
A plausible implication is that fine-grained taxonomies are not neutral descriptive objects. They encode decisions about scope, observability, admissible overlap, hierarchy depth, aggregation, and operational purpose. Debates about “the right taxonomy” are therefore often debates about intended use: diagnosis, safe routing, pedagogical feedback, partial-credit evaluation, or tactical modeling.
7. Significance and emerging directions
Across domains, the chief significance of fine-grained outcome taxonomies is that they convert opaque outputs into interpretable intermediate variables. In LLM diagnostics, ability profiles enable targeted training, ability-guided model selection, and ability-aware benchmark design (Zhang et al., 14 Apr 2026). In clinical agents, class labels directly drive blocking, warning, logging, and deterministic tool orchestration (Seo et al., 26 Sep 2025). In multimodal evaluation, aspect-level judgments improve consistency and transfer, while revealing systematic weaknesses such as poor compositionality in image generation and instruction inconsistency in visual LLMs (Hong et al., 19 May 2025). In educational technology, fine-grained type, severity, and explanation labels create learner profiles and support targeted interventions (Ye et al., 28 Nov 2025).
Several concrete future directions recur. OpenTT recommends task-specific splits, label consolidation for rare types, and, if more labels are added, inter-annotator agreement measurement (Abdul et al., 22 Dec 2025). Diagnostic LLM evaluation identifies balanced, ability-aware benchmarks, higher expert-review rates for item tagging, and richer cognitive diagnosis models beyond monotonic logistic MIRT as priorities (Zhang et al., 14 Apr 2026). FineMuSe points toward better multimodal fusion and targeted data collection for rare rhetorical and visual categories (Grazia et al., 17 Feb 2026). FRABench proposes extension to video and audio modalities and retrieval-augmented evaluation to improve visual grounding (Hong et al., 19 May 2025). FEANEL suggests broader age groups, languages, and richer human-centered metrics for explanation quality (Ye et al., 28 Nov 2025).
Taken together, these works indicate that fine-grained outcome taxonomies are becoming infrastructural components of contemporary AI datasets, benchmarks, and agent systems. They formalize distinctions that coarse labels erase, and they do so in ways that can be computationally modeled, audited, and reused. This suggests that future progress in evaluation, safety, diagnosis, and interactive systems will depend not only on stronger models, but also on better taxonomic design.