Hierarchical Deep Evaluation

• Hierarchical Deep Evaluation is a paradigm that organizes evaluations into multiple levels, ensuring parent–child consistency and capturing fine-grained performance nuances.
• It employs structured techniques like taxonomy-based classification, multi-fidelity search, and criteria decomposition to refine both automated and human assessments.
• Practical applications span document, image, and artifact evaluation, where hierarchical metrics reveal nuanced strengths and weaknesses undetectable by flat methods.

Hierarchical Deep Evaluation denotes a family of evaluation procedures in which assessment is explicitly organized over a hierarchy rather than a flat label space, a single undifferentiated score, or a one-shot model judgment. Across the literature, the hierarchy may be a taxonomy of classes, a staged evaluation pipeline, a tree of human criteria, a multi-fidelity search process, or a decomposition of generated artifacts into object-, part-, and material-level units. What remains constant is the attempt to measure performance at multiple levels simultaneously, preserve parent–child or coarse–fine consistency, and distinguish near-misses from structurally distant errors (Kowsari et al., 2017, Yuan et al., 2021, Liu et al., 2024, Narsupalli et al., 2024, Xie et al., 2023, Zhang et al., 7 Aug 2025, Sani et al., 10 Mar 2025, Kosmopoulos et al., 2013).

1. Conceptual foundations

In hierarchical classification, the underlying structure is commonly represented as a rooted tree or DAG $H=(V,E)$ with parent–child relations, ancestors, descendants, and distances defined over the taxonomy. For an instance with true labels $Y \subseteq V$ and predicted labels $\hat{Y} \subseteq V$, flat evaluation ignores whether errors remain within the correct branch, whereas hierarchical evaluation uses paths, ancestors, or distances to quantify structural closeness (Kosmopoulos et al., 2013). This distinction is operational in text classification, where HDLTex treats document classification as a two-level hierarchy with Level 1 parent domains and Level 2 child sub-fields, and in image classification, where LH-DNN and Hier-COS assess both fine predictions and their agreement with coarser levels (Kowsari et al., 2017, Fiaschi et al., 2024, Sani et al., 10 Mar 2025).

A central design principle is that deeper levels usually require finer discrimination than upper levels. In HDLTex, Level 1 comprises 7 top-level fields in the Web of Science dataset, while Level 2 comprises 134 sub-categories distributed under those 7 parents as $\{17,16,19,9,11,53,9\}$; the method therefore evaluates domain recognition and sub-topic specialization separately and jointly (Kowsari et al., 2017). In LH-DNN, labels are organized as a rigid parent–child tree, each example follows exactly one path from root to leaf, and evaluation includes both per-level accuracy and cross-level coherency rates such as $(1 \text{ vs } 2)$, $(2 \text{ vs } 3)$, and $(1 \text{ vs } 3)$ (Fiaschi et al., 2024). In Hier-COS, hierarchy-aware representations are intended to ensure that semantically closer classes lie closer in feature space, so that mistakes are less severe and coarse-level predictions remain consistent with leaf predictions (Sani et al., 10 Mar 2025).

This same logic extends beyond classification. HESGA treats evaluation itself as a hierarchy, separating inexpensive lower-level fast evaluation from expensive higher-level full evaluation during GNN hyperparameter optimization (Yuan et al., 2021). ReFeR organizes multiple LLM judges into a two-level review structure with peers and an area chair (Narsupalli et al., 2024). TencentLLMEval decomposes human-aligned LLM capability into a three-level task tree of major areas, categories, and tasks (Xie et al., 2023). Hi3DEval formalizes “hierarchical validity” for 3D assets at object, part, and material subject levels (Zhang et al., 7 Aug 2025). A plausible implication is that Hierarchical Deep Evaluation is best understood as a general evaluation paradigm rather than a single metric or model family.

2. Structural patterns of hierarchical evaluation

Several recurring architectural patterns appear across the literature.

Setting	Hierarchy	Principal evaluation outputs
HDLTex	Level 1 parent domains and Level 2 parent-specific child models	Level 1 accuracy, Level 2 accuracy, overall hierarchical accuracy
HESGA	Lower-level fast evaluation and higher-level full evaluation	$\Delta F(1,t)$ and RMSE
ReFeR	Peer Review Body and Area Chair Module	Spearman’s $\rho$, Kendall’s $\tau$, final analysis and rating
TencentLLMEval	Major areas, categories, tasks	win_rate, Excellent rate, Elo
Hi3DEval	Object level, part level, material subject level	dimension scores and unweighted overall sums
Hier-COS	Taxonomy-consistent feature space and preference-based ranking	HOPS, HOPS@k, FPA

In model-stacked hierarchical classification, the upper level routes inputs to specialized lower-level models. HDLTex first predicts a parent domain and then routes the document to one of 7 separate Level 2 models trained only within that parent domain. The operational path is therefore

$Y \subseteq V$0

with $Y \subseteq V$1 in the reported setting, and overall correctness requires both levels to be correct (Kowsari et al., 2017). The related DHC framework for e-commerce likewise shares representations top-down through a Hierarchical Embedding Network and adds a Hierarchical Loss Network to penalize adjacent-layer inconsistencies (Gao et al., 2020).

In multi-fidelity search, the hierarchy is temporal and resource-based rather than taxonomic. HESGA defines a lower level using interrupted training and a higher level using complete training. The fast score is

$Y \subseteq V$2

where $Y \subseteq V$3 is validation RMSE at epoch $Y \subseteq V$4, typically with $Y \subseteq V$5 at 10–20% of the maximum number of epochs. Only the top $Y \subseteq V$6 fraction of offspring by fast score are promoted to full evaluation, while an elite archive preserves high-performing individuals (Yuan et al., 2021).

In criteria decomposition and collective judgment, the hierarchy is evaluative rather than predictive. HD-Eval recursively decomposes a task into finer-grained criteria and trains a white-box aggregator over criterion-level scores (Liu et al., 2024). ReFeR uses three peer evaluators, an optional critic, and an area chair that samples $Y \subseteq V$7 responses and averages the ratings,

$Y \subseteq V$8

to improve correlation with human judgments (Narsupalli et al., 2024). HDCEval adopts a divide-and-conquer evaluator for medical LLM outputs, invoking three expert evaluators aligned to Patient Question Relevance, Medical Knowledge Correctness, and Expression, each with sub-aspect rubrics on a 0–5 scale (Zheng et al., 12 Jan 2025).

In hierarchy-aware representation learning, the hierarchy is embedded directly into the feature space. Hier-COS maps deep features into a vector space defined by a fixed orthogonal frame aligned with the taxonomy tree, and class prediction is made by selecting the leaf subspace with the largest projected energy (Sani et al., 10 Mar 2025). LH-DNN instead constrains shared-feature gradients with projection operators

$Y \subseteq V$9

so that lower-priority heads cannot increase higher-priority losses to first order (Fiaschi et al., 2024).

3. Metrics, objectives, and aggregation schemes

The metric design space is broad, but the dominant distinction is between per-level evaluation, pair-based or set-based hierarchy-aware metrics, and preference-based ranking metrics.

For hierarchical text classification, HDLTex reports Level 1 accuracy, Level 2 accuracy, and overall hierarchical accuracy, where overall correctness requires both levels to be correct. The paper also gives a composite cost/accuracy expression, but empirical reporting is centered on those three accuracy measures rather than precision, recall, or F1 (Kowsari et al., 2017). DHC similarly reports layer-wise accuracy at the coarse and fine levels, while explicitly noting that path accuracy and hierarchical precision/recall are not reported (Gao et al., 2020). LH-DNN augments per-level top-1 accuracy with cross-level coherency rates and motivates path-level metrics for rigid hierarchies (Fiaschi et al., 2024).

For generic hierarchical classification, the classical metric literature distinguishes pair-based and set-based views. Pair-based evaluation assigns a cost $\hat{Y} \subseteq V$0 based on shortest-path distance in the hierarchy and solves an optimal pairing problem; set-based evaluation augments predictions and truths with hierarchical context, such as ancestor sets, and then computes precision, recall, F1, or symmetric-difference loss (Kosmopoulos et al., 2013). The same paper proposes Multi-label Graph Induced Accuracy,

$\hat{Y} \subseteq V$1

and LCA-based hierarchical precision, recall, and F1 to address over-penalization, alternative DAG paths, and multi-label pairing problems (Kosmopoulos et al., 2013).

For hierarchy-aware ranking evaluation, Hier-COS argues that Mistake Severity, Average Hierarchical Distance, hierarchical precision/recall, MRR, MNR, and NDCG@k have important failure modes. It introduces HOPS and HOPS@k, based on LCA-derived preference tiers and an exponentially weighted penalty over rank inversions, with

$\hat{Y} \subseteq V$2

and $\hat{Y} \subseteq V$3 equal to top-1 accuracy by design (Sani et al., 10 Mar 2025).

For human and LLM-based evaluation, metrics are matched to the evaluation protocol. TencentLLMEval uses pairwise comparison labels A better, B better, equally good, and equally bad, and defines

$\hat{Y} \subseteq V$4

It also reports Excellent rate for single-model scoring, evaluator-bias screening through the GSB score and a $\hat{Y} \subseteq V$5-score threshold, and Elo ratings for automated pairwise ranking with $\hat{Y} \subseteq V$6 (Xie et al., 2023). ReFeR reports Spearman’s $\hat{Y} \subseteq V$7 and Kendall’s $\hat{Y} \subseteq V$8 between model-generated evaluations and human annotations (Narsupalli et al., 2024). HD-Eval uses Pearson $\hat{Y} \subseteq V$9 and Spearman $\{17,16,19,9,11,53,9\}$0 when regressing human scores from criterion-level signals (Liu et al., 2024). HDCEval reports pairwise accuracy, Pearson correlation, ICC, and human-evaluated Reference Match for rationales (Zheng et al., 12 Jan 2025).

For artifact evaluation, Hi3DEval uses absolute per-dimension scores rather than pairwise-only judgments. Object-level overall score is defined as

$\{17,16,19,9,11,53,9\}$1

and material-level overall score as

$\{17,16,19,9,11,53,9\}$2

Training combines Smooth L1 regression with a pairwise ranking loss,

$\{17,16,19,9,11,53,9\}$3

while part-level scores are reported per part rather than folded into the object-level leaderboard score (Zhang et al., 7 Aug 2025).

4. Representative application domains

Hierarchical document classification is a prototypical use case. HDLTex frames WOS-46985 as 46,985 documents with 7 parent domains and 134 child sub-fields, trains one Level 1 model on all data and 7 Level 2 models conditioned on the parent domain, and reports that the best stack on WOS-46985 is RNN/RNN with overall accuracy 76.58, Level 1 accuracy 90.45, and Level 2 accuracy 84.66. On WOS-11967, the best overall stack is RNN/DNN with overall 86.07, Level 1 93.98, and Level 2 91.58; on WOS-5736, the best overall stack is CNN/CNN with overall 90.93, Level 1 98.47, and Level 2 92.34 (Kowsari et al., 2017). DHC applies a related hierarchy-aware design to e-commerce category prediction and reports gains over SVM, FastText, TextCNN, HSVM, and HiNet, including Query-Category leaf accuracy 73.37 and Title-Category leaf accuracy 69.02 (Gao et al., 2020).

In image classification, LH-DNN evaluates three-level hierarchies on CIFAR-10, CIFAR-100, and Fashion-MNIST. On CIFAR-10, the larger LH-DNN reports level-1 accuracy 97.36%, level-2 accuracy 88.74%, level-3 accuracy 84.59%, and coherency $\{17,16,19,9,11,53,9\}$4 of 98.80%; on CIFAR-100 it reports 75.07%, 64.00%, 52.47%, and 88.69%; on Fashion-MNIST it reports 99.78%, 96.69%, 93.34%, and 99.92% (Fiaschi et al., 2024). Hier-COS evaluates deep label hierarchies ranging from 3 to 12 levels and reports state-of-the-art hierarchical performance across all datasets, while also improving top-1 accuracy in all but one case (Sani et al., 10 Mar 2025).

Hierarchical evaluation is also used to allocate computation in search. HESGA applies a two-level evaluation strategy to GC and MPNN hyperparameter optimization on ESOL, FreeSolv, and Lipophilicity. The lower level uses early-stage RMSE improvement, the higher level uses final RMSE, and the total cost is approximated by

$\{17,16,19,9,11,53,9\}$5

With $\{17,16,19,9,11,53,9\}$6, $\{17,16,19,9,11,53,9\}$7, and $\{17,16,19,9,11,53,9\}$8, the paper derives a speedup of about $\{17,16,19,9,11,53,9\}$9 relative to a baseline GA, corresponding to about 73% cost reduction, while reporting better or comparable final RMSE relative to Bayesian optimization on the tested datasets (Yuan et al., 2021).

For LLM and NLG evaluation, HD-Eval, ReFeR, TencentLLMEval, the Hierarchical Evaluation Framework for human evaluation, and HDCEval instantiate distinct but related forms of hierarchical deep evaluation. HD-Eval recursively decomposes criteria and trains a transparent aggregator over LLM-generated criterion scores, improving average SummEval Pearson correlation from about 0.538 for GPT-4 Eval to about 0.617 for HD-Eval-NN and Topical-Chat average Pearson from about 0.567 to about 0.616 (Liu et al., 2024). ReFeR uses three peer evaluators and an area chair and reports SummEval average $(1 \text{ vs } 2)$0 and $(1 \text{ vs } 2)$1, compared with the best baseline Analyze-Rate at $(1 \text{ vs } 2)$2 and $(1 \text{ vs } 2)$3 (Narsupalli et al., 2024). TencentLLMEval organizes human-aligned capability into 7 major areas, 200+ categories, and 800+ tasks, with a released test set of over 3,000 instances (Xie et al., 2023). HDCEval applies hierarchical divide-and-conquer evaluation to medical LLM outputs with REL, COR, and EXP evaluators and reports an overall 23.92% improvement in consistency with human evaluations compared to PandaLM (Zheng et al., 12 Jan 2025). The Hierarchical Evaluation Framework for human evaluation evaluates both inputs and outputs in a human-AI health-coaching MRC system, using sequential gates over relevance, factoidness, answerability, spelling, grammar, difficulty, clarity, relevance, clinical accuracy, and usefulness (Bojic et al., 2023).

For generative assets and multimodal understanding, Hi3DEval evaluates 3D assets at object level, part level, and material subject level, supported by Hi3DBench with 15,300 assets from 30 methods and a multi-agent annotation pipeline (Zhang et al., 7 Aug 2025). In video understanding, a different but related use of hierarchy appears in the YouTube-8M system combining frame-level sequence modeling and video-level classification, where HMoE and classifier chains exploit the 25 coarse verticals above 4,716 fine labels, and the final 18-model ensemble achieves GAP@20 of 0.84346 on public test and 0.84333 on private test (Tang et al., 2017).

5. Empirical regularities

A recurrent empirical regularity is that upper levels are easier than lower levels. HDLTex reports very high Level 1 accuracies, such as up to about 94% on WOS-11967, about 90% on WOS-46985, and about 98% on WOS-5736, while Level 2 accuracies are lower and overall hierarchical accuracy is lower still because both levels must be correct (Kowsari et al., 2017). LH-DNN shows the same pattern: coarse-level accuracy and coherency are high, while leaf-level accuracy is lower, especially on CIFAR-100 (Fiaschi et al., 2024). This suggests that hierarchical decomposition is useful precisely when label cardinality and semantic overlap make fine-grained discrimination difficult.

A second regularity is that hierarchy often improves evaluation quality by specialization or structured aggregation, but not every additional layer helps. In HESGA, early-stage RMSE improvement provides a useful promotion signal and yields equal or better final RMSE than Bayesian optimization on two of three datasets, yet the paper also notes that noisy early trends can misrank candidates (Yuan et al., 2021). In ReFeR, the full peer–chair hierarchy improves alignment to human judgments, but critic-based variants are consistently worse than the “No Critic” configuration on both SummEval and TopicalChat (Narsupalli et al., 2024). In video understanding, PLSTM improves over ATT+BiLSTM, while HLSTM underperforms single-level PLSTM, indicating that additional hierarchical recurrence can increase parameter count and training instability without guaranteed benefit (Tang et al., 2017).

A third regularity is that hierarchy-aware metrics can change system rankings relative to flat metrics. The unified study of hierarchical classification measures shows that flat accuracy, GIE, symmetric-difference loss, hierarchical F1, MGIA, and FLCA can rank the same systems differently, especially on multi-label DAGs (Kosmopoulos et al., 2013). Hier-COS further shows that AHD@k can remain unchanged under radically different orderings of the same top-$(1 \text{ vs } 2)$4 set, whereas HOPS is explicitly order-sensitive (Sani et al., 10 Mar 2025). A plausible implication is that “hierarchical deep evaluation” is inseparable from metric choice: a hierarchy-aware model evaluated with flat metrics can obscure the very behavior it was designed to improve.

A fourth regularity is that human-centered hierarchical evaluation exposes weaknesses not visible to automatic scores alone. TencentLLMEval reports average human agreement of 0.6392, with Reasoning at 0.8135 and Dialogue at 0.4949, and shows that GPT-4 judging has substantial agreement gaps relative to humans, especially in multi-turn dialogue and reasoning (Xie et al., 2023). The Hierarchical Evaluation Framework for MRC reports that only 63.8% of questions and 49.4% of answers satisfied all required criteria, and that input quality and output quality were significantly associated with $(1 \text{ vs } 2)$5 and $(1 \text{ vs } 2)$6 (Bojic et al., 2023). HDCEval reports that structured medical decomposition substantially improves alignment with doctors relative to generic evaluators (Zheng et al., 12 Jan 2025).

6. Failure modes, misconceptions, and limitations

One common misconception is that hierarchy automatically guarantees better final decisions. The literature is more cautious. HDLTex explicitly identifies error propagation: a wrong Level 1 prediction routes a document to the wrong Level 2 model and guarantees a wrong final label (Kowsari et al., 2017). HESGA notes that some architectures may learn slowly at the beginning and thus be unfairly penalized by $(1 \text{ vs } 2)$7, while aggressive hyperparameters can show large early drops yet generalize poorly (Yuan et al., 2021). LH-DNN observes a trade-off in which strict lexicographic priority can limit fine-level adjustments if they conflict with coarser levels (Fiaschi et al., 2024).

Another misconception is that adding more evaluators or more heads necessarily improves alignment. ReFeR finds that critic variants underperform the simpler no-critic configuration (Narsupalli et al., 2024). The YouTube-8M system shows that Multi-ATT underperforms Single-ATT and that HLSTM underperforms PLSTM (Tang et al., 2017). TencentLLMEval shows that even GPT-4, the strongest evaluated judge, is far from perfect on multi-dialogue and reasoning and that automated judging remains unreliable in those areas (Xie et al., 2023). These results indicate that hierarchical structure is beneficial only when the interfaces between levels are informative and stable.

Metric design also has well-documented pathologies. Ancestor-closure precision and recall can over-reward ancestor predictions and over-penalize deep nodes; pair-based costs can over-count shared paths; and some normalized variants remain sensitive to tree shape or prediction cardinality (Kosmopoulos et al., 2013). Hier-COS adds that AHD@k and related metrics can be permutation-invariant in top-$(1 \text{ vs } 2)$8 rankings and thus blind to severe ordering failures (Sani et al., 10 Mar 2025). This criticism is especially important in settings where evaluation is intended to distinguish “good near-misses” from “bad distant misses.”

Domain-specific constraints create additional limitations. Hi3DEval is object-centric and does not cover scene composition or dynamic content; part-level evaluation depends on segmentation quality and can be problematic for deformable or abstract shapes (Zhang et al., 7 Aug 2025). HDCEval does not report inter-annotator agreement and does not standardize a single scalar overall score (Zheng et al., 12 Jan 2025). The Hierarchical Evaluation Framework case study does not report a controlled comparison of time savings relative to a flat baseline (Bojic et al., 2023). HDLTex, DHC, LH-DNN, and Hier-COS all assume tree-structured or rigid hierarchies in their main formulations, while DAG or multi-path assignments require generalized constraints or metrics (Kowsari et al., 2017, Gao et al., 2020, Fiaschi et al., 2024, Sani et al., 10 Mar 2025).

7. Research directions

Several directions recur across the surveyed work. In hierarchical classification, explicit future directions include deeper hierarchies beyond two levels, hierarchical losses that penalize near-misses less severely, richer path-based or precision/recall-style metrics, and broader evaluation across domains and label structures (Kowsari et al., 2017). LH-DNN identifies DAGs and multi-path assignments as natural extensions of its projection-based training scheme (Fiaschi et al., 2024). Hier-COS proposes extensions to DAG taxonomies, kernelized variants, hierarchical neural collapse in subspaces, and dynamic or probabilistic hierarchies (Sani et al., 10 Mar 2025).

In evaluator alignment, the agenda centers on finer decomposition, stronger calibration, and broader modality coverage. HD-Eval formalizes iterative decomposition, attribution-guided pruning, and transparent aggregation as a reusable framework for aligning evaluators to human preferences (Liu et al., 2024). ReFeR suggests that generated analyses can be repurposed into instruction-tuning data for smaller evaluators (Narsupalli et al., 2024). TencentLLMEval points toward multilingual and real-world application coverage while emphasizing continued human oversight in dialogue and reasoning (Xie et al., 2023). HDCEval proposes extension to more clinical attributes, calibration, uncertainty estimates, and multimodal inputs (Zheng et al., 12 Jan 2025).

In artifact evaluation, Hi3DEval points toward scene-level and dynamic 3D evaluation, adaptive segmentation, and integration of scene graphs and physics-informed checks (Zhang et al., 7 Aug 2025). A plausible implication is that the next phase of Hierarchical Deep Evaluation will be less about proving that hierarchies matter and more about making them operationally robust: selecting the right level interfaces, choosing metrics that faithfully encode structural preferences, and aligning automated evaluation with expert judgment without losing interpretability.