---
title: Hierarchy-Aware Cross-Entropy (HACE)
url: https://www.emergentmind.com/topics/hierarchy-aware-cross-entropy-hace
type: topic
---

# Hierarchy-Aware Cross-Entropy (HACE)

Searching arXiv for recent and foundational papers on hierarchy-aware cross-entropy and closely related hierarchical classification losses.
I'll check arXiv for papers explicitly using the term "Hierarchy-Aware Cross-Entropy" and for foundational hierarchical cross-entropy formulations.
arXiv query: "Hierarchy-Aware Cross-Entropy"
Hierarchy-Aware Cross-Entropy (HACE) is a hierarchy-aware generalization of standard cross-entropy for classification problems in which labels are organized by a known taxonomy. In its explicit formulation, HACE is presented as a drop-in replacement for standard cross-entropy that incorporates a class hierarchy directly into the loss by combining prediction aggregation with ancestral label smoothing [2605.06274]. More broadly, the literature uses closely related but not always identical terminology: earlier work often refers to hierarchical cross-entropy (HXE), hierarchical loss, per-level loss, or sibling-smoothed cross-entropy, and some papers that are conceptually aligned with HACE do not use the term itself [1912.09393].

## 1. Terminology, scope, and naming

The term **Hierarchy-Aware Cross-Entropy** is used explicitly in "When Labels Have Structure: Improving Image Classification with Hierarchy-Aware Cross-Entropy" [2605.06274]. In that paper, HACE is defined as a classification loss that expands supervision from leaf labels to all nodes in a hierarchy, modifies the predicted distribution by upward aggregation through the taxonomy, and modifies the target distribution by redistributing mass along the ancestral path.

The surrounding literature is terminologically heterogeneous. "Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks" introduces **Hierarchical Cross-Entropy (HXE)**, which is a hierarchy-aware modification of cross-entropy based on a tree factorization of class probabilities and depth-dependent weighting [1912.09393]. "Harnessing Superclasses for Learning from Hierarchical Databases" introduces a **hierarchical loss** for supervised hierarchical classification on a tree of classes, but explicitly does **not** name that loss HACE [2411.16438]. "Hybrid Losses for Hierarchical Embedding Learning" uses a **Per-Level Loss (PL)** that applies standard multi-class cross-entropy at multiple hierarchy levels, which is strongly aligned with the HACE intuition but is not formally identical in mechanism [2501.12796]. "Hierarchy-Aware Fine-Tuning of Vision-Language Models" introduces **Hierarchy-Sibling Smoothed Cross-Entropy (HiSCE)**, again conceptually close to HACE but framed as a sibling-smoothed target construction within a larger hierarchy-aware fine-tuning objective [2512.21529].

A recurrent misconception is therefore to treat all hierarchy-aware classification losses as the same object. The literature instead supports a narrower usage in which **HACE** refers to the specific formulation of [2605.06274], while a broader family includes HXE, superclass-aware losses, per-level cross-entropy, sibling-smoothed cross-entropy, and related hierarchy-aware objectives.

## 2. Core mathematical construction

In the explicit HACE formulation, both the target distribution and the prediction are expanded from the leaf label space to the full set of hierarchy nodes [2605.06274]. Standard cross-entropy and HACE retain the same general form,
\[
\mathcal{L} = -\sum_i p(i)\log q(i),
\]
but differ in how \(p\) and \(q\) are defined.

For a hierarchy with \(N\) total nodes and logits \(z \in \mathbb{R}^N\), HACE first computes
\[
\hat{q} = \mathrm{softmax}(z) \in \mathbb{R}^N.
\]
It then applies **prediction aggregation**:
\[
q^*(i) = \hat{q}(i) + \sum_{j \in \mathcal{D}(i)} \hat{q}(j),
\]
where \(\mathcal{D}(i)\) is the set of descendants of node \(i\). Equivalently, using the reachability matrix \(\mathcal{R}\),
\[
q^* = \mathcal{R}\hat{q},
\]
with \(\mathcal{R}_{ij}=1\) iff \(j\) is a descendant of \(i\), or \(j=i\). This construction ensures that internal nodes accumulate the confidence of their descendants.

The target is built by **ancestral label smoothing**. For a ground-truth leaf \(a_0\), an optional standard label-smoothing step defines
\[
\tilde{p}(i) = (1 - \varepsilon)\mathbf{1}[i = a_0] + \frac{\varepsilon}{n},
\]
over the \(n\) leaf classes. HACE then redistributes this mass vertically along the ancestral path. If \(b_0, b_1, \ldots, b_k\) is the path from a leaf \(c\) to the root, then
\[
p^*(b_j) \mathrel{+}= d(1-d)^j \tilde{p}(c) \quad \text{for } j<k,
\]
and the remaining mass is assigned to the root:
\[
p^*(b_k) \mathrel{+}= \tilde{p}(c) - \sum_{j=0}^{k-1} d(1-d)^j\tilde{p}(c).
\]
The dilution parameter \(d \in (0,1]\) controls how much mass stays near the leaf versus being pushed upward.

The final HACE loss is
\[
\mathcal{L}_{\text{HACE}} = -\sum_i p^*(i)\log q^*(i).
\]

This mechanism has two distinct effects. Prediction aggregation makes the model’s output hierarchy-consistent in the sense that parent-node probabilities reflect the confidence of their full subtrees. Ancestral label smoothing makes internal nodes informative targets even when supervision is given only at the leaf level.

## 3. Relation to standard cross-entropy, HXE, and proper-scoring formulations

Relative to ordinary cross-entropy, HACE changes both sides of the loss. Standard cross-entropy uses a one-hot target over leaf classes and a flat softmax over the same leaf classes. HACE instead uses node-level predictions and node-level targets shaped by the hierarchy [2605.06274]. The explicit claim in that work is that prediction aggregation alone or ancestral label smoothing alone is not sufficient in the leaf-label setting; the two must be combined.

The earlier HXE formulation proceeds differently. HXE assumes the hierarchy is a tree, factorizes the probability of a leaf class \(C\) along its path to the root,
\[
p(C) = \prod_{l=0}^{h-1} p(C^{(l)} \mid C^{(l+1)}),
\]
and defines the loss
\[
\mathcal L_{\text{HXE}}(p, C) = -\sum_{l=0}^{h-1} \lambda(C^{(l)}) \log p(C^{(l)} \mid C^{(l+1)}),
\]
with a depth-based weight schedule
\[
\lambda(C) = \exp(-\alpha h(C)).
\]
In this sense, HXE is a hierarchy-sensitive reweighting of pathwise conditional cross-entropies rather than the prediction-and-target transformation used by HACE [1912.09393].

A different theoretical perspective is developed in "Harnessing Superclasses for Learning from Hierarchical Databases" [2411.16438]. There, classes are organized in a weighted rooted tree \(T=(\mathcal V,\mathcal E,\mathcal W)\), leaves are fine-grained classes, and internal nodes are superclasses. For a sample with true class \(y\), the loss is
\[
L_T(f(x;\theta),y) = -\sum_{j\in a(y)} w_j \log\Big(\sum_{k\in v_j} f_k(x;\theta)\Big).
\]
This loss uses the fact that all ancestors of \(y\) are partially correct. If the hierarchy is flat, it reduces to standard cross-entropy up to a constant factor \(1/2\). The paper proves that, when the weighting scheme is balanced, this hierarchical loss is a **proper scoring rule**, meaning its expectation is minimized by the true posterior class probabilities. It also proves that if the tree is not balanced, the loss is not a proper scoring rule.

That result is important for the broader HACE landscape because it shows that hierarchy-aware supervision can be formulated without introducing an obligatory trade-off between fine-grained and coarse-grained objectives. The paper explicitly states that superclass probabilities remain consistent because they are computed as sums of leaf probabilities:
\[
P(Y\in v_j\mid X)=\sum_{k\in v_j}P(Y=k\mid X).
\]
This suggests a theoretical contrast with approaches that use multiple independent heads or post hoc hierarchy corrections.

## 4. Hierarchy representation and implementation regimes

The main HACE formulation uses a **tree** in the main paper, while noting in the appendix that the method extends naturally to **DAGs** [2605.06274]. For trees, the reachability matrix \(\mathcal{R}\) provides an efficient implementation of upward aggregation as a single differentiable matrix-vector product. The paper also recommends precomputing both the reachability matrix and the ancestral label smoothing matrix \(\mathcal{T}\) once before training, so that minibatch target construction reduces to a gather operation.

A practical implication of this formulation is that the output layer dimension changes from the number of leaf classes \(n\) to the total number of hierarchy nodes \(N\). By contrast, the superclass-aware loss of [2411.16438] retains a standard classifier with a softmax output over leaf classes and computes superclass probabilities by summing leaf probabilities. That paper states that superclass aggregation can be implemented via matrix computations or as a very sparse, non-trainable linear layer over the softmax outputs, with negligible computational cost relative to standard cross-entropy.

Other hierarchy-aware variants use different implementation regimes. The following contrasts are explicit in the literature:

| Method | Hierarchy enters through | Prediction setup |
|---|---|---|
| HACE | Prediction aggregation and ancestral label smoothing | Output over all hierarchy nodes |
| HXE | Weighted conditional losses along a root-to-leaf path | Leaf probabilities, conditionals derived from them |
| Hierarchical loss | Sums of superclass probability masses over ancestors | Standard softmax over leaf classes |
| PL | Sum of ordinary cross-entropies at selected levels | Multiple prediction heads |

The **Per-Level Loss (PL)** in [2501.12796] performs standard multi-class classification at each tree level that contains more than one node:
\[
\sum\limits_{j=1}^{N}\mathcal{L}_{\text{multi-class}}(\hat{y}_{\ell_j},y_{\ell_j}).
\]
The hierarchy is encoded by decomposing the task into multiple levelwise classification problems. The paper explicitly notes that this is strong conceptual alignment with HACE, but not exact formulation equivalence.

In the VLM setting, HiSCE in [2512.21529] does not introduce a separate classifier head over all hierarchy nodes. Instead, it works in CLIP’s shared embedding space, where each class label at each hierarchy level is represented by a text embedding and logits are image-text cosine similarities. This establishes that hierarchy-aware cross-entropy-style supervision is not tied to a single architectural pattern.

## 5. Empirical behavior and benchmark evidence

The explicit HACE paper evaluates the method on **CIFAR-100**, **FGVC Aircraft**, and **NABirds**, in two regimes: end-to-end training across six architectures and linear probing on frozen **DINOv2-Large** features [2605.06274]. In end-to-end training, HACE improves accuracy over standard cross-entropy in **15 out of 18 architecture--dataset pairs**, with a **mean gain of 4.66\%** and a **maximum decrease of 1.57\%**. In linear probing, HACE outperforms all competing methods on all three datasets, with a **mean improvement of 2.18\%** over the next best baseline. Appendix results further state that HACE outperforms standard cross-entropy in **17 out of 18** end-to-end top-5 comparisons, with **mean improvement 4.46\%**.

The same paper reports that gains are largest on the deeper or more irregular hierarchies, specifically **FGVC Aircraft** and **NABirds**. An ablation on **NABirds** compares HACE without horizontal smoothing, HACE with standard label smoothing, standard cross-entropy, standard cross-entropy with label smoothing, LCA-based soft labels, and HXE. The main findings are that base HACE already beats standard cross-entropy variants, and that HACE plus standard label smoothing is best in **4 of 6 architectures**.

The earlier HXE literature emphasizes a different empirical pattern. On **tieredImageNet-H** and **iNaturalist’19**, HXE and hierarchy-aware soft labels improve hierarchy-sensitive metrics but often trace out an explicit trade-off against top-1 error [1912.09393]. For example, on tieredImageNet-H, increasing the HXE hyperparameter improves the hierarchical distance of mistakes while worsening top-1 error. That paper therefore frames HXE and soft labels as defining a better Pareto trade-off between top-1 error and mistake severity than prior methods, rather than as uniformly improving both.

The superclass-aware loss of [2411.16438] reports experiments on **iNaturalist19**, **TinyImageNet**, and **ImageNet-1K (downsampled)** with **ResNet50** and **MobileNetV3-Small**. For small training sets, the method and HXE both improve over cross-entropy in **hierarchical distance**. Its reported distinction is that the new loss achieves these hierarchical gains **without sacrificing standard accuracy**, reduces **coarse/gross errors**, and reduces the **Wasserstein distance** between predicted softmax mass and the true label. For larger datasets, the differences shrink.

This body of evidence supports two empirically distinct regimes. One line of work uses hierarchy-aware losses to improve the semantic quality of errors, sometimes at a cost in exact accuracy. Another line, including [2411.16438] and [2605.06274], reports hierarchy-aware gains without that coarse/fine trade-off under the tested settings.

## 6. Related variants, extensions, and broader generalizations

Several later or parallel methods preserve the central HACE intuition while altering the mechanism. In [2512.21529], **HiSCE** constructs a sibling-smoothed target table
\[
T^{(l)}_{ij} =
\begin{cases}
1 - \varepsilon_l, & \text{if } i=j,\\
\frac{\varepsilon_l}{|\mathcal{S}(i)|}, & \text{if } j \in \mathcal{S}(i),\\
0, & \text{otherwise,}
\end{cases}
\]
so that probability mass is redistributed only among sibling classes. This is combined with **Tree-Path KL Divergence (TP-KL)**, which concatenates levelwise predictions and penalizes mismatch with the full ground-truth path. The paper reports consistent improvements in **FPA (Full Path Accuracy)** and **TICE (Tree-based Inconsistency Error)** under CLIP-LoRA adaptation, with the strongest results obtained by the joint TP-KL + HiSCE objective.

In [2501.12796], the hierarchy-aware classification component is embedded inside a multi-task framework that combines PL, binary nodewise losses, and triplet loss. The paper reports that losses including PL outperform baselines on hierarchy-sensitive metrics by around **4–8% in MNR** and **2–4% in NDCG**, and that PL-based models achieve the strongest **Lowest Seen Ancestor** prediction performance for unseen classes. A plausible implication is that hierarchy-aware cross-entropy-style supervision can function not only as a classification objective but also as a structural prior on representation geometry.

A broader generalization appears in "Loss Functions for Classification using Structured Entropy" [2206.07122]. That work does not require an explicit hierarchy at all. Instead, it defines **structured cross-entropy** as a weighted average of ordinary cross-entropies over multiple coarsenings of the label space induced by a random partition. The paper explicitly contrasts this with HACE-style methods: HACE assumes a tree or DAG and exploits ancestor structure, whereas structured cross-entropy assumes only a family of partitions, which may encode hierarchies, circular structure, graphs, or other domain-specific similarities. In the paper’s own summary, HACE is a special case of a broader structure-aware loss idea.

The literature also contains methods that are related in goal but distinct in mechanism. **HAF** learns hierarchy-aware features using ordinary fine-level cross-entropy, Jensen-Shannon divergence between adjacent levels, a geometric consistency loss, and a margin-based loss [2207.12646]. **HCOT** combines standard cross-entropy with a hierarchical complement entropy that separates same-parent errors from errors outside the parent branch [1911.07257]. These methods address the same dissatisfaction with flat cross-entropy, but they are not HACE in the strict sense.

## 7. Limitations and open questions

The explicit HACE paper identifies three limitations [2605.06274]. First, HACE **requires a known hierarchy**; if no taxonomy exists, it cannot be directly applied. Second, it introduces a new hyperparameter \(d\), and the paper states that there is no theoretically derived optimal choice yet. Third, the authors identify the **theoretical understanding** of optimal ancestral smoothing functions and the conditions under which they improve generalization as an open problem.

The broader literature adds further constraints. HXE, as originally defined, requires a **tree**, not a general graph, so real hierarchies such as WordNet may need pruning or modification [1912.09393]. The performance of hierarchy-aware losses also depends on hierarchy quality: when the hierarchy is random or semantically uninformative, hierarchy-sensitive training can degrade the meaning of “better mistakes” [1912.09393]. The superclass-aware proper-scoring result of [2411.16438] depends on a **balanced weighted tree**; if the tree is not balanced, the loss is not a proper scoring rule.

These limitations delimit the current state of HACE research. The common premise across formulations is that label structure contains supervision that ordinary cross-entropy discards. The unresolved questions concern which hierarchy-aware transformation is theoretically principled, when hierarchical supervision improves exact classification rather than only hierarchical metrics, how to choose weighting or smoothing schedules, and how far tree-based methods extend to more general taxonomic or graph-structured label spaces.

Source: https://www.emergentmind.com/topics/hierarchy-aware-cross-entropy-hace