Hyperbolic Hierarchical Contrastive Loss (HHCL)

• HHCL is a family of contrastive learning objectives designed for hyperbolic embedding spaces that efficiently capture hierarchical, tree-like structures.
• It combines a geodesic alignment loss with a uniformity penalty that maps embeddings to an isotropic Gaussian in the tangent space to prevent dimensional collapse.
• HHCL enforces hierarchical consistency across multiple semantic levels, leading to enhanced performance in tasks like node classification and fine-grained visual recognition.

The Hyperbolic Hierarchical Contrastive Loss (HHCL) is a family of contrastive learning objectives specifically formulated for hyperbolic embedding spaces. It is motivated by the observation that many real-world datasets—particularly those with latent or explicit hierarchical structure, such as graphs, taxonomies, part-whole ontologies, fine-grained visual categories, and multimodal biomedical data—exhibit geometric properties poorly captured by Euclidean metrics. HHCL exploits the exponential expansion of hyperbolic geometry to encode multi-level hierarchies, employs specialized uniformity and alignment terms to prevent collapse, and integrates pairwise or multi-level constraints to enforce hierarchical consistency during self-supervised or supervised representation learning.

1. Hyperbolic Geometry and Motivation

HHCL is defined in models of hyperbolic space such as the Poincaré ball or Lorentz (hyperboloid) manifold, each with constant negative curvature $-c$ for some $c>0$. The Poincaré ball in $d$ dimensions is given by

$$\mathbb{D}^d_c = \{x \in \mathbb{R}^d: \|x\|_2 < 1/\sqrt{c}\},$$

where geodesic distance is

$$D_c(p, q) = \frac{2}{\sqrt{c}} \tanh^{-1}(\sqrt{c} \|(-p) \oplus q\|_2),$$

with Möbius addition $\oplus$. Hyperbolic volume increases exponentially with radius ($\sim \sinh^d(\sqrt{c}\, r)$), making the geometry well tailored for tree-like, multi-branching structures. Unlike Euclidean space, which cannot efficiently encode deep hierarchies in low dimensions, hyperbolic geometry admits low-distortion embeddings of trees, ontologies, and partonomical structures.

2. Core HHCL Objective: Alignment and Uniformity

The canonical HHCL for node-level or image instance-level learning in hyperbolic graph contrastive frameworks (Zhang et al., 2023) is composed of two terms:

• Alignment Loss ($\mathcal{L}_A^{\mathbb{D}_c}$): For two positive views of a node $i$, with embeddings $z_i, z_i' \in \mathbb{D}^d_c$, the loss encourages geodesic proximity:

$$\mathcal{L}_A^{\mathbb{D}_c} = \frac{2}{N \sqrt{c}} \sum_{i=1}^N \tanh^{-1}\left(\sqrt{c} \|(-z_i) \oplus z_i'\|_2\right)$$

This term enforces the hierarchical-invariant alignment of positive pairs along the hyperbolic manifold.

• Uniformity (Outer-Shell Isotropy) Loss ($\mathcal{L}_U^{\mathcal{T}}$): To prevent dimensional collapse—a pathology in which embeddings cluster into low-dimensional high-density regions—the uniformity constraint enforces an isotropic Gaussian distribution in the tangent plane at the origin, pulling ambient points to form a uniform “outer shell.” After mapping $z_i$ back to the tangent space ($y_i = \log_{0}(z_i)$), isotropy is imposed via

$$D_{\mathrm{KL}}(\mathcal{N}(\mu, \Sigma) \,\|\, \mathcal{N}(0, I)) = \frac{1}{2}[\operatorname{tr}(\Sigma) - \log\det(\Sigma) - d + \|\mu\|^2],$$

applied to both views, yielding

$$\mathcal{L}_U^{\mathcal{T}} = \operatorname{tr}(\Sigma) + \operatorname{tr}(\Sigma') - \log\det(\Sigma \Sigma') + \|\mu\|^2 + \|\mu'\|^2.$$

This regularizer ensures well-utilized embedding capacity across both the “angular” (leaf-level) and radial (height-level) axes, corresponding to tree directions and depths.

The full HHCL combines these terms with a scalar $\lambda$: $$\mathcal{L}_{\mathrm{HHCL}} = \mathcal{L}_A^{\mathbb{D}_c} + \lambda \mathcal{L}_U^{\mathcal{T}}$$ Minimizing $\mathcal{L}_U^{\mathcal{T}}$ provably increases the effective rank (spectral diversity) of the embedding covariance [Thm. 5.4, (Zhang et al., 2023)].

3. Hierarchical Semantics and Multi-Level Constraints

Many HHCL variants enforce hierarchical consistency not only at the instance level but across multiple semantic levels or along arbitrary hierarchies. In the H3Former model for fine-grained visual classification (Zhang et al., 13 Nov 2025):

• Hybrid Contrastive Distance: Combines Euclidean and hyperbolic distances:

$$D_{i,j} = \|z_i - z_j\|_2 + \lambda d_{\mathcal{L}}(\exp_0(z_i), \exp_0(z_j))$$

with $\exp_0$ mapping to the Lorentz model and $d_{\mathcal{L}}$ the Lorentzian geodesic.

• Supervised Multi-Level Contrastive Term:

$$\mathcal{L}_{\mathrm{con}} = -\sum_{i} \frac{1}{|P(i)|} \sum_{p \in P(i)} \log \frac{\exp(-D_{i,p}/\tau)}{\sum_{a \neq i} \exp(-D_{i,a}/\tau)}$$

where positives $P(i)$ are those at the same hierarchy level and class.

• Hypergraph Partial-Order Preservation (HPOP): Enforces that parent and child region-nodes at different hierarchical levels remain close in hyperbolic space:

$$\mathcal{L}_{\mathrm{hpop}} = \frac{1}{L-1} \sum_{\ell=1}^{L-1} \mathrm{ReLU}\Bigl( d_{\mathcal{L}} (\exp_0(H_i^{\ell+1}), \exp_0(H_i^{\ell})) \Bigr)$$

The aggregate HHCL is $$\mathcal{L}_{\mathrm{HHCL}} = \mathcal{L}_{\mathrm{con}} + \beta \mathcal{L}_{\mathrm{hpop}}$$.

This suggests HHCL can be adapted to enforce multi-level semantic or hierarchical regularities in arbitrary domain or task hierarchies.

4. Prevention of Hyperbolic Dimensional Collapse

A core technical challenge in hyperbolic contrastive learning is the tendency for embeddings to collapse onto narrow regions of the manifold (“dimensional collapse”). Empirically, naïve geodesic minimization without additional constraints causes most mass to accumulate toward the Poincaré boundary, leading to poor cluster separation and reduced classification accuracy. The two-level isotropy penalty directly addresses this by imposing ring density (leaf-level) and radial density (height-level) objectives, which are mapped from hyperbolic space to an equivalent isotropic Gaussian in the tangent plane (Zhang et al., 2023). Theoretical results relate the minimization of the uniformity penalty to maximization of the effective rank, ensuring diversified usage of the ambient space.

5. Algorithmic Implementation

The HHCL training loop can be summarized as follows (Zhang et al., 2023):

1. For each minibatch, sample two augmented views of each node (e.g., via node/edge dropout for graphs).
2. Encode each view with a (graph) encoder to give Euclidean embeddings $Z, Z'$.
3. Project into the Poincaré ball: $z_i \leftarrow \mathrm{clamp}(Z_i)$.
4. Compute geodesic distances for positive pairs to obtain the alignment loss.
5. Map each hyperbolic embedding back to the origin tangent space via logarithmic map.
6. Calculate batch-wise mean and covariance in the tangent plane; apply the KL divergence to enforce isotropy.
7. Add alignment and uniformity losses; backpropagate using Riemannian SGD, propagating gradients through the Möbius, exponential/logarithmic, and moment steps.

6. Experimental Validation and Performance

Across diverse domains, HHCL delivers state-of-the-art or highly competitive performance:

• Node classification (Cora, Citeseer, Pubmed) and collaborative filtering benchmarks show HHCL outperforms both Euclidean contrastive learning and earlier hyperbolic GNNs, yielding higher recall and sharper embedding separation (Zhang et al., 2023).
• Ablation studies demonstrate that removing the outer-shell isotropy penalty causes collapse, while alignment alone is insufficient; naive uniformity results in boundary crowding.
• Effective rank analyses confirm that HHCL achieves higher spectral dispersion in the embedding space, correlating with improved downstream task performance.

7. Context and Relation to Broader Hyperbolic Contrastive Learning

The HHCL design is part of a broader movement toward hierarchical representation learning within non-Euclidean geometry. Analogous approaches include Lorentzian and Poincaré-based InfoNCE, entailment-cone angle-based losses for semantic hierarchies, and multi-view or multi-level regularizers for both unsupervised and supervised applications. Across self-supervised graph learning (Liu et al., 2022), image retrieval (Qiu et al., 14 Jan 2024), multimodal survival analysis, and fine-grained visual recognition, explicit HHCL-style objectives are consistently shown to robustly encode both inter-instance and multi-level taxonomic relationships within a unified optimization framework.

This suggests HHCL is likely adaptable for any domain featuring partially observed or soft hierarchy, and is adaptable to various base architectures (GNNs, CNNs, Transformer backbones) and optimization approaches (Riemannian SGD or Adam). The critical mechanism is the joint enforcement of manifold-local geodesic alignment and global geometric uniformity at multiple semantic scales.