Hyperbolic Neural Networks Overview

• Hyperbolic Neural Networks are architectures that operate in hyperbolic space, leveraging exponential expansion to robustly model hierarchical and relational data.
• The Klein model, with its straight-line geodesics and Einstein arithmetic, provides a computationally efficient and algebraically elegant framework.
• Empirical studies show that HNNs deliver competitive accuracy in tasks like node classification and graph analysis while offering enhanced computational performance.

Hyperbolic Neural Networks (HNNs) are neural architectures in which all latent representations and operations are defined over hyperbolic space, a non-Euclidean geometry of constant negative curvature. Motivated by the intrinsic tree-likeness and exponential expansion of hyperbolic geometry, these networks have shown distinct advantages in modeling hierarchical data, outperforming their Euclidean counterparts in tasks involving complex relational and graph structures. This entry provides a comprehensive technical overview of the mathematical foundations, architectural developments, empirical performance, and ongoing challenges in the design and application of hyperbolic neural networks.

1. Mathematical Foundations of Hyperbolic Neural Networks

Hyperbolic Geometry and Coordinate Models

Hyperbolic spaces, denoted generically as 𝔻ⁿ (Poincaré ball), ℍⁿ (hyperboloid), or 𝕂ⁿ (Klein model), are Riemannian manifolds with constant negative sectional curvature. The following models are widely used in HNN research:

• Poincaré Ball Model: 𝔻ⁿ = { x ∈ ℝⁿ : ‖x‖ < 1 } with metric tensor gₓ = (2/(1 – ‖x‖²))² Iₙ.
• Hyperboloid Model (Lorentz): ℍⁿ = { x ∈ ℝⁿ⁺¹ : ⟨x, x⟩_L = –1, x₀ > 0 } with the Lorentzian inner product ⟨x, y⟩_L = –x₀y₀ + x₁y₁ + ... + xₙyₙ.
• Klein Model: 𝕂ⁿ = { x ∈ ℝⁿ : ‖x‖ < 1 } with metric tensor g_{ij}(x) = [δ_{ij}(1 – ‖x‖²) + x_i x_j]⁄(1 – ‖x‖²)².

Maps and Algebraic Structure

Operations generalizing vector addition and scalar multiplication are as follows:

• Möbius Addition (Poincaré):

$$x \oplus_c y = \frac{(1 + 2c\langle x, y\rangle + c\|y\|^2)x + (1 – c\|x\|^2)y}{1 + 2c\langle x, y\rangle + c^2\|x\|^2\|y\|^2}$$

• Einstein Addition (Klein):

$$u \oplus_E v = \frac{(1 + u^T v)u + v + (\gamma_u – 1)\frac{\langle u, v \rangle}{\|u\|^2}u}{1 + u^T v}$$

with Lorentz factor $\gamma_u = 1/\sqrt{1-\|u\|^2}$.

• Exponential and Logarithmic Maps:

$$\exp_x(v) = x + \frac{\sinh(\|v\|)}{\|v\|}v\,/\, [\cosh(\|v\|) + \lambda_x \langle x, v\rangle \sinh(\|v\|)]$$

The tangent space formulation, central to most HNNs, enables "lifting" Euclidean operations to curved geometry.

Model Mappings

Isometries and projections connect different hyperbolic models:

• Poincaré ↔ Klein: $\pi_{B\rightarrow K}(x) = 2x/(1+\|x\|^2)$, $\pi_{K\rightarrow B}(x) = x/(1+\sqrt{1-\|x\|^2})$
• Klein ↔ Hyperboloid: via gnomonic projection and rescaling

2. Hyperbolic Neural Network Layers and Operations

Hyperbolic neural layers generalize standard neural computations by replacing Euclidean vector space operations with their hyperbolic analogs in the manifold.

• Feed-Forward (Hyperbolic Linear) Layer (Klein Model):
  • Map input $x$ to the tangent space via $\log_0(x)$.
  • Apply linear transformation: $y' = W \cdot \log_0(x) + b$.
  • Map back: $y = \exp_0(y')$.
  • This is shown to be equivalent to Einstein scalar multiplication and addition.
• Activation Functions: Nonlinearities are often defined as $\sigma^{\otimes}(x) = \exp_0(\sigma(\log_0(x)))$, i.e., applying the nonlinearity in the tangent space and mapping the result back.
• Aggregation and Attention (Klein Model):
  • Weighted aggregation via the Einstein midpoint: in 𝕂ⁿ, for weights $\alpha_j$ and vectors $v_j$,

  $$m = \sum_j \left( \frac{\alpha_j \gamma(v_j)}{\sum_l \alpha_l \gamma(v_l)} \right)v_j,\quad \gamma(v) = 1/\sqrt{1-\|v\|^2}$$ - This aggregation is permutation-invariant and leverages the straight-line geodesics in the Klein model.

• Parallel Transport and Scalar Multiplication:

  • Scalar multiplication: $$r \otimes_E x = \exp_x(r \cdot \log_x(x)) = \tanh(r \cdot \tanh^{-1}(\|x\|))(x/\|x\|)$$.

3. Empirical Evaluation and Performance

The Klein HNN is empirically compared to Poincaré ball and hyperboloid-based HNNs on standard datasets (Texas, Wisconsin, Chameleon, Actor, Cora, Pubmed). Results indicate:

Dataset	Klein HNN Acc. (%)	Poincaré HNN Acc. (%)	Hyperboloid HNN Acc. (%)
Texas	~96.97	~96.97	~96.97
Wisconsin	Comparable	Comparable	Comparable
Chameleon	Comparable	Comparable	Comparable
Actor, Cora, ...	Comparable or better	Comparable	Comparable

The average per-epoch runtime for Klein HNN matches or outperforms hyperboloid HNNs, attributed to a lack of multiple tangent space projections. Accuracy is consistently on par with both alternatives.

4. Algebraic and Computational Advantages of the Klein Model

• Geodesics as Straight Lines: The Klein model represents geodesics as straight segments, simplifying computational schemes for operations like centroids, aggregation, and parallel transport.
• Einstein Gyrovector Space: All necessary hyperbolic operations such as addition, scalar multiplication, and even higher-order constructs like midpoints and centroids are elegantly realized as Einstein operations in the Klein model. This aligns tangent-space constructions with algebraic manipulations on the manifold.
• Model Interoperability: Isometric mappings to and from other models (Poincaré ball, hyperboloid) are explicit and closed-form, ensuring that results are directly comparable.

5. Significance and Role as HNN Foundations

• Competitiveness: Klein HNNs are shown to be as accurate as Poincaré and hyperboloid HNNs for node classification, link prediction, and general manifold representation tasks.
• Architectural Simplicity: The compactness of Einstein operations (closed-form, coordinate-free for key constructs) enables computationally efficient and numerically stable HNN frameworks, making the Klein model a practical candidate for deployment and research.
• Building Block for Advanced Models: The pure algebraic and geometric structure positions Klein HNNs as fundamental blocks for more elaborate architectures—potentially including advanced attention and aggregation mechanisms, large-scale graph networks, and generative models.

6. Future Directions and Mathematical Extensions

The Klein model invites several avenues for further research:

• Extending Beyond Linear Layers: Exploration of pooling, normalization, and complex attention over the Klein model using Einstein aggregation schemes.
• Hierarchical and Structured Data: The Klein model's properties are advantageous for hierarchical and tree-like data, suggesting its use in knowledge graphs, taxonomies, and biological networks.
• Theoretical Robustness: Investigations into the numerical stability and robustness of Klein HNNs in deeper, wider networks and regimes of high negative curvature.
• Software Libraries and Frameworks: Potential for optimized libraries targeting Einstein gyrovector arithmetic and the Klein manifold for integration with geometric deep learning toolchains.

7. Comparative Summary Table: HNN Model Properties

Model	Geodesics	Arithmetic Core	Pros
Poincaré Ball	Circles/arcs	Möbius addition	Conformal, standard
Hyperboloid	Hyperbolas	Lorentz transformations	Numerically stable
Klein	Straight segments	Einstein addition	Algebraic, efficient

Klein HNNs thus provide a fully viable and mathematically elegant geometric framework for deep learning on non-Euclidean data, with performance and computational costs directly comparable to existing Poincaré and hyperboloid-based approaches (Mao et al., 22 Oct 2024, Gulcehre et al., 2018).