---
title: 'Hyper++: Unified Hyperbolic Neural Networks'
url: https://www.emergentmind.com/topics/hyper-06b21383-85a5-494e-bfec-8bc77b1bd913
type: topic
---

# Hyper++: Unified Hyperbolic Neural Networks

Searching arXiv for "Hyper++" and closely related exact titles to disambiguate the topic.
Hyper++ commonly denotes **“Hyperbolic Neural Networks++”**, a systematic extension of the original hyperbolic neural network program that reconstructs several standard neural-network primitives inside a single geometric model, the **Poincaré ball**. Its central claim is that Euclidean affine-type expressions can be reformulated in hyperbolic space through **signed distances to hyperbolic hyperplanes**, yielding a unified treatment of **multinomial logistic regression**, **fully connected layers**, **convolutional layers**, and **attention mechanisms** without increasing the parameter count relative to Euclidean analogues, and with better parameter efficiency than prior hyperbolic formulations [2006.08210].

## 1. Scope and relation to earlier hyperbolic neural networks

Hyper++ is positioned as a generalization of the earlier Hyperbolic Neural Networks framework rather than as an isolated new layer. The main technical departure appears first in hyperbolic multinomial logistic regression. Earlier hyperbolic MLR replaced
$$
\langle \mathbf a_k,\mathbf x\rangle - b_k
$$
with
$$
\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,
$$
which uses \(2n\) parameters per class through \((\mathbf a_k,\mathbf p_k)\). Hyper++ observes that this parameterization is redundant, because many \(\mathbf p_k\) define the same discriminative hyperplane. It therefore reparameterizes the bias using only a scalar \(r_k\) along the direction of \(\mathbf a_k\),
$$
\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,
$$
restoring the Euclidean \(n+1\) parameters per class while preserving the same representational range [2006.08210].

This same signed-distance construction is then generalized to the broader network toolkit. The resulting framework supplies a new Poincaré fully connected layer, a hyperbolic convolution layer built from concatenation and fully connected transformations, and a Poincaré attention mechanism based on hyperbolic centroids. A plausible implication is that the “++” designation refers less to a single architectural novelty than to a unified hyperbolic replacement for several core Euclidean building blocks.

## 2. Geometric model and basic operations

The framework is built in the \(n\)-dimensional Poincaré ball of constant negative curvature \(-c\), \(c>0\):
$$
(\mathbb B_c^n,\mathfrak g^c),\qquad \mathbb B_c^n=\{\mathbf x\in\mathbb R^n \mid c\|\mathbf x\|^2<1\}.
$$
Its metric tensor is
$$
\mathfrak g^c_{\mathbf x} = (\lambda^c_{\mathbf x})^2 I_n,\qquad \lambda^c_{\mathbf x} = \frac{2}{1-c\|\mathbf x\|^2}.
$$
Accordingly,
$$
\langle \mathbf u,\mathbf v\rangle_{\mathbf x}^c = (\lambda^c_{\mathbf x})^2 \langle \mathbf u,\mathbf v\rangle,\qquad
\|\mathbf v\|_{\mathbf x}^c = \lambda^c_{\mathbf x}\|\mathbf v\|.
$$

The construction relies on the standard gyrovector toolkit. Möbius addition is
$$
\mathbf x\oplus_c \mathbf y = \frac{ (1+2c\langle \mathbf x,\mathbf y\rangle + c\|\mathbf y\|^2)\mathbf x + (1-c\|\mathbf x\|^2)\mathbf y }{ 1+2c\langle \mathbf x,\mathbf y\rangle + c^2\|\mathbf x\|^2\|\mathbf y\|^2 },
$$
with
$$
\mathbf x\ominus_c \mathbf y = \mathbf x\oplus_c(-\mathbf y).
$$
Möbius scalar multiplication is
$$
r\otimes_c \mathbf x = \frac{1}{\sqrt c} \tanh\!\Bigl( r\,\tanh^{-1}(\sqrt c\,\|\mathbf x\|) \Bigr)\hat{\mathbf x}
= \exp^c_{\mathbf 0}\!\bigl(r\,\log^c_{\mathbf 0}(\mathbf x)\bigr).
$$
The exponential and logarithmic maps are
$$
\exp_{\mathbf x}^c(\mathbf v) = \mathbf x \oplus_c \left( \frac{1}{\sqrt c} \tanh\!\left( \frac{\sqrt c\,\lambda^c_{\mathbf x}\|\mathbf v\|}{2} \right) \hat{\mathbf v} \right),
$$
$$
\log_{\mathbf x}^c(\mathbf y) =
\frac{2}{\sqrt c\,\lambda^c_{\mathbf x}}
\tanh^{-1}\!\bigl(\sqrt c\,\|-\mathbf x\oplus_c \mathbf y\|\bigr)\,
\widehat{(-\mathbf x\oplus_c \mathbf y)}.
$$
The Poincaré distance is
$$
d_c(\mathbf x,\mathbf y) =
\frac{2}{\sqrt c}\tanh^{-1}\!\bigl(\sqrt c\,\|-\mathbf x\oplus_c \mathbf y\|\bigr)
= \|\log_{\mathbf x}^c(\mathbf y)\|_{\mathbf x}^c.
$$

Although the network layers are realized in \(\mathbb B_c^n\), the hyperboloid model \(\mathbb H_c^n\) and the Beltrami–Klein model \(\mathbb K_c^n\) are also used in the centroid analysis. This suggests that Hyper++ should be read not as a purely coordinate-level reformulation, but as a geometry-driven program whose different model realizations are exploited where analytically convenient [2006.08210].

## 3. Multinomial logistic regression and fully connected layers

The key geometric object is the Poincaré hyperplane through \(\mathbf p\in\mathbb B_c^n\) orthogonal to \(\mathbf a\in T_{\mathbf p}\mathbb B_c^n\):
$$
\tilde H_{\mathbf a,\mathbf p}^c =
\{\mathbf x\in\mathbb B_c^n \mid \langle -\mathbf p\oplus_c \mathbf x,\mathbf a\rangle = 0 \}.
$$
Its point-to-hyperplane distance is
$$
d_c(\mathbf x,\tilde H_{\mathbf a,\mathbf p}^c) =
\frac{1}{\sqrt c}\sinh^{-1}
\!\left(
\frac{ 2\sqrt c\,|\langle -\mathbf p\oplus_c \mathbf x,\mathbf a\rangle| }
{ (1-c\|-\mathbf p\oplus_c \mathbf x\|^2)\|\mathbf a\| }
\right).
$$

Hyper++ defines the class score of its unidirectional Poincaré MLR as
$$
v_k(\mathbf x) =
\operatorname{sign} \bigl( \langle \mathbf a_k,-\mathbf q_{\mathbf a_k,r_k}\oplus_c \mathbf x\rangle \bigr)
\, d_c(\mathbf x,\bar H_{\mathbf a_k,r_k}^c) \,
\|\mathbf a_k\|_{\mathbf q_{\mathbf a_k,r_k}}^c,
$$
with
$$
\mathbf q_{\mathbf a_k,r_k} = \exp_{\mathbf 0}^c(r_k\hat{\mathbf a}_k),\qquad
\bar H_{\mathbf a_k,r_k}^c =
\{\mathbf x\in\mathbb B_c^n\mid \langle \mathbf a_k,-\mathbf q_{\mathbf a_k,r_k}\oplus_c \mathbf x\rangle=0 \}.
$$
To avoid circular dependence between \(\mathbf a_k\) and \(\mathbf q_k\), the model parameterizes by \(\mathbf z_k\in T_{\mathbf 0}\mathbb B_c^n\) and parallel-transports it:
$$
\mathbf a_k = P_{\mathbf 0\to \mathbf q_{\mathbf a_k,r_k}}^c(\mathbf z_k)
= \operatorname{sech}^2(\sqrt c\,r_k)\,\mathbf z_k.
$$
The resulting closed form is
$$
v_k(\mathbf x)=
2c^{-1/2}\|\mathbf z_k\| \sinh^{-1}\!\left(
\lambda_{\mathbf x}^c \langle \sqrt c\,\mathbf x,\hat{\mathbf z}_k\rangle \cosh(2\sqrt c\,r_k)
- (\lambda_{\mathbf x}^c-1)\sinh(2\sqrt c\,r_k)
\right).
$$
In the Euclidean limit,
$$
\lim_{c\to 0}v_k(\mathbf x) = 4\bigl(\langle \mathbf a_k,\mathbf x\rangle-b_k\bigr),
$$
so the construction reduces to ordinary logistic regression up to a constant factor \(4\).

The fully connected layer is then defined by requiring each output coordinate to be a signed hyperbolic distance to an output-space hyperplane. With parameters
$$
\mathbf Z=\{\mathbf z_k\}_{k=1}^m,\qquad \mathbf r=\{r_k\}_{k=1}^m,
$$
the layer is
$$
\mathbf y = \mathcal F^c(\mathbf x;\mathbf Z,\mathbf r)
\coloneqq \frac{\mathbf w}{1+\sqrt{1+c\|\mathbf w\|^2}},
\qquad
\mathbf w \coloneqq \left( c^{-1/2}\sinh(\sqrt c\, v_k(\mathbf x)) \right)_{k=1}^m.
$$
This differs from the earlier HNN fully connected form
$$
\mathbf y=\exp_{\mathbf 0}^c\bigl(\mathbf A\,\log_{\mathbf 0}^c(\mathbf x)\bigr)\oplus_c \mathbf b.
$$
In Hyper++, the discriminative surfaces are genuine Poincaré hyperplanes, and the parameter count matches Euclidean affine layers rather than introducing additional geometry-specific parameters [2006.08210].

## 4. Split, concatenation, convolution, and attention

Hyper++ treats split and concatenation as first-class operations because convolution and multi-head attention require them. A naive coordinate split changes norms, and repeated Möbius-addition concatenation is noncommutative, nonassociative, and sequential. Hyper++ therefore introduces Poincaré **\(\beta\)-split** and **\(\beta\)-concatenation**, designed to preserve expected norms under a tangent-space normality assumption. With
$$
\beta_n = B\!\left(\frac n2,\frac 12\right),
$$
\(\beta\)-split maps
$$
\mathbf v=\log_{\mathbf 0}^c(\mathbf x) = (\mathbf v_1^\top,\ldots,\mathbf v_N^\top)^\top
$$
to
$$
\mathbf y_i = \exp_{\mathbf 0}^c\!\left( \beta_{n_i}\beta_n^{-1}\mathbf v_i \right),
$$
while \(\beta\)-concatenation maps \(\mathbf x_i\in\mathbb B_c^{n_i}\) to
$$
\mathbf v = \left( \beta_n\beta_{n_1}^{-1}\mathbf v_1^\top,\ldots,
\beta_n\beta_{n_N}^{-1}\mathbf v_N^\top \right)^\top,\qquad
\mathbf y=\exp_{\mathbf 0}^c(\mathbf v).
$$
The paper proves that \(\beta\)-concatenation is the inverse of \(\beta\)-split.

The convolutional layer is then built compositionally. A receptive field of gyrovectors is first Poincaré-\(\beta\)-concatenated into a single element of \(\mathbb B_c^{nK}\), where \(K=\prod_i K_i\) is the kernel size, and the result is passed through the Poincaré fully connected layer. This yields arbitrary-dimensional convolution without introducing a separate convolution-specific hyperbolic algebra.

The attention mechanism centers on a theorem equating three midpoint notions: the **Möbius gyromidpoint** in the Poincaré ball, the **Einstein gyromidpoint** in the Beltrami–Klein model, and the centroid minimizing squared Lorentzian distance in the hyperboloid model. The paper further extends weighted centroids to arbitrary real weights:
$$
\bar{\mathbf b} = \operatorname{aggr}\{\mathbf b_i,\nu_i\}
\coloneqq
\frac12\otimes_c
\left(
\frac{ \sum_{i=1}^N \nu_i \lambda_{\mathbf b_i}^c \mathbf b_i }
{ \sum_{i=1}^N |\nu_i|(\lambda_{\mathbf b_i}^c-1) }
\right).
$$
It also characterizes the gyromidpoint as the solution of
$$
\bar{\mathbf b} =
\arg\min_{\tilde{\mathbf b}}
\sum_{i=1}^N
\nu_i\, \lambda^c_{-\tilde{\mathbf b}\oplus_c \mathbf b_i}
\, \|-\tilde{\mathbf b}\oplus_c \mathbf b_i\|^2.
$$

Poincaré multi-head attention uses Poincaré fully connected layers for \(Q\), \(K\), and \(V\), then applies \(\beta\)-split, computes weights
$$
\pi^i_{t,s}=g(f^c(\mathbf q_t^i,\mathbf k_s^i)),
$$
aggregates values by the weighted centroid,
$$
\mathbf x_t^i = \operatorname{aggr}_{1\le s\le L_s}(\mathbf v_s^i,\pi^i_{t,s}),
$$
and merges heads through \(\beta\)-concatenation. The paper discusses both tangent-space inner products and distance-based similarities; in experiments it uses the distance-based form
$$
f^c(\mathbf q,\mathbf k) = -\tau\, d_c(\mathbf q,\mathbf k)-\gamma
$$
with exponential activation \(g(x)=e^x\) [2006.08210].

## 5. Optimization protocol and empirical results

The reported implementation uses **Riemannian Adam** through **Geoopt**. For the MLR experiment, the training setup is learning rate \(0.001\), batch size \(16\), and 30 epochs. For the Set Transformer experiments, Riemannian Adam uses
$$
\beta_1=0.9,\quad \beta_2=0.999,\quad \epsilon=10^{-8},\quad \eta=10^{-3},
$$
and for ConvSeq2Seq translation it uses
$$
\beta_1=0.9,\quad \beta_2=0.98,\quad \epsilon=10^{-9},
$$
with warm-up for the first 4000 steps and inverse-square-root schedule
$$
\eta=(Dt)^{-1/2}.
$$
The reported initializations are
$$
Z_{ij}\sim \mathcal N(0,n^{-1/2}),\qquad \mathbf r=\mathbf 0
$$
for unidirectional Poincaré MLR,
$$
Z_{ij}\sim \mathcal N(0,(2nm)^{-1/2}),\qquad \mathbf r=\mathbf 0
$$
for the Poincaré fully connected layer, and
$$
Z_{ij}\sim \mathcal N(0,(2nKm)^{-1/2}),\qquad \mathbf r=\mathbf 0
$$
for Poincaré convolution. Hyperbolic Set Transformer experiments omit ReLU and normalization layers, and the translation model initializes Poincaré embeddings with gain \(\epsilon_E=10^{-2}\).

Three experimental programs are reported. On **WordNet subtree classification**, Hyper++ is compared with Euclidean MLR and earlier HNN hyperbolic MLR using pretrained Poincaré embeddings with \(D\in\{2,3,5,10\}\). Representative F1 results include **animal.n.01, \(D=5\)**: \(86.26\pm 4.66\) for Hyper++, \(85.89\pm 3.77\) for HNNs, and \(66.20\pm 2.11\) for Euclidean MLR; and **mammal.n.01, \(D=5\)**: \(99.30\pm 0.30\), \(98.89\pm 1.29\), and \(60.53\pm 3.27\), respectively. The paper interprets this as maintaining or improving accuracy while reducing the parameter count from \(2D\) per class in the earlier HNN MLR to \(D+1\) [2006.08210].

On **Set Transformer** experiments for amortized clustering of mixtures of Gaussians, Hyper++ is competitive on Euclidean Gaussian data and stronger on hyperbolic Gaussian data. For Euclidean data with \(K=4\), the reported test negative log-likelihood is \(1.558\pm 0.008\) for Hyper++, against \(1.558\pm 0.032\) for Set Transformer and \(1.556\pm 0.214\) for Set Transformer without LayerNorm, the latter with 3 diverged trials. For hyperbolic Gaussian data, the advantage is clearer: with \(K=4\), Set Transformer gives \(3.084\pm 0.305\) while Hyper++ gives \(2.920\pm 0.029\); with \(K=7\), the corresponding values are \(3.923\pm 1.632\) and \(3.375\pm 0.033\).

On **WMT’17 English–German translation** using a ConvSeq2Seq architecture, Hyper++ shows its strongest gains in small dimensions. BLEU-4 scores are reported as follows: at \(D=16\), ConvSeq2Seq \(2.68\) versus Hyper++ \(9.81\); at \(D=32\), \(8.43\) versus \(14.11\); at \(D=64\), \(14.92\) versus \(16.95\); at \(D=128\), \(20.02\) versus \(19.40\); and at \(D=256\), \(21.84\) versus \(21.76\). This supports the paper’s claim that hyperbolic layers are especially parameter-efficient in low-dimensional regimes.

## 6. Limitations and place in the hyperbolic deep-learning literature

The framework does not eliminate tangent-space mappings entirely. Split and concatenation are implemented through \(\log_{\mathbf 0}^c\) and \(\exp_{\mathbf 0}^c\), and some input-output interfaces remain tangent-space based. The \(\beta\)-split and \(\beta\)-concatenation analysis depends on a Gaussian assumption in tangent space, so its norm-preservation rationale is heuristic rather than universal. The paper also omits normalization layers for hyperbolic models, explicitly noting that hyperbolic normalization is not yet well understood. Empirically, the translation results show that the low-dimensional advantage fades as Euclidean model dimension increases, indicating that Hyper++ is not uniformly superior across all capacity regimes [2006.08210].

Within hyperbolic deep learning, Hyper++ is therefore best understood as a consolidation and refinement step. It replaces earlier tangent-space affine surrogates with a more geometric account based on Poincaré hyperplanes, signed distances, and hyperbolic centroids, while simultaneously reducing the parameter cost of hyperbolic multinomial logistic regression to the Euclidean \(D+1\) form. This suggests that its lasting significance lies in two directions: first, as a mathematically unified library of hyperbolic neural components; second, as evidence that hyperbolic architectures are most compelling when hierarchical structure and parameter efficiency matter more than sheer ambient dimensionality.

Source: https://www.emergentmind.com/topics/hyper-06b21383-85a5-494e-bfec-8bc77b1bd913