Hyper++: Unified Hyperbolic Neural Networks

• Hyper++ is a unified hyperbolic neural network framework that reformulates standard Euclidean layers using Poincaré hyperplanes and signed distances.
• It replaces redundant parameterizations in hyperbolic multinomial logistic regression with a scalar bias to achieve Euclidean-level efficiency.
• It generalizes core operations—fully connected layers, convolution, and attention—demonstrating improved performance in low-dimensional regimes.

Searching arXiv for "Hyper++" and closely related exact titles to disambiguate the topic. Hyper++ commonly denotes “Hyperbolic Neural Networks++”, a systematic extension of the original hyperbolic neural network program that reconstructs several standard neural-network primitives inside a single geometric model, the Poincaré ball. Its central claim is that Euclidean affine-type expressions can be reformulated in hyperbolic space through signed distances to hyperbolic hyperplanes, yielding a unified treatment of multinomial logistic regression, fully connected layers, convolutional layers, and attention mechanisms without increasing the parameter count relative to Euclidean analogues, and with better parameter efficiency than prior hyperbolic formulations (Shimizu et al., 2020).

1. Scope and relation to earlier hyperbolic neural networks

Hyper++ is positioned as a generalization of the earlier Hyperbolic Neural Networks framework rather than as an isolated new layer. The main technical departure appears first in hyperbolic multinomial logistic regression. Earlier hyperbolic MLR replaced

$\langle \mathbf a_k,\mathbf x\rangle - b_k$

with

$$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$

which uses $2n$ parameters per class through $(\mathbf a_k,\mathbf p_k)$. Hyper++ observes that this parameterization is redundant, because many $\mathbf p_k$ define the same discriminative hyperplane. It therefore reparameterizes the bias using only a scalar $r_k$ along the direction of $\mathbf a_k$,

$$\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,$$

restoring the Euclidean $n+1$ parameters per class while preserving the same representational range (Shimizu et al., 2020).

This same signed-distance construction is then generalized to the broader network toolkit. The resulting framework supplies a new Poincaré fully connected layer, a hyperbolic convolution layer built from concatenation and fully connected transformations, and a Poincaré attention mechanism based on hyperbolic centroids. A plausible implication is that the “++” designation refers less to a single architectural novelty than to a unified hyperbolic replacement for several core Euclidean building blocks.

2. Geometric model and basic operations

The framework is built in the $n$-dimensional Poincaré ball of constant negative curvature $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$0, $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$1:

$$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$2

Its metric tensor is

$$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$3

Accordingly,

$$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$4

The construction relies on the standard gyrovector toolkit. Möbius addition is

$$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$5

with

$$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$6

Möbius scalar multiplication is

$$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$7

The exponential and logarithmic maps are

$$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$8

$$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$9

The Poincaré distance is

$2n$0

Although the network layers are realized in $2n$1, the hyperboloid model $2n$2 and the Beltrami–Klein model $2n$3 are also used in the centroid analysis. This suggests that Hyper++ should be read not as a purely coordinate-level reformulation, but as a geometry-driven program whose different model realizations are exploited where analytically convenient (Shimizu et al., 2020).

3. Multinomial logistic regression and fully connected layers

The key geometric object is the Poincaré hyperplane through $2n$4 orthogonal to $2n$5:

$2n$6

Its point-to-hyperplane distance is

$2n$7

Hyper++ defines the class score of its unidirectional Poincaré MLR as

$2n$8

with

$2n$9

To avoid circular dependence between $(\mathbf a_k,\mathbf p_k)$0 and $(\mathbf a_k,\mathbf p_k)$1, the model parameterizes by $(\mathbf a_k,\mathbf p_k)$2 and parallel-transports it:

$(\mathbf a_k,\mathbf p_k)$3

The resulting closed form is

$(\mathbf a_k,\mathbf p_k)$4

In the Euclidean limit,

$(\mathbf a_k,\mathbf p_k)$5

so the construction reduces to ordinary logistic regression up to a constant factor $(\mathbf a_k,\mathbf p_k)$6.

The fully connected layer is then defined by requiring each output coordinate to be a signed hyperbolic distance to an output-space hyperplane. With parameters

$(\mathbf a_k,\mathbf p_k)$7

the layer is

$(\mathbf a_k,\mathbf p_k)$8

This differs from the earlier HNN fully connected form

$(\mathbf a_k,\mathbf p_k)$9

In Hyper++, the discriminative surfaces are genuine Poincaré hyperplanes, and the parameter count matches Euclidean affine layers rather than introducing additional geometry-specific parameters (Shimizu et al., 2020).

4. Split, concatenation, convolution, and attention

Hyper++ treats split and concatenation as first-class operations because convolution and multi-head attention require them. A naive coordinate split changes norms, and repeated Möbius-addition concatenation is noncommutative, nonassociative, and sequential. Hyper++ therefore introduces Poincaré $\mathbf p_k$0-split and $\mathbf p_k$1-concatenation, designed to preserve expected norms under a tangent-space normality assumption. With

$\mathbf p_k$2

$\mathbf p_k$3-split maps

$\mathbf p_k$4

to

$\mathbf p_k$5

while $\mathbf p_k$6-concatenation maps $\mathbf p_k$7 to

$\mathbf p_k$8

The paper proves that $\mathbf p_k$9-concatenation is the inverse of $r_k$0-split.

The convolutional layer is then built compositionally. A receptive field of gyrovectors is first Poincaré-$r_k$1-concatenated into a single element of $r_k$2, where $r_k$3 is the kernel size, and the result is passed through the Poincaré fully connected layer. This yields arbitrary-dimensional convolution without introducing a separate convolution-specific hyperbolic algebra.

The attention mechanism centers on a theorem equating three midpoint notions: the Möbius gyromidpoint in the Poincaré ball, the Einstein gyromidpoint in the Beltrami–Klein model, and the centroid minimizing squared Lorentzian distance in the hyperboloid model. The paper further extends weighted centroids to arbitrary real weights:

$r_k$4

It also characterizes the gyromidpoint as the solution of

$r_k$5

Poincaré multi-head attention uses Poincaré fully connected layers for $r_k$6, $r_k$7, and $r_k$8, then applies $r_k$9-split, computes weights

$\mathbf a_k$0

aggregates values by the weighted centroid,

$\mathbf a_k$1

and merges heads through $\mathbf a_k$2-concatenation. The paper discusses both tangent-space inner products and distance-based similarities; in experiments it uses the distance-based form

$\mathbf a_k$3

with exponential activation $\mathbf a_k$4 (Shimizu et al., 2020).

5. Optimization protocol and empirical results

The reported implementation uses Riemannian Adam through Geoopt. For the MLR experiment, the training setup is learning rate $\mathbf a_k$5, batch size $\mathbf a_k$6, and 30 epochs. For the Set Transformer experiments, Riemannian Adam uses

$\mathbf a_k$7

and for ConvSeq2Seq translation it uses

$\mathbf a_k$8

with warm-up for the first 4000 steps and inverse-square-root schedule

$\mathbf a_k$9

The reported initializations are

$$\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,$$0

for unidirectional Poincaré MLR,

$$\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,$$1

for the Poincaré fully connected layer, and

$$\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,$$2

for Poincaré convolution. Hyperbolic Set Transformer experiments omit ReLU and normalization layers, and the translation model initializes Poincaré embeddings with gain $$\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,$$3.

Three experimental programs are reported. On WordNet subtree classification, Hyper++ is compared with Euclidean MLR and earlier HNN hyperbolic MLR using pretrained Poincaré embeddings with $$\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,$$4. Representative F1 results include animal.n.01, $$\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,$$5: $$\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,$$6 for Hyper++, $$\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,$$7 for HNNs, and $$\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,$$8 for Euclidean MLR; and mammal.n.01, $$\mathbf q_{\mathbf a_k,r_k}=r_k \,\hat{\mathbf a}_k,\qquad b_k=r_k\|\mathbf a_k\|,$$9: $n+1$0, $n+1$1, and $n+1$2, respectively. The paper interprets this as maintaining or improving accuracy while reducing the parameter count from $n+1$3 per class in the earlier HNN MLR to $n+1$4 (Shimizu et al., 2020).

On Set Transformer experiments for amortized clustering of mixtures of Gaussians, Hyper++ is competitive on Euclidean Gaussian data and stronger on hyperbolic Gaussian data. For Euclidean data with $n+1$5, the reported test negative log-likelihood is $n+1$6 for Hyper++, against $n+1$7 for Set Transformer and $n+1$8 for Set Transformer without LayerNorm, the latter with 3 diverged trials. For hyperbolic Gaussian data, the advantage is clearer: with $n+1$9, Set Transformer gives $n$0 while Hyper++ gives $n$1; with $n$2, the corresponding values are $n$3 and $n$4.

On WMT’17 English–German translation using a ConvSeq2Seq architecture, Hyper++ shows its strongest gains in small dimensions. BLEU-4 scores are reported as follows: at $n$5, ConvSeq2Seq $n$6 versus Hyper++ $n$7; at $n$8, $n$9 versus $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$00; at $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$01, $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$02 versus $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$03; at $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$04, $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$05 versus $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$06; and at $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$07, $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$08 versus $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$09. This supports the paper’s claim that hyperbolic layers are especially parameter-efficient in low-dimensional regimes.

6. Limitations and place in the hyperbolic deep-learning literature

The framework does not eliminate tangent-space mappings entirely. Split and concatenation are implemented through $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$10 and $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$11, and some input-output interfaces remain tangent-space based. The $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$12-split and $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$13-concatenation analysis depends on a Gaussian assumption in tangent space, so its norm-preservation rationale is heuristic rather than universal. The paper also omits normalization layers for hyperbolic models, explicitly noting that hyperbolic normalization is not yet well understood. Empirically, the translation results show that the low-dimensional advantage fades as Euclidean model dimension increases, indicating that Hyper++ is not uniformly superior across all capacity regimes (Shimizu et al., 2020).

Within hyperbolic deep learning, Hyper++ is therefore best understood as a consolidation and refinement step. It replaces earlier tangent-space affine surrogates with a more geometric account based on Poincaré hyperplanes, signed distances, and hyperbolic centroids, while simultaneously reducing the parameter cost of hyperbolic multinomial logistic regression to the Euclidean $$\langle \mathbf a_k,-\mathbf p_k+\mathbf x\rangle,\qquad b_k=\langle \mathbf a_k,\mathbf p_k\rangle,$$14 form. This suggests that its lasting significance lies in two directions: first, as a mathematically unified library of hyperbolic neural components; second, as evidence that hyperbolic architectures are most compelling when hierarchical structure and parameter efficiency matter more than sheer ambient dimensionality.