Fully Hyperbolic LLMs: Intrinsic Hyperbolic Computation

• Fully hyperbolic LLMs are models that natively compute all operations—attention, normalization, and activations—in hyperbolic space, preserving intrinsic geometric properties.
• Intrinsic modules like Hyperbolic Transformation with Curvatures (HTC) and Hyperbolic Multi-Head Latent Attention (HMLA) enable scalable, efficient processing and reduce parameter redundancy compared to Euclidean methods.
• Empirical results show that these models achieve competitive performance and improved numerical stability while addressing limitations inherent in tangent-space hybrid architectures.

Fully hyperbolic LLMs are LLMs that operate entirely within hyperbolic space—typically the Lorentz hyperboloid or the Poincaré ball—from input embeddings through outputs and all intermediate computations. In this formulation, attention, feedforward transformations, normalization, activation, residual composition, positional encoding, and output heads are defined natively on the manifold rather than by repeatedly mapping to a Euclidean tangent space. The stated objectives are hierarchical reasoning, curvature-awareness, and numerical stability and efficiency through the reduction or elimination of back-and-forth exponential and logarithmic maps (Patil et al., 6 Sep 2025). The area emerged from earlier fully hyperbolic neural architectures for classification and general deep learning, then progressed to fully hyperbolic Transformers and, subsequently, billion-parameter hyperbolic LLMs (López et al., 2020, Chen et al., 2021, Yang et al., 2024, He et al., 30 May 2025).

1. Definition and research trajectory

The defining property of a fully hyperbolic LLM is end-to-end intrinsic computation in hyperbolic space. The survey "Hyperbolic LLMs" identifies fully hyperbolic LLMs as one of four principal technique categories for Hyperbolic LLMs and distinguishes them from models that rely on exp/log maps, hyperbolic fine-tuning, or hyperbolic state-space models (Patil et al., 6 Sep 2025). In that taxonomy, fully hyperbolic models never leave the manifold during computation, whereas hybrid systems perform substantial computation in tangent Euclidean spaces.

The broader lineage predates LLM-scale systems. "A Fully Hyperbolic Neural Model for Hierarchical Multi-Class Classification" describes a model in which every operation—input encoding, sequence modeling, attention, feature aggregation, and final classification—is performed natively in the Poincaré ball model, and reports performance on par with state-of-the-art methods together with up to 91% parameter reduction over BERT and 70% over ELMo-based denoised approaches on fine-grained entity typing (López et al., 2020). "Fully Hyperbolic Neural Networks" then generalized the program in the Lorentz model, arguing that core operations should be reformulated directly in hyperbolic space rather than in tangent-space relaxations (Chen et al., 2021). "Nested Hyperbolic Spaces for Dimensionality Reduction and Hyperbolic NN Design" extended the fully hyperbolic perspective to nested hyperbolic representations, intrinsic aggregation, and graph convolution (Fan et al., 2021).

The transition to Transformer-scale architectures is marked by Hypformer, described as a fully hyperbolic Transformer with native linear transformations, LayerNorm, activations, dropout, and concatenation in hyperbolic space, together with linear hyperbolic self-attention (Yang et al., 2024). HELM subsequently introduced fully hyperbolic LLMs at billion-parameter scale, with dense and mixture-of-curvature variants, hyperbolic latent attention, hyperbolic rotary positional encoding, and hyperbolic RMS normalization (He et al., 30 May 2025).

2. Geometric foundations and the case against tangent-space hybrids

The most common geometric substrate is the Lorentz model. In Hypformer and HELM, a point is split into time-like and space-like components, and the Lorentzian inner product is written as

$$\langle\mathbf{x}, \mathbf{y}\rangle_\mathcal{L} = -x_t y_t + \mathbf{x}_s^\top \mathbf{y}_s .$$

This model is repeatedly preferred for numerical stability and for intrinsic operations that preserve the hyperboloid constraint (Yang et al., 2024, He et al., 30 May 2025).

The theoretical motivation for full hyperbolicity is not merely aesthetic. "Fully Hyperbolic Neural Networks" proves that tangent-space linear transformation in prior hyperbolic networks is a relaxation of the Lorentz rotation and does not include the boost, thereby limiting expressive power (Chen et al., 2021). In that account, the common pattern of encoding in hyperbolic space but formalizing major operations in the tangent space restricts the network to a subset of Lorentz transformations.

Subsequent work sharpened this critique. "Fast and Geometrically Grounded Lorentz Neural Networks" proves that, with the current formulation of Lorentz linear layers, the hyperbolic norms of the outputs scale logarithmically with the number of gradient descent steps, nullifying the key advantage of hyperbolic geometry. The paper identifies this as a fundamental pathology of prevailing Lorentz model-based hyperbolic neural networks and states the practical consequence directly: to embed hierarchical structures that require large hyperbolic distance, a network must exponentially increase the weight matrix norm, yielding impractically slow training and an inability to leverage hyperbolic geometry’s hierarchy embedding properties (Klis et al., 29 Jan 2026).

A related negative result appears in "Hyperbolic Fine-tuning for LLMs". That work is not a fully hyperbolic LLM, but it shows why naive transfer of Euclidean adaptation recipes is inadequate: if embeddings and weight matrices both reside in Euclidean space, a naive application of exponential and logarithmic maps reduces to the Euclidean update through cancellation. The authors present this as a fundamental limitation of naive hyperbolic fine-tuning and use it to motivate direct manifold-based low-rank adaptation (Yang et al., 2024). Taken together, these results frame fully hyperbolic LLMs as a response to expressivity loss, geometric distortion, and computational overhead introduced by hybridization.

3. Intrinsic modules and architectural primitives

A fully hyperbolic LLM requires a complete set of intrinsic modules rather than a hyperbolic attention mechanism alone. Hypformer states that previous attempts mainly focused on modifying self-attention and fell short because well-defined modules for linear transformations, LayerNorm, activation functions, dropout, and related operations were missing in hyperbolic space (Yang et al., 2024). Its two foundational blocks address precisely that gap. Hyperbolic Transformation with Curvatures (HTC) implements linear transformations directly in hyperbolic space and can map between manifolds of different curvatures while preserving relative distances and orderings. Hyperbolic Readjustment and Refinement with Curvatures (HRC) generalizes LayerNorm, activation, dropout, and concatenation as intrinsic refinements acting on the space-like coordinates.

Hypformer’s attention mechanism is likewise intrinsic in construction but computationally rearranged through the space-like coordinates:

$$Z_s = \frac{Q_s (K_s^T V_s)}{Q_s (K_s^T \mathbf{1})}.$$

The paper states that this yields $\mathcal{O}(N)$ time complexity in sequence length or number of nodes, assuming head dimension $d' \ll N$, and thereby addresses the quadratic bottleneck of prior hyperbolic self-attention (Yang et al., 2024).

HELM expands the intrinsic module set in LLM-specific directions. It introduces Hyperbolic Multi-Head Latent Attention (HMLA), Hyperbolic Rotary Positional Encoding (HoPE), and hyperbolic RMS normalization. HoPE generalizes RoPE to Lorentzian hyperbolic space by applying Euclidean rotations to the space-like part while leaving the time-like component intact, and the paper states that it maintains relative position dependence, long-range decay, and expressiveness for arbitrary attention patterns (He et al., 30 May 2025). RMSNorm$_\mathcal{L}$ is presented as the first fully hyperbolic root mean square normalization layer and is described as invariant to input scaling while maintaining stability during both forward and backward passes (He et al., 30 May 2025).

Independent work on Lorentz-native modules has been important for the feasibility of fully hyperbolic LLMs. "Fast and Geometrically Grounded Lorentz Neural Networks" replaces problematic Lorentz linear layers with a distance-to-hyperplane formulation and introduces Lorentzian activation functions and a caching strategy. The paper states that the new layer restores linear scaling of output hyperbolic norms with respect to the number of gradient descent steps and that the total package bridges the computation gap to Euclidean neural networks (Klis et al., 29 Jan 2026). "Intrinsic Lorentz Neural Network" similarly replaces Euclidean affine logits with a point-to-hyperplane fully connected layer and adds GyroLBN, a Lorentz patch-concatenation operator, a gyro-additive bias, and Lorentz dropout, with the explicit goal of performing all computations intrinsically within the Lorentz model (Shi et al., 27 Feb 2026).

System	Core intrinsic modules	Reported property
Hypformer (Yang et al., 2024)	HTC, HRC, linear hyperbolic self-attention	First complete hyperbolic Transformer; $\mathcal{O}(N)$ attention
HELM / HELM-MiCE (He et al., 30 May 2025)	HMLA, HoPE, RMSNorm$_\mathcal{L}$, Mixture-of-Curvature Experts	First fully hyperbolic LLMs at billion-parameter scale
FGG-LNN (Klis et al., 29 Jan 2026)	Distance-to-hyperplane Lorentz linear layer, Lorentzian activations, caching	2–3× faster inference than prior Lorentz implementations
ILNN (Shi et al., 27 Feb 2026)	Point-to-hyperplane FC, GyroLBN, log-radius concatenation, Lorentz dropout	State-of-the-art performance and computational cost among hyperbolic models

The module-level picture is therefore broader than attention alone. Fully hyperbolic LLMs depend on a closure property: every standard Transformer primitive must admit a geometry-respecting analogue that keeps representations on the manifold and does not reintroduce Euclidean artifacts through fallback operations.

4. Representative systems

Hypformer is the first comprehensive fully hyperbolic Transformer in the supplied corpus. It is built on the Lorentz model, defines all key modules natively in hyperbolic space, and introduces linear hyperbolic self-attention to enable processing of billion-scale graph data and long-sequence inputs for the first time (Yang et al., 2024). Its formulation allows trainable curvature, avoids repeated tangent-space mappings, and is intended to preserve relative ordering of distances under curvature-changing transformations. Although presented as a Transformer rather than an autoregressive LLM, it is a direct architectural precursor because it solves the missing-module problem that previously prevented a complete hyperbolic Transformer stack.

HELM is the first work in the corpus to present fully hyperbolic LLMs at billion-parameter scale (He et al., 30 May 2025). It defines two architectures. HELM-D is a dense, purely hyperbolic baseline analogous to LLaMA but entirely in hyperbolic space. HELM-MiCE introduces a Mixture-of-Curvature Experts design in which each expert operates in a distinct curvature space. The stated rationale is that token spaces are highly variable in negative curvature, so a single curvature is insufficient to capture fine-grained geometric structure. Expert outputs are combined through a Lorentzian centroid, and the model uses HMLA to reduce KV-cache and computational demands during training and inference (He et al., 30 May 2025).

The mixture-of-curvature design is one of the distinctive conceptual departures of fully hyperbolic LLMs from earlier hyperbolic models. Whereas many prior systems used a single curvature, HELM-MiCE treats curvature as a representation axis in its own right. The paper reports that variable-curvature MiCE outperforms fixed-curvature MiCE-Const, presenting this as evidence for curvature diversity rather than a single-manifold assumption (He et al., 30 May 2025). This suggests that, within fully hyperbolic LLMs, curvature selection may be as architecturally consequential as the choice of attention mechanism or feedforward block.

5. Scaling behavior, efficiency, and empirical evidence

The empirical record is distributed across several stages of the literature. Early fully hyperbolic NLP work already showed that end-to-end hyperbolic computation could be parameter-efficient and hierarchy-sensitive. The 2020 fully hyperbolic entity typing model infers latent hierarchy from class distribution, captures implicit hyponymic relations in the inventory, and achieves on-par performance with state-of-the-art methods while reducing parameter size; its Hyperbolic xLarge model reports ultra-fine F1 of 25.7 against 23.4 for BERT and 25.2 for the denoised baseline, at 9.5M parameters rather than 110M or 31M (López et al., 2020).

Hypformer provides the strongest Transformer-scale evidence for efficient intrinsic hyperbolic computation prior to LLM-scale pretraining. On ogbn-papers100M, described as having 0.11 billion nodes and 1.6 billion edges, Hypformer is reported as the first hyperbolic approach to complete tasks at this scale and achieves 66.1% accuracy versus a runner-up at 65.8% (Yang et al., 2024). The same work reports a 10× reduction in GPU cost and halved training time compared to prior hyperbolic softmax attention, with memory usage scaling linearly rather than suffering quadratic out-of-memory behavior (Yang et al., 2024).

HELM is the direct scaling result for fully hyperbolic LLMs. It reports stable training up to 1B parameters and evaluation on MMLU, ARC (Challenging), CommonsenseQA, HellaSwag, and OpenbookQA. The paper states that HELM architectures show consistent gains—up to 4%—over Euclidean architectures used in LLaMA and DeepSeek, and that while hyperbolic operations incur roughly 1.5–1.8× more runtime than Euclidean ones, the HMLA design narrows the gap (He et al., 30 May 2025).

Module-focused Lorentz work strengthens the claim that fully hyperbolic LLMs are no longer blocked by obvious efficiency barriers. FGG-LNN reports inference accelerated by a factor of 2–3× over prior Lorentz implementations and more than 8× over Poincaré implementations, while training times for ResNet-18 on CIFAR-100 are reduced by 3.5× relative to the previous Lorentz state of the art and 7.5× relative to Poincaré-model networks (Klis et al., 29 Jan 2026). ILNN reports 95.36% accuracy on CIFAR-10 and 78.41% on CIFAR-100, together with state-of-the-art performance and computational cost among hyperbolic models; it also improves hyperbolic graph baselines such as Hypformer on AIRPORT from 95.0% to 96.0% through its point-to-hyperplane head (Shi et al., 27 Feb 2026). These are not LLM benchmarks, but they support the practical claim that intrinsic Lorentz modules can now be trained at competitive speeds and depths.

6. Distinctions, misconceptions, and emerging directions

A persistent misconception is that a model becomes fully hyperbolic once token embeddings are placed in hyperbolic space. The survey literature rejects that equivalence: fully hyperbolic models require that all operations, including attention, MLPs, normalization, activations, and residuals, are performed on the hyperbolic manifold via Möbius or Lorentz frameworks (Patil et al., 6 Sep 2025). By this criterion, many earlier “hyperbolic” systems are better described as partially hyperbolic or hybrid.

A second misconception is that tangent-space mediation is a harmless implementation detail. The literature instead presents several concrete failure modes: loss of Lorentz boosts in tangent-space linearization, logarithmic rather than linear growth of output hyperbolic norms under prevailing Lorentz linear layers, and exp/log cancellation in naive hyperbolic fine-tuning (Chen et al., 2021, Klis et al., 29 Jan 2026, Yang et al., 2024). These arguments do not imply that hybrid methods are useless; rather, they locate a principled boundary between geometric approximation and intrinsic hyperbolic computation.

A third open question concerns whether a single curvature should govern the whole model. HELM-MiCE argues against that assumption by using experts at distinct curvatures and reporting that variable-curvature MiCE outperforms fixed-curvature alternatives (He et al., 30 May 2025). A plausible implication is that future fully hyperbolic LLMs may treat curvature as a learned, modular resource, especially when token spaces exhibit heterogeneous local geometry.

The application frontier is also broadening. GRAIL is not a fully hyperbolic LLM, because the LLM is used only as a constrained inference-time reranker, but it is instructive as a geometry-aware deployment pattern. It embeds a unified clinical graph in hyperbolic space, summarizes visits as hyperbolic Central Events, retrieves hierarchy- and trajectory-consistent candidates, and then combines geometric ranking with LLM reranking. On MIMIC-IV, the framework is reported to improve multi-type next-visit prediction and produce more hierarchy-consistent forecasts (Qu et al., 13 Feb 2026). This suggests a near-term interface between hyperbolic representation learning and LLM reasoning even when the LLM itself is not yet fully intrinsic.

Across the literature, fully hyperbolic LLMs are therefore best understood as the convergence of three lines of work: intrinsic hyperbolic neural operations, scalable hyperbolic Transformer design, and LLM-specific modules for attention, positional encoding, normalization, and expert routing. The field’s central claim is not that hyperbolicity is universally preferable, but that when language or downstream structure is hierarchical, tree-like, long-tailed, or curvature-heterogeneous, end-to-end hyperbolic computation can preserve representational properties that hybrid Euclideanized pipelines partially erase (Patil et al., 6 Sep 2025, He et al., 30 May 2025).