Hyperbolic Deep Reinforcement Learning

• Hyperbolic deep reinforcement learning is a method that embeds RL components in hyperbolic spaces to efficiently encode hierarchical structures.
• It transforms state features, actions, and value functions using Riemannian mappings and Möbius operations to capture complex decision processes.
• HDRL employs hyperbolic discounting and stabilization techniques like RMSNorm and spectral regularization to enhance training stability and performance.

Hyperbolic deep reinforcement learning (HDRL) integrates hyperbolic geometry—a geometry of constant negative curvature—directly into the algorithmic core of reinforcement learning systems, including feature representation, policy parameterization, and temporal credit assignment. Hyperbolic models exploit their exponential volume growth to efficiently encode hierarchical structures, enabling reinforcement learning (RL) agents to reason about complex, multi-level decision processes and achieve robust generalization. This approach encompasses three primary axes: (i) embedding agent representations, actions, or value functions in hyperbolic manifolds, (ii) leveraging hyperbolic discounting as a temporal inductive bias, and (iii) developing stable optimization and regularization procedures required for non-Euclidean deep learning dynamics.

1. Mathematical Foundations of Hyperbolic Spaces in RL

The mathematical core of HDRL relies on two primary models of hyperbolic geometry: the Poincaré ball and the hyperboloid (Lorentz) model (Cetin et al., 2022, Jaćimović et al., 2024, Klein et al., 16 Dec 2025). In $d$ dimensions and curvature $-c<0$, the Poincaré ball is: $$\mathbb B^d_c = \{x \in \mathbb R^d : \|x\| < 1/\sqrt{c}\}$$ with Riemannian metric

$$g^{\mathbb B}_x(u, v) = \frac{4}{(1 - c \|x\|^2)^2} \langle u, v \rangle.$$

Geodesic distance is

$$d_{\mathbb B}(x,y) = \operatorname{arcosh}\left(1 + 2c \frac{\|x-y\|^2}{(1-c\|x\|^2)(1-c\|y\|^2)}\right).$$

Hyperbolic operations such as Möbius addition and exponential/logarithmic maps underpin hyperbolic neural operations: $$\mathrm{Exp}_0(v) = \tanh(\sqrt{c}\|v\|) \frac{v}{\sqrt{c}\|v\|}$$ and its inverse

$$\mathrm{Log}_0(x) = \operatorname{artanh}(\sqrt{c}\|x\|) \frac{x}{\sqrt{c}\|x\|}.$$

On the hyperboloid, $$d_\mathbb{H}(x,y) = \operatorname{arcosh}\left(-\langle x, y \rangle_L\right)$$ where $\langle\cdot,\cdot\rangle_L$ is the Lorentzian form. Mapping between models is standard and offers route-dependent tradeoffs in optimization and gradient behavior.

Hyperbolic geometry permits embedding of tree-like or hierarchical data with exponentially lower distortion and higher compactness than Euclidean space, motivating its use when RL environments exhibit implicit or explicit hierarchy (Cetin et al., 2022, Xu et al., 21 Jul 2025).

2. Hyperbolic Representations in Deep RL Architectures

HDRL applies Riemannian geometry at multiple representational levels. State features, actions, or value functions are embedded in the Poincaré ball or hyperboloid via exponential maps of Euclidean neural features, producing representations $x_H = \mathrm{Exp}_0(x_E)$ (Cetin et al., 2022, Klein et al., 16 Dec 2025). Hyperbolic neural layers extend classical affine and activation operators using Möbius-algebra counterparts and Riemannian mappings.

A representative HDRL network follows this sequence:

• Standard (Euclidean) neural trunk processes the input.
• Final feature vector is regularized and normalized (e.g., via RMSNorm or spectral normalization) to ensure stable hyperbolic mapping (Klein et al., 16 Dec 2025, Cetin et al., 2022).
• Hyperbolic exponential map embeds features in $-c<0$0 or $-c<0$1.
• Policy and critic heads are implemented as hyperbolic multi-class logistic regression, using gyroplane projections or Lorentz-compatible mappings.
• All updates to parameters on the manifold use natural (Riemannian) gradients or their practical surrogates.

Hyperbolic Transformers further apply these methods within sequential-decision architectures, replacing all affine and attention modules with their Möbius-algebra analogues (Xu et al., 21 Jul 2025).

3. Temporal Structure: Hyperbolic Discounting and Multi-Horizon Value Learning

Classical RL uses exponential discounting $-c<0$2. In contrast, hyperbolic discounting uses

$-c<0$3

with $-c<0$4, matching human temporal preference data and approximating discounting via a mixture-of-exponentials (Fedus et al., 2019). Any hyperbolic (or more generally, non-exponential) discount can be written as

$-c<0$5

inducing

$-c<0$6

Practical algorithms maintain $-c<0$7 value heads $-c<0$8, each with a different $-c<0$9, and synthesize hyperbolic returns by weighted sum (Fedus et al., 2019).

Empirically, hyperbolic discounting provides robust value estimation under episodic hazard, and multi-horizon auxiliary value learning (learning Q-values for many $$\mathbb B^d_c = \{x \in \mathbb R^d : \|x\| < 1/\sqrt{c}\}$$0 in parallel) consistently improves sample efficiency and final performance on challenging RL domains (Fedus et al., 2019).

4. Optimization, Stability, and Regularization in Hyperbolic Deep RL

Naïve implementation of hyperbolic layers in RL commonly results in gradient instability, with phenomena such as exploding or vanishing gradients, especially harmful under the nonstationarity of PPO or TD losses (Cetin et al., 2022, Klein et al., 16 Dec 2025). Root causes include:

• Gradient amplification due to the conformal factor $$\mathbb B^d_c = \{x \in \mathbb R^d : \|x\| < 1/\sqrt{c}\}$$1 in the Poincaré model.
• Unbounded feature norms leading to divergent Jacobians in both Poincaré and hyperboloid exponential maps.
• Trust-region violations in PPO, with off-batch states failing to satisfy the KL constraint when hyperbolic encoders drift.

Key regularization/optimization strategies developed for HDRL include:

• Spectrally-Regularized Hyperbolic Mappings (S-RYM): Spectral normalization of all trunk layers plus output rescaling to $$\mathbb B^d_c = \{x \in \mathbb R^d : \|x\| < 1/\sqrt{c}\}$$2 norm (Cetin et al., 2022).
• RMSNorm Layer and Learned Scaling: Normalize and re-scale features just prior to hyperbolic mapping, bounding all downstream Jacobians (Klein et al., 16 Dec 2025).
• Categorical Critic Loss: Replace value regression by categorical value distribution matching (cross-entropy versus mean-square), which bounds critic gradients and harmonizes geometry with hyperbolic MLR (Klein et al., 16 Dec 2025).
• Optimization-Friendly Hyperbolic Layers: Design hyperboloid-parameterized logistic regression layers immune to conformal factor blow-up and with well-behaved gradients (Klein et al., 16 Dec 2025).

Table: Summary of stability-driven techniques

Issue	Solution	Cited Paper
Exploding/vanishing gradients	S-RYM, RMSNorm, learned scaling	(Cetin et al., 2022, Klein et al., 16 Dec 2025)
PPO trust-region violations	Bounding feature norm, categorical loss	(Klein et al., 16 Dec 2025)
Hyperbolic layer gradient pathology	Hyperboloid MLR, no conformal factor	(Klein et al., 16 Dec 2025)

5. Algorithmic Instantiations and Applications

HDRL algorithms have been instantiated as:

• Hyperbolic PPO and DQN: All operations from state encoding to actor/critic heads embedded and processed in hyperbolic space, optimized via Riemannian-adapted SGD or Adam (Cetin et al., 2022, Klein et al., 16 Dec 2025).
• Hyperbolic Transformers for RL: Sequential policy representations apply Möbius and hyperbolic operations throughout, particularly effective in multi-step mathematical reasoning and control tasks (Xu et al., 21 Jul 2025).
• Mixture-of-Exponentials Q-learning: Hyperbolic discounting and multi-horizon value learning implemented as multi-head network architectures, robust under hazard and improving generalization (Fedus et al., 2019).
• Black-box optimization in hyperbolic space: Covariance Matrix Adaptation or similar evolutionary strategies adapted to sample and optimize policies on hyperbolic manifolds (Jaćimović et al., 2024).

Empirical benchmarks demonstrate:

• Substantial improvements in sample efficiency and normalized returns in ProcGen and Atari-100K (Cetin et al., 2022, Klein et al., 16 Dec 2025).
• Gains in generalization, particularly with small latent dimensions (e.g., low-dimensional hyperbolic representation still outperforming high-dimensional Euclidean) (Cetin et al., 2022).
• Enhanced accuracy and computational efficiency in multi-step reasoning (FrontierMath, nonlinear optimal control), with 32–44% improvement in accuracy and up to 32% reduction in wall-clock time (Xu et al., 21 Jul 2025).

6. Interpretations, Limitations, and Open Problems

Hyperbolic geometry provides a natural inductive bias for tasks characterized by hierarchy, tree expansion, or chain-of-thought reasoning. Volume growth allows compact encoding of exponentially growing uncertainty or action/state trees (Cetin et al., 2022, Xu et al., 21 Jul 2025). Geodesic separation improves credit assignment and reduces path overlap in multi-step decision making (Xu et al., 21 Jul 2025).

Primary limitations include:

• Numerical instability due to Möbius operations near manifold boundaries.
• Fixed curvature parameter; no current mechanisms adapt curvature online or per layer.
• Additional implementation complexity due to Möbius algebra and Riemannian gradients.
• Lack of rigorous convergence guarantees for Riemannian RL with function approximation (Xu et al., 21 Jul 2025, Cetin et al., 2022).

Potential directions include:

• Learnable or schedule-adaptive curvature.
• Extensions to actor–critic, model-based, and offline RL with hyperbolic manifolds.
• Deeper theoretical analysis of generalization and convergence.
• Symbolic-hyperbolic RL hybrids and application to massive-scale transformers (Xu et al., 21 Jul 2025).

7. Variants, Empirical Findings, and Design Principles

Distinct design patterns recur in successful HDRL systems:

• Policy and value heads should use hyperboloid MLR for robust training gradients (Klein et al., 16 Dec 2025).
• Layerwise feature normalization is critical to ensure stability for both on-policy and off-policy settings (Klein et al., 16 Dec 2025, Cetin et al., 2022).
• Multi-horizon value learning (multi-γ auxiliary heads) is universally beneficial, independent of whether hyperbolic discounting is actually used for policy/prediction (Fedus et al., 2019).
• Embedding models exploiting tree-structured or hierarchical dependencies outperform Euclidean baselines on tasks with such latent structure, often with higher parameter efficiency (Xu et al., 21 Jul 2025, Cetin et al., 2022).
• Black-box optimization remains feasible with manifold-valued policy parameters if proper tangent space mappings and retractions are used (Jaćimović et al., 2024).

The cumulative evidence indicates that HDRL methodologies confer distinctive advantages in hierarchical, hazardous, or reasoning-centric RL domains, with ongoing innovation in optimization stability, architecture, and geometric task alignment (Cetin et al., 2022, Klein et al., 16 Dec 2025, Jaćimović et al., 2024, Xu et al., 21 Jul 2025, Fedus et al., 2019).