Generalized Residual Network (GRN)

• Generalized Residual Networks (GRNs) are network architectures that extend standard ResNets by adapting residual connections to non-Euclidean spaces using tools like exponential maps and retractions.
• GRNs incorporate data-dependent gating mechanisms, such as the Gated Linear Unit (GLU), to modulate feature propagation and improve the flow of information in deep networks.
• Empirical evaluations of GRNs demonstrate enhanced performance, with improvements in recall, precision, and F1 scores on tasks involving manifold-valued data and transformer-based models.

A Generalized Residual Network (GRN) extends the classic residual neural network (ResNet) architecture to more general settings by systematically adapting the residual operation, in particular to Riemannian manifolds or by introducing nontrivial gating mechanisms in the residual pathway. Two distinct but complementary strands of research on GRN are prominent: (i) the Riemannian Residual Network, where residual updates respect the geometry of arbitrary Riemannian manifolds (Katsman et al., 2023), and (ii) the Gated Residual Network, which augments the residual function in a Transformer or feed-forward network with a data-dependent gating mechanism, such as a Gated Linear Unit (GLU) (Le et al., 2024). Both directions share the principle of dynamically modulating the contribution of the residual transformation, but differ in mathematical foundation, implementation, and application scope.

1. Generalizing Residual Networks: Algebraic and Geometric Foundations

The core mechanism of a Euclidean ResNet is the update $x_{\ell+1} = x_\ell + F(x_\ell; W_\ell)$, where $F:\mathbb{R}^n\to\mathbb{R}^n$ is a learned transformation. This summation operation is valid for Euclidean vector spaces but fails to generalize to nonlinear spaces where the notion of vector addition is undefined.

For a Riemannian manifold $(M,g)$, the update generalizes via the exponential map: $$x_{\ell+1} = \mathrm{Exp}_{x_\ell}(F(x_\ell; W_\ell)), \qquad F(x_\ell; W_\ell) \in T_{x_\ell}M$$ where $T_{x_\ell}M$ is the tangent space at $x_\ell$ and $\mathrm{Exp}_{x_\ell}$ maps a tangent vector onto the manifold along the geodesic. Retractions $R_{x}(\cdot)$ may be substituted for $\mathrm{Exp}_x$ to reduce computational cost (Katsman et al., 2023).

Alternatively, in the context of Gated Residual Networks, the residual mapping $F(x)$ in

$F:\mathbb{R}^n\to\mathbb{R}^n$0

is replaced by a gated function. For input $F:\mathbb{R}^n\to\mathbb{R}^n$1, the GRN computes: $F:\mathbb{R}^n\to\mathbb{R}^n$2 where $F:\mathbb{R}^n\to\mathbb{R}^n$3 is a non-linear projection of $F:\mathbb{R}^n\to\mathbb{R}^n$4, and the GLU selectively modulates feature propagation (Le et al., 2024).

2. Construction of Residual Maps and Gating Mechanisms

On Riemannian manifolds, the construction of $F:\mathbb{R}^n\to\mathbb{R}^n$5 must yield elements of $F:\mathbb{R}^n\to\mathbb{R}^n$6:

• Embedded Design: Embed $F:\mathbb{R}^n\to\mathbb{R}^n$7 and project a standard neural network output to the tangent space:

$F:\mathbb{R}^n\to\mathbb{R}^n$8

• Feature-Map-Induced Design: Map $F:\mathbb{R}^n\to\mathbb{R}^n$9 via $(M,g)$0 and use the pushforward and pullback to obtain tangent vectors as intrinsic combinations of gradients.

For gating in transformer-based GRNs, the GLU module is formally defined for input $(M,g)$1 as: $(M,g)$2 where $(M,g)$3 is a gating activation (e.g., sigmoid, Mish), $(M,g)$4 is element-wise multiplication, and all $(M,g)$5, $(M,g)$6 are learnable parameters (Le et al., 2024).

The following table compares the architectural insertion points and empirical performance effects for GRN blocks in Transformer models applied to artifact detection tasks:

GRN Integration Point	Validation Fluctuation	Recall (max)	F1 (max)
Intermediate Filter Layer	Low	0.97	0.93
Inside Attention Mechanism	High	0.95	0.91

Empirical results for GRN-Transformers on PPG artifact detection (selected gating activations; values from (Le et al., 2024)).

3. Algorithmic Implementation and Training on Manifolds

Riemannian ResNets require manifold-specific implementations of exponential maps, logarithmic maps, and parallel transport. For example, on the manifold of symmetric positive-definite (SPD) matrices with the affine-invariant metric:

• $(M,g)$7 comprises all symmetric matrices,
• $(M,g)$8,
• $(M,g)$9.

Residual blocks are stacked, possibly with layerwise mappings $$x_{\ell+1} = \mathrm{Exp}_{x_\ell}(F(x_\ell; W_\ell)), \qquad F(x_\ell; W_\ell) \in T_{x_\ell}M$$0 between manifolds. During training, gradients must be back-propagated using the differential of $$x_{\ell+1} = \mathrm{Exp}_{x_\ell}(F(x_\ell; W_\ell)), \qquad F(x_\ell; W_\ell) \in T_{x_\ell}M$$1 and, if necessary, parallel transported between tangent spaces. For efficiency, first-order retractions may be substituted for exponential maps when computational cost or closed-form intractability are issues.

For gated GRNs in standard models, dropout and layer normalization are combined with the gating mechanism. Dropout (rate $$x_{\ell+1} = \mathrm{Exp}_{x_\ell}(F(x_\ell; W_\ell)), \qquad F(x_\ell; W_\ell) \in T_{x_\ell}M$$2) is typically applied before the final gating layer.

4. Theoretical Properties: Stability, Expressivity, and Information Flow

A critical theoretical motivation for both Riemannian and gated GRNs is the preservation of gradient flow:

• Riemannian GRNs: Small-norm geodesic updates yield Jacobians close to the identity; the deviation is $$x_{\ell+1} = \mathrm{Exp}_{x_\ell}(F(x_\ell; W_\ell)), \qquad F(x_\ell; W_\ell) \in T_{x_\ell}M$$3 and governed by sectional curvature, preventing gradient explosion or vanishing (Katsman et al., 2023).
• Gated GRNs: The gating mechanism adaptively suppresses non-informative nonlinearities, safeguarding against unnecessary distortion of learned representations (Le et al., 2024).

Riemannian GRNs reduce to classic Euclidean ResNets when $$x_{\ell+1} = \mathrm{Exp}_{x_\ell}(F(x_\ell; W_\ell)), \qquad F(x_\ell; W_\ell) \in T_{x_\ell}M$$4 and manifold structure is flat. In this special case, $$x_{\ell+1} = \mathrm{Exp}_{x_\ell}(F(x_\ell; W_\ell)), \qquad F(x_\ell; W_\ell) \in T_{x_\ell}M$$5 and all geometric machinery becomes standard addition.

MINE (Mutual Information Neural Estimation) quantitatively demonstrates that GRN gating markedly increases mutual information between features and targets, with observed post-GRN mutual information $$x_{\ell+1} = \mathrm{Exp}_{x_\ell}(F(x_\ell; W_\ell)), \qquad F(x_\ell; W_\ell) \in T_{x_\ell}M$$6 pre-GRN in PPG artifact detection, indicating improved task-specific feature concentration (Le et al., 2024).

5. Empirical Performance and Benchmarks

For Riemannian GRNs:

• On hyperbolic graph datasets with varying Gromov $$x_{\ell+1} = \mathrm{Exp}_{x_\ell}(F(x_\ell; W_\ell)), \qquad F(x_\ell; W_\ell) \in T_{x_\ell}M$$7-hyperbolicity (e.g., Disease, Airport, PubMed, CoRA), RResNet Horo yielded superior recall and node classification F1, most notably on highly hyperbolic graphs ($$x_{\ell+1} = \mathrm{Exp}_{x_\ell}(F(x_\ell; W_\ell)), \qquad F(x_\ell; W_\ell) \in T_{x_\ell}M$$8), with up to $$x_{\ell+1} = \mathrm{Exp}_{x_\ell}(F(x_\ell; W_\ell)), \qquad F(x_\ell; W_\ell) \in T_{x_\ell}M$$9 F1 compared to prior HNN at $T_{x_\ell}M$0 (Katsman et al., 2023).
• In graph-neural RResNet (G-RResNet Horo), state-of-the-art F1=$T_{x_\ell}M$1 was achieved for node classification.
• On SPD-valued data (AFEW, FPHA, NTU RGB+D, HDM05), RResNet with log-Euclidean geometry outperformed SPDNet and its batch-normalized variant, reaching $T_{x_\ell}M$2 accuracy on AFEW versus $T_{x_\ell}M$3 for SPDNet+BN.

For transformer-based GRNs:

• The sigmoid-activated GLU and Mish-activated MiGLU blocks yielded highest accuracy and F1 (0.98, 0.93), with recall peaking at 0.97 (GLU) and precision at 0.91 (MiGLU).
• Fully-gated GnLU (sigmoid-sigmoid) achieved the highest overall F1 (0.94) and precision (0.91), underscoring the utility of symmetric gating (Le et al., 2024).
• Insertion of GRN as an intermediate filter layer substantially outperformed GRN-Attention fusion in data-scarce regimes.

6. Practical Considerations, Limitations, and Extensions

Selecting the vector field design $T_{x_\ell}M$4 (Riemannian terminology) or the gating activation (gated GRNs) is task-dependent:

• Riemannian GRNs: Horosphere projections in hyperbolic space and spectral-map designs in SPD-valued tasks provide strong trade-offs between expressivity and efficiency. Retraction can be used to approximate exponential map computation when closed-form expressions are unavailable or expensive.
• Gated GRNs: Sigmoid, Mish, and fully-gated activations consistently deliver high accuracy, precision, and recall.

Other practical insights include:

• Shallow network depths (2–4 blocks) suffice with expressive blocks.
• Learning rates: $T_{x_\ell}M$5 for SPD RResNet, $T_{x_\ell}M$6 for hyperbolic.
• GRN overhead is negligible relative to base Transformer complexity in data-constrained artifact detection tasks.
• Closed-form or efficient computation of manifold operations (Exp, Log, parallel transport) is a prerequisite; non-complete or non-Riemannian settings require careful design of retractions.

Extension to more exotic symmetric spaces (e.g., Grassmannians) and learning curvature adaptively are highlighted as promising future directions (Katsman et al., 2023).

7. Connections to Broader Architectures and Research Directions

The generalization of residual networks to non-Euclidean spaces via Riemannian structure fundamentally widens the applicability of deep neural models to data with intrinsic geometric or hierarchical structure. In parallel, the gating paradigm in residual connections aligns with ongoing trends optimizing information flow and feature selection within deep models, as observed in the equivalency to Gated Attention Units (GAU) (Le et al., 2024). The combined use of mutual information estimation provides a quantitative foundation for evaluating architectural innovations in representation learning.

The categorical conclusion is that Generalized Residual Networks, in both geometric (Riemannian) and algebraic (gated) realizations, deliver enhanced expressivity and stability, and outperform manifold- or task-specific alternatives on representative benchmarks while retaining elegant reduction to classical residual architectures in limiting cases (Katsman et al., 2023, Le et al., 2024).