Resonant Sparse Geometry Networks

Abstract: We introduce Resonant Sparse Geometry Networks (RSGN), a brain-inspired architecture with self-organizing sparse hierarchical input-dependent connectivity. Unlike Transformer architectures that employ dense attention mechanisms with O(n^2) computational complexity, RSGN embeds computational nodes in learned hyperbolic space where connection strength decays with geodesic distance, achieving dynamic sparsity that adapts to each input. The architecture operates on two distinct timescales: fast differentiable activation propagation optimized through gradient descent, and slow Hebbian-inspired structural learning for connectivity adaptation through local correlation rules. We provide rigorous mathematical analysis demonstrating that RSGN achieves O(n*k) computational complexity, where k << n represents the average active neighborhood size. Experimental evaluation on hierarchical classification and long-range dependency tasks demonstrates that RSGN achieves 96.5% accuracy on long-range dependency tasks while using approximately 15x fewer parameters than standard Transformers. On challenging hierarchical classification with 20 classes, RSGN achieves 23.8% accuracy (compared to 5% random baseline) with only 41,672 parameters, nearly 10x fewer than the Transformer baselines which require 403,348 parameters to achieve 30.1% accuracy. Our ablation studies confirm the contribution of each architectural component, with Hebbian learning providing consistent improvements. These results suggest that brain-inspired principles of sparse, geometrically-organized computation offer a promising direction toward more efficient and biologically plausible neural architectures.

• The paper introduces Resonant Sparse Geometry Networks that leverage sparse, hyperbolic embeddings and dynamic routing to achieve efficient hierarchical processing.
• The paper demonstrates sub-quadratic computational complexity and competitive performance on hierarchical classification and long-range dependency tasks with significantly fewer parameters.
• The paper validates that combining slow-timescale Hebbian plasticity with fast gradient descent yields stable, interpretable models suitable for neuromorphic and edge computing applications.

Resonant Sparse Geometry Networks: A Technical Evaluation

Biological Inspiration and Motivation

Resonant Sparse Geometry Networks (RSGN) (2601.18064) introduce a biologically-motivated neural network architecture that leverages principles of sparse, hierarchical, and dynamic connectivity. RSGN is directly motivated by the architectural and operational efficiency observed in biological neural systems, notably the cortex, which operates with extreme activation sparsity (1-2% of neurons active at any moment), input-dependent routing, local self-organized plasticity, and multiscale hierarchical embedding within real-world geometry. In contrast to mainstream architectures such as Transformers, which utilize dense global self-attention ($O(n^2)$ complexity), RSGN employs a learned hyperbolic geometry for node embedding and connectivity.

Figure 1: Bio-inspired principles underlying RSGN, mapping biological mechanisms onto the architecture's four design components.

Architectural Design

Core to RSGN is its embedding of $N$ computational nodes within a $d$-dimensional hyperbolic space (Poincaré ball model), where inter-node connection strengths decay exponentially with geodesic distance. Each node maintains both fast-evolving activation states (per input) and slow-adapting meta-parameters (embedding position, activation threshold, hierarchical level), explicitly separating fast activation dynamics from slow structural plasticity. The forwarding computation comprises four phases: input-dependent ignition (token-based spark points), iterative sparse propagation, local inhibition (winner-take-more competition), and output readout.

Dynamic sparsity is achieved by the ignition process, wherein input tokens are mapped to "spark points" in the hyperbolic space; only nodes proximate in hyperbolic distance become activated. The connection function is further modulated by learnable low-rank affinity factors and a hierarchical softplus gating favoring feedforward (i.e., up the hierarchy) connections.

Figure 2: RSGN architecture—nodes in hyperbolic space, input ignition, sparse activation, propagation, and local inhibition.

Activation propagation exploits a differentiable soft-threshold gating to maintain sparsity and enable end-to-end training via gradient descent, while the slow-timescale Hebbian learning adapts connectivity based on co-activation correlation modulated by task reward, enabling ongoing synaptic plasticity, pruning, and sprouting.

Theoretical Analysis

RSGN's computational complexity per step is $O(k m d_h^2)$, where $k$ is the number of active nodes and $m$ is average neighborhood size, making it sub-quadratic and practically linear under biologically plausible sparsity assumptions ($k, m \ll N$). The architecture supports universal approximation, as proven via embedding theorems and the capability to reconstruct layer-wise and tree-like computation with appropriately positioned nodes and thresholds.

The use of hyperbolic geometry ensures that tree-like hierarchical relationships can be embedded with logarithmically fewer dimensions and low distortion compared to Euclidean space, which is theoretically optimal for multiscale structures.

Empirical Evaluation

RSGN is evaluated on synthetic hierarchical sequence classification and long-range dependency tasks, contrasting against strong baselines (MLP, Transformer, LSTM, Sparse Transformer). Key results include:

• Hierarchical Classification (20-way): RSGN achieves 23.8% accuracy with 41,672 parameters, compared to Transformer at 30.1% with 403,348 parameters and random baseline at 5%. This represents nearly 5× the accuracy of random with 10× fewer parameters than Transformer.

  Figure 3: Accuracy comparison on hierarchical classification—RSGN achieves strong accuracy with an order of magnitude fewer parameters.

• Long-range dependency (length 128): RSGN attains 96.5% accuracy with 40,382 parameters—compared to Transformer/LSTM at 100% but using ~15× more parameters, confirming the ability to propagate information across long contexts efficiently.

  Figure 4: RSGN achieves near-maximal long-range accuracy with 15× parameter efficiency advantage.

• Ablation studies indicate that Hebbian structural learning contributes to convergence stability and provides modest but consistent accuracy gains, while the model maintains robustness to changes in propagation steps or node count.

  Figure 5: Ablation results—modularity and Hebbian plasticity contribute to parameter-insensitive stable performance.

• Parameter efficiency: RSGN occupies a favorable position on the accuracy-vs-parameter Pareto frontier, notably outperforming LSTM and Sparse Transformer baselines per-parameter and approaching Transformer accuracy with an order-of-magnitude fewer parameters.

  Figure 6: Pareto analysis—RSGN achieves competitive accuracy in the parameter-constrained regime.

Furthermore, training dynamics exhibit stable convergence and generalization across model sizes and seeds, indicating good optimization properties for sparse geometric architectures.

Figure 7: Training/validation dynamics show stable and consistent convergence for RSGN.

Figure 8: Summary plots—RSGN combines high accuracy, long-range capability, and parameter efficiency.

Implications and Limitations

Practical Implications

RSGN demonstrates a strong trade-off between parameter efficiency and accuracy, making it suitable for deployment scenarios with strict resource constraints (e.g., edge devices, embedded systems, or real-time BCI decoders). Critically, RSGN's scaling benefits become pronounced as sequence length and model size increase, due to sub-quadratic complexity and dynamic routing overhead being amortized by sparsity.

The architecture's explicit and visualizable hierarchical organization (from hyperbolic embeddings) provides natural interpretability, exposing network structure and node routing for inspection and debugging.

Theoretical Relevance

RSGN validates the computational viability of integrating slow-timescale unsupervised plasticity (Hebbian learning) with fast-timescale SGD in large, non-Euclidean, sparse neural architectures. This hybrid paradigm bridges the mechanistic gap between artificial and biological networks and may inform continual/online and unsupervised learning, as well as inspire future architecture innovations in non-Euclidean domains.

Limitations and Future Directions

While the parameter efficiency is compelling, absolute classification accuracy remains modestly below that of Transformers. Scaling to realistic NLP or vision benchmarks, optimizing GPU kernel efficiency for dynamic sparse computation, or exploring hybrid RSGN-Transformer backbones are immediate priorities. Furthermore, the hardware-agnostic advantages may be most fully realized in dedicated neuromorphic or event-driven computation platforms.

Potential extensions include:

• Application in neuromorphic hardware for real-time sparse event-based inference.
• Multimodal or cross-domain learning exploiting spatial separation in hyperbolic branches.
• Integration with continual learning frameworks to leverage structural plasticity for resilience to catastrophic forgetting.
• Validation on real biological data (EEG, ECoG, BCI signals) for clinical or prosthetic settings.

Conclusion

RSGN establishes a computational framework for biologically-inspired, sparse, hierarchical, and dynamically-routed deep learning. It achieves strong parameter-efficiency in structured tasks, competitive long-range modeling, and stable optimization. The approach reinforces the value of biological inductive biases—sparse activation, hyperbolic geometry, slow plasticity, and competitive inhibition—for neural architecture design. It opens new directions in efficient deep learning, interpretable models, and biologically plausible learning paradigms. For parameter-constrained or neuromorphic settings, RSGN is a compelling architecture for further investigation.