Optimal Recurrent Network Topologies for Dynamical Systems Reconstruction

Abstract: In dynamical systems reconstruction (DSR) we seek to infer from time series measurements a generative model of the underlying dynamical process. This is a prime objective in any scientific discipline, where we are particularly interested in parsimonious models with a low parameter load. A common strategy here is parameter pruning, removing all parameters with small weights. However, here we find this strategy does not work for DSR, where even low magnitude parameters can contribute considerably to the system dynamics. On the other hand, it is well known that many natural systems which generate complex dynamics, like the brain or ecological networks, have a sparse topology with comparatively few links. Inspired by this, we show that geometric pruning, where in contrast to magnitude-based pruning weights with a low contribution to an attractor's geometrical structure are removed, indeed manages to reduce parameter load substantially without significantly hampering DSR quality. We further find that the networks resulting from geometric pruning have a specific type of topology, and that this topology, and not the magnitude of weights, is what is most crucial to performance. We provide an algorithm that automatically generates such topologies which can be used as priors for generative modeling of dynamical systems by RNNs, and compare it to other well studied topologies like small-world or scale-free networks.

• The paper introduces geometric pruning that identifies essential network connections, preserving invariant dynamical attractor structures.
• The paper finds that specific topological features, akin to small-world or scale-free networks, correlate with enhanced reconstruction performance.
• The paper presents an algorithm generating GeoHub network topologies that achieve efficient sparsity and faster training without sacrificing model fidelity.

Optimal Recurrent Network Topologies for Dynamical Systems Reconstruction

The paper "Optimal Recurrent Network Topologies for Dynamical Systems Reconstruction" presents a study exploring the suitability of various recurrent network topologies for reconstructing dynamical systems from time-series data using recurrent neural networks (RNNs). The authors address the challenge of distilling a dynamical system's model from observed data with a focus on reducing parameter load while maintaining or improving model performance.

Core Contributions

The paper introduces "geometry-based pruning" as a technique to identify essential network connections based on their contribution to the invariant geometrical structure of the system attractor. This approach is a departure from traditional magnitude-based pruning, which removes network parameters based on their absolute value, an approach found to be insufficiently accurate in the context of dynamical systems reconstruction (DSR).

1. Geometric Pruning Overcomes Limitations of Magnitude-Based Pruning: The authors demonstrate that low-magnitude parameters can still play a crucial role in the system dynamics, implying that magnitude-based pruning is suboptimal for DSR tasks. Geometric pruning instead focuses on the contribution to invariant geometric structures, leading to significantly sparser models without a marked loss in DSR quality.
2. Topological Insights from Pruned Networks: By analyzing the networks resulting from geometric pruning, the authors identify consistent topological features that correlate with model performance. The presence of specific network topologies, akin to small-world or scale-free structures, emerged as crucial. It was determined that the topology, rather than the specific values of the parameters, was key to maintaining high model fidelity post-pruning.
3. Algorithm for Generating Optimal Network Topologies: The research proposes an algorithm that generates RNN topologies reflecting those observed after geometric pruning. These topologies, dubbed GeoHub, serve as performance-enhancing priors for network initialization in generative modeling tasks. GeoHub networks achieved a balance between connection sparsity and model robustness, aligning closely with natural systems known for sparse yet efficient topological arrangements.

Numerical Results and Comparative Analysis

The paper provides quantitative evidence through experiments on multiple benchmarks, including classical chaotic systems like the Lorenz-63 and the Rössler system, as well as real-world data like human ECG signals. The results consistently show that networks initialized with GeoHub topologies outperform those with random or traditional topologies, particularly in maintaining attractor geometry and temporal structure fidelity.

1. Performance Metrics: Evaluations based on attractor geometry (via state space divergence) and temporal dynamics (via power spectrum helicon) verify the enhanced capability of GeoHub initializations. The experiments affirm that these networks can maintain performance levels equivalent to densely connected networks, with significantly fewer parameters.
2. Training Efficiency: GeoHub networks not only satisfy performance benchmarks but also exhibit faster convergence during training, effectively reducing computational expense.

Implications and Future Directions

This work contributes to the broader understanding of how neural network topology influences learning and generalization in dynamical systems reconstruction. By establishing a method to derive near-optimal network structures, the research potentially shifts focus from parameter tuning to topological design in DSR applications.

Future studies might explore the application of these findings across different types of neural networks beyond RNNs, such as attention-based models or convolutional networks, especially for tasks where capturing long-term dependencies is critical. Additionally, exploration into how these pruning-focused insights can interlock with other neural architecture search methods could lead to more generalized solutions with broader applicability in neural engineering and systems neuroscience.