DFORM: Diffeomorphic vector field alignment for assessing dynamics across learned models (2402.09735v1)

Abstract: Dynamical system models such as Recurrent Neural Networks (RNNs) have become increasingly popular as hypothesis-generating tools in scientific research. Evaluating the dynamics in such networks is key to understanding their learned generative mechanisms. However, comparison of learned dynamics across models is challenging due to their inherent nonlinearity and because a priori there is no enforced equivalence of their coordinate systems. Here, we propose the DFORM (Diffeomorphic vector field alignment for comparing dynamics across learned models) framework. DFORM learns a nonlinear coordinate transformation which provides a continuous, maximally one-to-one mapping between the trajectories of learned models, thus approximating a diffeomorphism between them. The mismatch between DFORM-transformed vector fields defines the orbital similarity between two models, thus providing a generalization of the concepts of smooth orbital and topological equivalence. As an example, we apply DFORM to models trained on a canonical neuroscience task, showing that learned dynamics may be functionally similar, despite overt differences in attractor landscapes.

• The paper introduces DFORM as a method to align vector fields of learned models via a diffeomorphic transformation.
• It employs a Lie derivative-based loss and bidirectional invertible residual networks to establish smooth orbital equivalence.
• DFORM demonstrates high orbital similarity in diverse applications, highlighting its potential for analyzing complex RNN dynamics.

Assessing Dynamics Across Learned Models with DFORM

Introduction to DFORM

In the domain of dynamical system models, particularly Recurrent Neural Networks (RNNs), understanding the learned dynamics is crucial for grasping the generative mechanisms these models embody. The challenge arises when comparing different models, given their nonlinear nature and lack of a common coordinate system. Addressing this, the paper introduces DFORM (Diffeomorphic vector field alignment for comparing dynamics across learned models), a framework designed to align the dynamical behavior of different learned models by learning a nonlinear coordinate transformation. This transformation aims to provide a continuous, one-to-one mapping between models' trajectories, essentially approximating a diffeomorphism between them.

Theoretical Background

The paper situates DFORM within the broader scope of comparing dynamical systems, particularly focusing on smooth orbital equivalence - a concept from dynamical systems theory. Two systems are considered smoothly orbitally equivalent if there exists a one-to-one mapping (diffeomorphism) between their phase spaces, preserving their qualitative dynamics. However, establishing this equivalence for complex systems is non-trivial, and traditional methods, such as dimensionality reduction or visual inspection, fall short. DFORM addresses these limitations by introducing a novel approach based on learning an orbit-matching coordinate transformation, facilitated by invertible residual networks (i-ResNets).

Methodology

DFORM employs a loss function derived from the Lie derivative, enabling the comparison of two systems’ dynamics without needing to solve the systems numerically. A bidirectional architecture and training scheme enhance generalization and offer an approximate, differentiable inverse transformation. This setup allows DFORM to not only align vector fields of different dynamical systems but also to generate a continuous measure of orbital similarity, extending the idea of smooth orbital equivalence into a practical tool for comparing learned models.

Applications and Results

The paper presents several applications of DFORM, from simple linear transformations of known nonlinear systems (e.g., Van der Pol oscillators) to more complex scenarios involving large RNNs and task-specific neural network models. In these applications, DFORM is able to demonstrate high orbital similarity across systems that are analytically or functionally equivalent but may have vastly different representations or attractor landscapes. These results underscore the potential of DFORM as a powerful tool for dissecting and understanding the dynamics embedded within learned models.

Implications and Future Directions

The implications of DFORM extend beyond the theoretical novelty, offering practical utilities in comparing and understanding the dynamics of diverse learned models in scientific research. By providing a rigorous yet flexible framework for assessing dynamical similarity, DFORM opens new avenues for exploring the generative mechanisms of complex models. Future work could further refine the approach, for instance, by adapting DFORM to compare systems of different dimensionalities or incorporate regularization approaches to better match sample distributions.

Conclusion

The DFORM framework represents a significant step forward in comparing the dynamics of learned models. It overcomes previous hurdles associated with such comparisons by offering a principled, efficient, and scalable method grounded in the theory of dynamical systems. As such, it holds considerable promise for researchers seeking to explore the dynamics underlying complex learned models, paving the way for more informed interpretations and potentially the discovery of universal dynamics across diverse models.