Isomorphism in Neural Networks

• Neural Network Isomorphism is an equivalence relation ensuring functionally identical mappings despite differing internal parameterizations.
• It leverages quiver representation theory to model networks as directed graphs with invertible maps that preserve activation functions.
• The framework informs practical strategies like model compression, pruning, and initialization by revealing redundant symmetries in parameter spaces.

Isomorphism of Neural Networks refers to rigorous equivalence notions between neural network architectures and parameterizations that guarantee functionally identical input-output behavior. In algebraic terms, it involves formalizing when two neural networks represent the "same" map, despite potentially different internal parameter values. This concept is critical both for theoretical understanding—such as parameter space geometry, symmetry, and redundancy—and for practical implications like model compression, parameter initialization, and transfer learning.

1. Algebraic Formalization: Quiver Representation Theory

A canonical rigorous definition of neural network isomorphism is provided by quiver representation theory. Neural networks, specifically feedforward architectures, are modeled as "network quivers" $Q=(V,E,s,t)$, where $V$ is the set of vertices corresponding to input, hidden, bias, and output units, and $E$ the set of directed edges, each modeling a synaptic connection, complemented by loops for activation functions at each hidden or output neuron.

A neural network is precisely a quiver representation with activation functions: to each vertex $i\in V$ one associates a vector space (thin representation: $\mathbb{C}$), linear maps (multiplication by weight $W_\alpha$) for each edge $\alpha$, and a nonlinear map $f_i$ (the activation) for each loop $\ell_i$ at $i$. This formalism exactly recovers all standard neural network constructions (fully connected, convolutional, residual, batch-normalized, randomly wired) as quiver representations, enabling application of algebraic techniques from representation theory to neural network analysis (Armenta et al., 2020).

2. Isomorphism of Neural Network Representations

Two networks $V$0 and $V$1 on the same quiver $V$2 are isomorphic if there exist invertible linear maps $V$3 for each vertex $V$4 such that

• For each non-loop arrow $V$5, $V$6.
• For each loop $V$7 at $V$8, $V$9.

Isomorphic networks $E$0 and $E$1 yield identical input-output maps $E$2: for all $E$3, $E$4 (Armenta et al., 2020). In practical terms, isomorphism classes factor out functionally-invisible parameter reparametrizations or symmetries in the weight space. An important special case is ReLU networks, where positive-scaling invariance holds: simultaneous scaling of incoming weights and reciprocal scaling of outgoing weights at a neuron, with unchanged activations, yields isomorphic function classes.

3. The Moduli Space of Neural Network Isomorphism Classes

The space of all thin representations (weight assignments) for a network quiver $E$5 with dimension vector $E$6 is $E$7. The group $E$8 acts on this space by base change: $E$9 for each arrow $i\in V$0. The isomorphism classes of networks (modulo functional redundancy) are points of the GIT quotient, the "moduli space" $i\in V$1.

For thin representations (all $i\in V$2), the action is free on the locus of nonzero weights, and the dimension of the moduli space is $i\in V$3—thus, parameter counting modulo isomorphisms is governed by network wiring. Two networks are isomorphic if and only if their wiring and layer sizes match and their weight vectors lie in the same $i\in V$4-orbit; equivalently, they correspond to the same point in $i\in V$5 (Armenta et al., 2020).

4. Algebraic and Numerical Invariants for Isomorphism Detection

The classification up to isomorphism yields invariants:

• The dimension vector $i\in V$6 (layer sizes) and the quiver $i\in V$7 (topology) are discrete invariants.
• Within $i\in V$8, polynomial $i\in V$9-invariants such as "cycle-traces" or "path-products" generate numerical invariants. Practical algorithms gauge-fix $\mathbb{C}$0 weights using a spanning forest and compare remaining $\mathbb{C}$1 parameters—this enables efficient isomorphism testing or neural network pruning by detecting redundant parameterizations (Armenta et al., 2020).

5. Isomorphism in Permutation-Invariant and Graph Neural Architectures

For graph-structured data and permutation-invariant network classes, isomorphism is addressed in both architecture design and analysis:

• Message Passing Neural Networks (MPNNs) and GNNs reflect the ability to distinguish graph isomorphism classes through their expressive power compared to the Weisfeiler-Lehman tests. The classification capacity of such networks is constrained by communication capacity, which must scale at least as $\mathbb{C}$2 (general graphs) to realize full isomorphism discrimination (Loukas, 2020).
• Advanced neural architectures, such as Graph Partitioning Neural Networks (GPNNs), augment expressive power through permutation-invariant partitioning, achieving isomorphism discrimination close to higher-order Weisfeiler-Lehman tests with improved efficiency (Hevapathige et al., 2023).
• Hopfield networks trained on group-structured data can implicitly learn isomorphism classes by converging to a low-dimensional orbit-invariant parameter subspace, demonstrating the emergence of invariance under group actions without explicit architectural enforcement (Murray et al., 16 Dec 2025).

6. Practical and Theoretical Implications

Recognition of neural network isomorphisms directly impacts parameter space analysis (e.g., counting truly distinct models for a given architecture), optimization landscape topology, and the development of network compression and pruning methods. Function-preserving isomorphism theorems guarantee that various parameterizations are not functionally distinct, and moduli space characterizations provide a rigorous geometric framework for architecture comparison and transferability (Armenta et al., 2020).

Additionally, by relating isomorphism to group invariance and orbit structure, these frameworks link neural network theory to broader mathematical physics and geometry, enabling new directions for representation analysis and learning rule design (Murray et al., 16 Dec 2025).

7. Open Questions and Future Directions

Quiver-based isomorphism characterizations open numerous avenues for investigation:

• Determining the full structure of moduli spaces for networks with nonlinear activations beyond the thin representation case.
• Generalizing isomorphism criteria to richer activations and networks with parameter-sharing or heterogeneous wiring.
• Developing efficient algorithms for checking network isomorphism in large-scale practical settings.
• Investigating the learning dynamics and implicit biases of standard optimizers in the context of network moduli spaces and their quotient structures.

These directions underscore the foundational role of isomorphism theory in both theoretical understanding and algorithmic development in neural network research (Armenta et al., 2020).