Neural Nets as Quiver Representations

• Neural networks as quiver representations are a mathematical framework that encodes architectures through directed graphs and linear maps, highlighting symmetry and functional invariance.
• This approach connects neural compression, redundancy reduction, and optimization by leveraging tools from algebraic geometry and geometric invariant theory.
• It encompasses diverse architectures like MLPs, CNNs, and ResNets, offering insights into parameter space moduli and facilitating principled model reduction.

Neural networks can be formulated as quiver representations, providing a rigorous mathematical and geometric framework that directly encodes neural architectures, their parameter symmetries, moduli spaces, and functional invariances. This approach anchors the analysis of a broad spectrum of neural network architectures—including MLPs, convolutional nets, residual and skip connections—in the established toolkit of quiver representation theory, algebraic geometry, and geometric invariant theory. The moduli-theoretic perspective clarifies foundational issues such as redundancy, compression, optimization geometry, and functional equivalence.

1. Quiver Representations and Neural Network Architectures

A quiver $Q$ is a finite directed graph defined by a set of vertices $Q_0$ and a set of arrows $Q_1$, with source and target maps $s, t: Q_1 \to Q_0$. A representation of $Q$ with dimension vector $d = (d_i)_{i\in Q_0}$ assigns to each vertex $i$ a vector space $V_i \cong \mathbb{R}^{d_i}$ and to each arrow $\alpha \in Q_1$ a linear map $\varphi_\alpha: V_{s(\alpha)} \to V_{t(\alpha)}$ (Ganev et al., 2022, Armenta et al., 2020, Armenta et al., 2020).

A feed-forward neural network is encoded as an acyclic quiver:

• Vertices correspond to feature spaces (layers, neurons, or feature-maps).
• Arrows specify linear connections (weights) between these spaces.
• Loops at each hidden vertex encode pointwise activation functions.

Weights for each arrow are matrices (or scalars for thin representations), and activations $Q_0$0 are attached to non-source vertices. Typical architectures such as fully-connected nets, convolutional networks, residual/skip connections, and multi-branch architectures are directly expressed with specific quiver topologies and weight-sharing constraints (Ganev et al., 2022, Armenta et al., 2020). Bias terms are represented by augmenting the quiver with a distinguished bias vertex connected to every non-source vertex.

2. Parameter Symmetries, Isomorphisms, and Neural Teleportation

Quiver representations admit natural group actions corresponding to the change-of-basis (GL group) at hidden or internal vertices:

• For hidden vertex $Q_0$1, $Q_0$2 acts on the representation by $Q_0$3.
• In the scalar case, this is a rescaling at each neuron.

Neural teleportation refers to the action of these symmetries on the neural parameter space: for a choice of invertible $Q_0$4, set

• $Q_0$5 for each arrow,
• $Q_0$6 for each activation.

This transformation preserves the network function, i.e., for any input, the output remains invariant (Armenta et al., 2020, Armenta et al., 2020). For ReLU and other positive-scale-invariant activations, teleportation across positive $Q_0$7 leaves activations unchanged (intra-landscape symmetry).

Architectural constraints (e.g., preserving convolutional structure or residual identity maps) are enforced by restricting the allowable $Q_0$8, such as requiring constant scaling along feature maps or skip branches.

3. Moduli Spaces and Geometric Invariant Theory

The set of all quiver representations (with or without framings) modulo the symmetry group action forms the representation variety or moduli space $Q_0$9. For neural networks, this moduli space classifies all network parameterizations (weights, activations) that are functionally equivalent:

$Q_1$0

where $Q_1$1 denotes the appropriate (GIT or symplectic) quotient. Double or framed moduli spaces formalize inputs and outputs, producing spaces whose points classify entire functional equivalence classes of neural networks (Armenta et al., 2021, Armenta et al., 2020).

• The output of a neural network depends only on the corresponding moduli point: the network mapping factors through $Q_1$2.
• The dimension of $Q_1$3 governs functional capacity and redundancy; for thin ReLU nets, this is (number of arrows) minus (number of hidden neurons).
• Symplectic or Kähler structures on moduli spaces arise via canonical Hermitian metrics, moment maps, or Ricci curvature, equipping the learning landscape with an explicit geometric structure (Jeffreys et al., 2021, Armenta et al., 2021).

4. Applications: Compression, Redundancy, and Optimization

Symmetry reduction in parameter space leads to:

• Lossless model compression: Reduction of hidden layer widths or node dimensions without altering functionality, via QR-based algorithms when activations are rescaling or radial (Ganev et al., 2022). The reduced dimension at each vertex is $Q_1$4.
• Minimal subnetworks: The compressed subnetwork is minimal among all whose function coincides with the original.
• Gradient Descent Equivalence: Gradient descent on the compressed parameters is functionally equivalent (up to linear shifts and projections) to projected gradient steps on the original weights.
• Pruning and feature bottlenecks: Parameters unused by forward data flow are identified as "dead" and their moduli coordinates can be removed (pruned) without loss of expressivity (Armenta et al., 2020).

5. Invariances, Training Dynamics, and Moduli-Theoretic Implications

Functional invariance under parameter symmetries (teleportation) induces significant consequences:

• Level-set explorations: Parameter transformations traverse the same loss level sets or, with sign changes, new "inter-landscape" functions (Armenta et al., 2020).
• Loss landscape geometry: Teleportation (especially with large random $Q_1$5) is shown to sharpen local minima and increase normalized gradient norms, altering the loss landscape geometry in ways analogous to Dinh et al.'s findings (Armenta et al., 2020).
• Gradient rescaling: Gradients transform covariantly under teleportation, and in large-scale experiments, small teleportations are nearly orthogonal to batch-gradient updates, indicating SGD dynamics are compatible with moduli symmetries.
• Accelerated training: Initialization by a single large teleport can empirically accelerate convergence in standard tasks (CIFAR-10/100) for various architectures (Armenta et al., 2020).

Because the functional map factors through moduli, optimization and learning can theoretically be analyzed within the geometry of $Q_1$6—potentially with better-behaved flows and convergence properties.

6. Algebraic-Geometric, Kähler, and Toric Structures

For framed quiver representations (arising naturally for bias-incorporated neural nets), one obtains:

• Hermitian metrics (on universal bundles), with explicit Ricci curvature; these induce Kähler forms on the moduli spaces, making learning dynamics analyzable in a Kähler manifold context (Jeffreys et al., 2021).
• Toric moment maps facilitate construction of multi-variable activation functions (e.g., multi-softmax), connecting the activation geometry to symplectic and algebraic geometry.
• The universal approximation theorem extends to these settings: there exist network weights and multi-variable toric activations that approximate any continuous function on compact domains to arbitrary precision.

7. Architectural Variants and Open Directions

The quiver formalism is sufficiently general to encode:

• Standard and hybrid architectures (MLP, CNN, U-Net, ResNet).
• BatchNorm, layer-norm, pooling, and arbitrary wiring patterns via imposed combinatorial or weight-sharing constraints (Armenta et al., 2020).
• Pointwise, rescaling, and non-pointwise activations—though some compression and symmetry results require radial or rescaling types (Ganev et al., 2022).

Active research directions include:

• Extension of compression methods beyond rescaling activations.
• Moduli interpretations for convolutional/equivariant architectures over structured index sets.
• Explicit treatment of parameter symmetries arising from normalization layers.
• Study of non-convexities, global minima, and dynamics via Nakajima quiver varieties and Morse theory (Ganev et al., 2022, Jeffreys et al., 2021).

---

By adopting the perspective of neural networks as quiver representations, one achieves a unified, exact algebraic framework to analyze architecture, parameter redundancies, optimization geometry, and moduli-theoretic aspects. This allows for principled compression methods, rigorous identifiability results, and explicit links to geometric and topological invariants in both classical and modern neural architectures (Ganev et al., 2022, Armenta et al., 2020, Jeffreys et al., 2021, Armenta et al., 2020, Armenta et al., 2021).