---
title: Neural Nets as Quiver Representations
url: https://www.emergentmind.com/topics/neural-networks-as-quiver-representations
type: topic
---

# Neural Nets as Quiver Representations

Neural networks can be formulated as quiver representations, providing a rigorous mathematical and geometric framework that directly encodes neural architectures, their parameter symmetries, moduli spaces, and functional invariances. This approach anchors the analysis of a broad spectrum of neural network architectures—including MLPs, convolutional nets, residual and skip connections—in the established toolkit of quiver representation theory, algebraic geometry, and geometric invariant theory. The moduli-theoretic perspective clarifies foundational issues such as redundancy, compression, optimization geometry, and functional equivalence.

## 1. Quiver Representations and Neural Network Architectures

A quiver $Q$ is a finite directed graph defined by a set of vertices $Q_0$ and a set of arrows $Q_1$, with source and target maps $s, t: Q_1 \to Q_0$. A representation of $Q$ with dimension vector $d = (d_i)_{i\in Q_0}$ assigns to each vertex $i$ a vector space $V_i \cong \mathbb{R}^{d_i}$ and to each arrow $\alpha \in Q_1$ a linear map $\varphi_\alpha: V_{s(\alpha)} \to V_{t(\alpha)}$ [2207.12773, 2012.01118, 2007.12213].

A feed-forward neural network is encoded as an acyclic quiver:
- Vertices correspond to feature spaces (layers, neurons, or feature-maps).
- Arrows specify linear connections (weights) between these spaces.
- Loops at each hidden vertex encode pointwise activation functions.

Weights for each arrow are matrices (or scalars for thin representations), and activations $\rho_i$ are attached to non-source vertices. Typical architectures such as fully-connected nets, convolutional networks, residual/skip connections, and multi-branch architectures are directly expressed with specific quiver topologies and weight-sharing constraints [2207.12773, 2007.12213]. Bias terms are represented by augmenting the quiver with a distinguished bias vertex connected to every non-source vertex.

## 2. Parameter Symmetries, Isomorphisms, and Neural Teleportation

Quiver representations admit natural group actions corresponding to the change-of-basis (GL group) at hidden or internal vertices:
- For hidden vertex $i$, $g_i \in GL(d_i)$ acts on the representation by $(g \cdot W)_\alpha = g_{t(\alpha)} W_\alpha g_{s(\alpha)}^{-1}$.
- In the scalar case, this is a rescaling at each neuron.

Neural teleportation refers to the action of these symmetries on the neural parameter space: for a choice of invertible $\tau=(\tau_i)$, set
- $V_{ab} = \tau_b W_{ab} \tau_a^{-1}$ for each arrow,
- $g_b(x) = \tau_b f_b(x / \tau_b)$ for each activation.

This transformation preserves the network function, i.e., for any input, the output remains invariant [2012.01118, 2007.12213]. For ReLU and other positive-scale-invariant activations, teleportation across positive $\tau$ leaves activations unchanged (intra-landscape symmetry).

Architectural constraints (e.g., preserving convolutional structure or residual identity maps) are enforced by restricting the allowable $\tau$, such as requiring constant scaling along feature maps or skip branches.

## 3. Moduli Spaces and Geometric Invariant Theory

The set of all quiver representations (with or without framings) modulo the symmetry group action forms the representation variety or moduli space $\mathcal{M}$. For neural networks, this moduli space classifies all network parameterizations (weights, activations) that are functionally equivalent:
$$
\mathcal{M}(Q, d) = \operatorname{Rep}(Q, d) // G
$$
where $//$ denotes the appropriate (GIT or symplectic) quotient. Double or framed moduli spaces formalize inputs and outputs, producing spaces whose points classify entire functional equivalence classes of neural networks [2109.14589, 2007.12213].

- The output of a neural network depends only on the corresponding moduli point: the network mapping factors through $\mathcal{M}$.
- The dimension of $\mathcal{M}$ governs functional capacity and redundancy; for thin ReLU nets, this is (number of arrows) minus (number of hidden neurons).
- Symplectic or Kähler structures on moduli spaces arise via canonical Hermitian metrics, moment maps, or Ricci curvature, equipping the learning landscape with an explicit geometric structure [2101.11487, 2109.14589].

## 4. Applications: Compression, Redundancy, and Optimization

Symmetry reduction in parameter space leads to:
- **Lossless model compression**: Reduction of hidden layer widths or node dimensions without altering functionality, via QR-based algorithms when activations are rescaling or radial [2207.12773]. The reduced dimension at each vertex is $d_i^\wedge = \min \{ d_i, \sum_{j \to i} d_j^\wedge \}$.
- **Minimal subnetworks**: The compressed subnetwork is minimal among all whose function coincides with the original.
- **Gradient Descent Equivalence**: Gradient descent on the compressed parameters is functionally equivalent (up to linear shifts and projections) to projected gradient steps on the original weights.
- **Pruning and feature bottlenecks**: Parameters unused by forward data flow are identified as "dead" and their moduli coordinates can be removed (pruned) without loss of expressivity [2007.12213].

## 5. Invariances, Training Dynamics, and Moduli-Theoretic Implications

Functional invariance under parameter symmetries (teleportation) induces significant consequences:
- **Level-set explorations**: Parameter transformations traverse the same loss level sets or, with sign changes, new "inter-landscape" functions [2012.01118].
- **Loss landscape geometry**: Teleportation (especially with large random $\tau$) is shown to sharpen local minima and increase normalized gradient norms, altering the loss landscape geometry in ways analogous to Dinh et al.'s findings [2012.01118].
- **Gradient rescaling**: Gradients transform covariantly under teleportation, and in large-scale experiments, small teleportations are nearly orthogonal to batch-gradient updates, indicating SGD dynamics are compatible with moduli symmetries.
- **Accelerated training**: Initialization by a single large teleport can empirically accelerate convergence in standard tasks (CIFAR-10/100) for various architectures [2012.01118].

Because the functional map factors through moduli, optimization and learning can theoretically be analyzed within the geometry of $\mathcal{M}$—potentially with better-behaved flows and convergence properties.

## 6. Algebraic-Geometric, Kähler, and Toric Structures

For framed quiver representations (arising naturally for bias-incorporated neural nets), one obtains:
- Hermitian metrics (on universal bundles), with explicit Ricci curvature; these induce Kähler forms on the moduli spaces, making learning dynamics analyzable in a Kähler manifold context [2101.11487].
- Toric moment maps facilitate construction of multi-variable activation functions (e.g., multi-softmax), connecting the activation geometry to symplectic and algebraic geometry.
- The universal approximation theorem extends to these settings: there exist network weights and multi-variable toric activations that approximate any continuous function on compact domains to arbitrary precision.

## 7. Architectural Variants and Open Directions

The quiver formalism is sufficiently general to encode:
- Standard and hybrid architectures (MLP, CNN, U-Net, ResNet).
- BatchNorm, layer-norm, pooling, and arbitrary wiring patterns via imposed combinatorial or weight-sharing constraints [2007.12213].
- Pointwise, rescaling, and non-pointwise activations—though some compression and symmetry results require radial or rescaling types [2207.12773].

Active research directions include:
- Extension of compression methods beyond rescaling activations.
- Moduli interpretations for convolutional/equivariant architectures over structured index sets.
- Explicit treatment of parameter symmetries arising from normalization layers.
- Study of non-convexities, global minima, and dynamics via Nakajima quiver varieties and Morse theory [2207.12773, 2101.11487].

---

By adopting the perspective of neural networks as quiver representations, one achieves a unified, exact algebraic framework to analyze architecture, parameter redundancies, optimization geometry, and moduli-theoretic aspects. This allows for principled compression methods, rigorous identifiability results, and explicit links to geometric and topological invariants in both classical and modern neural architectures [2207.12773, 2012.01118, 2101.11487, 2007.12213, 2109.14589].

Source: https://www.emergentmind.com/topics/neural-networks-as-quiver-representations