Weight Agnostic Neural Networks (WANNs)

• WANNs are neural architectures where task performance is governed primarily by topology rather than precision-tuned weights.
• Evolutionary and gradient-based methods optimize these architectures, showing competitive MNIST accuracies and impressive sparsity even with fixed or binarized weights.
• The design emphasis on shared or binary weights enhances interpretability and efficiency, opening avenues for transfer learning and hardware-friendly neural deployments.

Weight Agnostic Neural Networks (WANNs) are neural network architectures specifically engineered so that task performance is predominantly determined by the network topology, rather than by explicit learning or tuning of the connection-specific weights. In WANNs, all edges typically share a small set of constant scalar weights, or are even binarized to {0,1}, enabling the analysis and selection of architectures that exhibit strong inductive biases independently of their precise parameterization. This paradigm fundamentally contrasts with standard neural models, where both topology and high-dimensional weights are co-optimized.

1. Formal Definition and Theoretical Foundation

The canonical form of a WANN is a directed acyclic graph (DAG) of nodes, each of which is associated with a chosen activation function, and edges that do not carry individual weights but instead share a single global parameter. For all edges $(i,j)$ in the architecture $\theta$, the shared weight $w$ leads to $w_{ij} = w$, producing the forward computation

$$x_j = \phi_j\left(\sum_{i \in \mathrm{pred}(j)} w x_i \right)$$

where $\phi_j$ denotes the nonlinear activation at node $j$ (Gaier et al., 2019). In "weight-free" or "weight-invariant" variants, each edge is constrained to be either present (1) or absent (0), i.e. $w_{ij} \in \{0,1\}$ and connection patterns themselves are optimized (Agrawal et al., 2019, Colombo et al., 2020).

The task objective becomes identifying an architecture $\theta$ (and, optionally, a shared weight $w$) that maximizes expected task performance, averaged over a random or fixed set of constant weights: $$\mathbb{E}_{w \sim U(a,b)}[\mathrm{Performance}(\theta, w)]$$ In this context, gain in performance is attributed to the topology itself, rather than fine-grained weight adjustment.

2. Core Search and Training Methodologies

Evolutionary Architecture Search

Gaier & Ha employ neuroevolution, specifically an adaptation of the NEAT framework, to search over architecture space. Each individual in the population represents a different graph topology (choice of edge set and synaptic nonlinearities). Mutation operators include node insertion, connection addition, and activation reassignment. Fitness evaluation relies on averaging task performance across a discrete set of $k$ shared weight values, forming a multi-objective selection pressure: maximize mean expected performance, maximize best performance for a single weight, and minimize total connection count (Gaier et al., 2019).

Gradient-Based Binary Search

Agrawal & Karlupia, and Colombo & Gao, introduce frameworks where weights are binarized on the fly. The real-valued "pre-weights" are projected to $\{0,1\}$ via a step function or sharp sigmoid: $$w_b = \begin{cases} 1 & \text{if } w \geq 0.5 \ 0 & \text{otherwise} \end{cases}$$ During training, stochastic gradient descent is performed on the surrogate (with binary weights), and the pre-weights are clipped to [0,1] after each update. The resulting architecture is the set of edges with $w_b=1$, producing an extremely sparse, weight-invariant subnetwork (Agrawal et al., 2019, Colombo et al., 2020).

Colombo & Gao additionally formalize the search as an approximate-gradient binary optimization in combinatorial topology space, proven to converge (under convexity and bounded gradient) to local minima with an explicit error bound due to the hard-sigmoid relaxation (Colombo et al., 2020).

3. Experimental Evaluation and Empirical Results

Across WANN approaches, performance is evaluated on both reinforcement learning and supervised classification tasks. Key findings include:

• For RL tasks (e.g., CartPoleSwingUp, BipedalWalker-v2), WANNs with a single shared tunable weight outperformed fixed-topology baselines by large margins when weights are untuned, and retained substantial performance even compared to fully tunable weight networks (Gaier et al., 2019).
• On MNIST classification, Gaier & Ha report tuned-shared-weight WANNs reaching 91–92% accuracy, significantly above chance, and forming ensembles of untuned networks that achieve ≈90% accuracy without any weight-specific learning (Gaier et al., 2019).
• Agrawal & Karlupia demonstrate that binarized, self-pruning networks with batch normalization, trained via SGD with binarization, achieve MNIST test accuracies of 96.7% and prune >99% of weights by thresholding, matching or surpassing WANNs (Agrawal et al., 2019).
• Colombo & Gao show, on digit classification and document categorization, that directly optimized binary networks ($\{0,1\}$-weights) match or outpace real-valued networks and outperform both random and purely pruned topologies. Binary networks found by gradient-based search yield AUCs up to 0.982 on subset MNIST tasks, exceeding lottery ticket and agnostic search baselines (Colombo et al., 2020).

The following table compiles representative supervised learning performance metrics:

Method	MNIST Accuracy (Test)	Pruned Weights (%)
Real-Valued (dense)	98.1%	0%
WANN (shared weight)	91.9%	—
Self-Pruning + BN	96.7%	>99%
Supermask (Zhou et al.)	86.3%	—
Random	80.2% (AUC)	—
Binary (Colombo & Gao)	98.2% (AUC)	Sparse, varies

4. Architectural Insights and Transferability

WANNs reveal that architecture encodes a substantial inductive bias, sometimes sufficient for strong task performance with no or minimal tuning of scalar weights. In binary-weighted or self-pruned regimes, the resulting topology often implements digital logic: with $\{0,1\}$ binarized weights and thresholds, each neuron computes logical OR over active inputs, and when coupled with batch normalization or learned negation gates (e.g., $x \mapsto x \cdot (1-\alpha) + (1-x)\cdot \alpha$ with $\alpha \to 0,1$), NOR logic is achieved, enabling the network to compute arbitrary Boolean functions. This digital circuit analogy is particularly evident in networks where internal activations saturate to $\{0,1\}$ (Agrawal et al., 2019).

Colombo & Gao introduce spectral analysis of network topology. Networks found by gradient-based binary search exhibit Laplacian spectral signatures that differ markedly from those produced by magnitude pruning or random thresholding, reflecting structurally distinct inductive biases. This suggests potential avenues for structural transfer learning via spectral alignment of topologies (Colombo et al., 2020).

5. Comparative Analysis and Computational Implications

The computational cost of WANN search varies sharply with the approach. Evolutionary NEAT-style neuroevolution (Gaier & Ha) requires thousands of fitness evaluations across sampled weights, making it resource-intensive. In contrast, SGD-based self-pruning and binary optimization frameworks fit within standard training pipelines, achieving complete architecture pruning and binarization over a single run and at orders-of-magnitude lower computational cost (Agrawal et al., 2019, Colombo et al., 2020).

A comparison of methodology dimensions:

Approach	Weight Regime	Search Method	Computational Cost
Gaier & Ha (WANN)	Shared scalar weight	Evolutionary	High (thousands evals)
Agrawal & Karlupia	Binary $\{0,1\}$	SGD/binarization	Low
Colombo & Gao	Binary $\{0,1\}$	Approx-gradient	Low

6. Implications, Interpretability, and Open Directions

WANNs recast the network design problem from weight optimization to the search for architectures that are either robust or entirely impervious to weight specification. This foregrounds topology as the substrate of intelligence, suggesting that substantial task performance can be attained through architectural bias alone, especially in discrete-input, logic-like regimes. Interpretability is enhanced as many produced networks have digital-circuit analogs and can, in principle, be compiled to NOR/OR gate cascades (Agrawal et al., 2019).

A plausible implication is that future neural network research can focus on combinatorial topology search and architectural transfer learning, minimizing or eliminating the reliance on per-edge parameterization. Spectral analysis provides a framework for comparing, transferring, or composing such topologies on new tasks (Colombo et al., 2020). Additionally, the structural simplicity and sparsity induced by WANN methodologies are favorable for hardware deployment and neuromorphic computing.

Existing research also connects this line of inquiry to the minimal description-length principle, neural implicit biases, and, by analogy, to evolution of "innate" priors in biological neural circuits (Gaier et al., 2019).

7. Summary and Prospective Developments

Weight Agnostic Neural Networks challenge the assumption that high-dimensional parameterization is essential for strong generalization. By disentangling the structural and parametric contributions to task performance, WANNs enable the discovery of robust, interpretable, and compact architectures that perform competitively even with random or fixed weight configurations. Emerging directions include differentiable architecture search tailored for the WANN objective, logic-gate-aware neural design, and further theoretical analysis of the expressivity and inductive bias of sparse, weight-invariant networks (Gaier et al., 2019, Agrawal et al., 2019, Colombo et al., 2020).