Steepest Voting Distributor (SVoD) Explained

Updated 12 July 2026

SVoD is a gradient-based distributor that allocates new neurons across layers based on probe-derived negative gradients to locally reduce loss.
It measures the loss gradient with respect to temporary probe gates to cast votes, ensuring neurons are added where they can most effectively improve performance.
Integrated with the Shared-Weights Extender (SWE), SVoD enables smooth, capacity-driven expansion in multi-layer neural networks to overcome dead neuron issues.

Searching arXiv for the cited paper and related context. arXiv search query: (Chatzis et al., 23 Sep 2025) arXiv search results for ([2509.18842](/papers/2509.18842)):

"Shared-Weights Extender and Gradient Voting for Neural Network Expansion" (Chatzis et al., 23 Sep 2025)

1. Position within the expansion framework

Within the growing framework of "Shared-Weights Extender and Gradient Voting for Neural Network Expansion," the Distributor $\mathbf{D}$ is the component that decides how many new neurons to place in each layer at a given expansion stage, whereas the Extender $\mathbf{E}$ specifies how to initialize and integrate those neurons (Chatzis et al., 23 Sep 2025). SVoD is the proposed gradient-based Distributor, and SWE is the proposed Extender. This division separates where to grow from how to initialize and integrate new neurons.

The pipeline per expansion stage is specified as follows. First, the Distributor $\mathbf{D}$ determines $m_l$ , the number of new neurons to add to each layer. Second, for each layer $l$ , the Extender $\mathbf{E}$ initializes new neurons, introduces coupling parameters, and performs the short coupling-adjustment phase. Third, the expanded network is jointly trained for regular epochs. In this sequence, SVoD operates before SWE in each stage: SVoD decides “how many neurons per layer” based on gradient votes, and SWE manages “how to integrate these neurons” so they do not become inactive and smoothly join the representation (Chatzis et al., 23 Sep 2025).

This design is motivated by a specific failure mode of dynamic expansion. Newly added neurons often fail to adjust to a trained network and become inactive, providing no contribution to capacity growth. A naive Distributor, such as uniform or random allocation, may add neurons to layers that are not capacity bottlenecks. SVoD identifies layers with high gradient demand, while SWE is intended to ensure that the newly added neurons in those layers actually become active and useful, addressing the dead ReLU problem. In the multi-layer classification experiments, the SWE+SVoD configuration is the full method (Chatzis et al., 23 Sep 2025).

2. Probe-gated formulation and local descent criterion

For a probe neuron, the preactivation is defined by

$u = \mathbf{w}^\top \mathbf{x} + b,$

with weight vector $\mathbf{w}$ , bias $z=0$ 0, and activation function $z=0$ 1, which is ReLU in the main experiments. SVoD defines the probe output as

$z=0$ 2

At $z=0$ 3, the probe behaves as a normal neuron with preactivation $z=0$ 4, but the network is considered as a function of the scalar gate $z=0$ 5, and the loss is analyzed around $z=0$ 6 (Chatzis et al., 23 Sep 2025).

Let $z=0$ 7 denote the loss as a function of the gate. Around $z=0$ 8, the method uses the first-order Taylor expansion

$z=0$ 9

$\mathbf{D}$ 0

then increasing $\mathbf{D}$ 1 is predicted to decrease the loss for small $\mathbf{D}$ 2. In the paper’s terminology, such a probe is “steepest-descent beneficial”: increasing the effective activation of the probe along its current direction $\mathbf{D}$ 3 is predicted to reduce the loss locally (Chatzis et al., 23 Sep 2025).

The gradient $\mathbf{D}$ 4 is computed by automatic differentiation during a single forward–backward pass, with $\mathbf{D}$ 5 treated as a learnable scalar. SVoD does not optimize the gate themselves; it uses the signed gradients as a diagnostic. A negative gate gradient is evidence that

$\mathbf{D}$ 6

This suggests that SVoD is best understood as a local sensitivity test over candidate insertion locations rather than as a gate-learning procedure (Chatzis et al., 23 Sep 2025).

3. Voting mechanism and layer-wise allocation

The SVoD algorithm proceeds layer-wise with temporary probe neurons. At an expansion stage, a set of probes is inserted into each hidden layer $\mathbf{D}$ 7. Each probe corresponds to a candidate direction $\mathbf{D}$ 8. The paper does not fully detail the sampling scheme for these directions, but the probes are used only to measure gradient sensitivity (Chatzis et al., 23 Sep 2025).

After probe insertion, one forward pass computes the current loss and one backward pass computes

$\mathbf{D}$ 9

for each probe $\mathbf{E}$ 0 in layer $\mathbf{E}$ 1. SVoD then applies gradient sign filtering: if

$\mathbf{E}$ 2

the probe is considered useful and casts one vote for its layer; probes with non-negative gradient are discarded for voting purposes. The layer-wise vote count is therefore

$\mathbf{E}$ 3

If the expansion stage is configured to add $\mathbf{E}$ 4 new neurons in total across all layers, SVoD converts votes into an allocation by normalizing

$\mathbf{E}$ 5

and then setting

$\mathbf{E}$ 6

with minor adjustments to ensure $\mathbf{E}$ 7. Layers with more negative-gradient probes therefore receive more new neurons (Chatzis et al., 23 Sep 2025).

Conceptually, this is a steepest-descent-inspired layer selection rule. The method uses the sign of gate gradients to determine where expansion seems most likely to reduce the loss, but it aggregates probe information only at the layer level. Unlike Firefly, which selects specific candidate neurons to insert directly based on gradient magnitudes, SVoD uses the probe information only to vote at the layer level and leaves actual neuron insertion and initialization to SWE. A common misunderstanding is therefore to treat SVoD as a candidate-neuron selector; in the formulation given in the paper, it is strictly a layer allocator (Chatzis et al., 23 Sep 2025).

4. Architectural assumptions, training regime, and variants

The reported setting is restricted to fully connected feed-forward networks with ReLU activations in the hidden layers. For CIFAR experiments, SVoD is applied only to the MLP head on top of a fixed CNN backbone. Image classification uses standard cross-entropy loss, reconstruction uses MSE, and optimization is performed with Adam at learning rate $\mathbf{E}$ 8. No special optimizer modifications are introduced for SVoD; gradients for gate variables $\mathbf{E}$ 9 are obtained via usual backprop (Chatzis et al., 23 Sep 2025).

Training alternates between expansion stages and regular training epochs. Expansion is staged; one example given in the single hidden-layer reconstruction experiments is 7 stages, each adding 30% more neurons. SVoD is invoked only at these expansion stages, not every epoch. The number of probes per layer is not explicitly specified, but the method relies on having multiple probes per layer to obtain a meaningful vote count $\mathbf{D}$ 0. The paper notes that, by analogy to Firefly, SVoD likely uses a modest number of probes per layer to balance diagnostic quality and computation, but the exact probe distribution scheme is not detailed (Chatzis et al., 23 Sep 2025).

For the Distributor role, the paper compares SVoD with RAS (Random Allocation Strategy). In this baseline, new neurons are distributed uniformly across layers or randomly, without gradient information. The experimental notation SWE+RAS refers to using the SWE extender with random layer allocation. In contrast, SWE+SVoD combines the proposed extender with the gradient-based distributor. In single hidden-layer settings, SVoD is irrelevant because there is no inter-layer distribution choice; expansion is per-layer only, and SVoD is not evaluated there (Chatzis et al., 23 Sep 2025).

5. Experimental usage and empirical behavior

SVoD is used specifically in multi-layer MLP heads for classification on MNIST, FashionMNIST (FMNIST), CIFAR-10, and CIFAR-100. The relevant architecture is a three hidden-layer MLP. Each hidden layer starts with 10 neurons for MNIST, FMNIST, and CIFAR-10, while CIFAR-100 uses $\mathbf{D}$ 1 width. The MLP head sits on top of a fixed VGG-style CNN backbone for the CIFAR experiments, with channels $\mathbf{D}$ 2 for CIFAR-10 and $\mathbf{D}$ 3 for CIFAR-100; expansion applies only to the MLP head. SVoD is invoked at each expansion stage to allocate new neurons across the three hidden layers (Chatzis et al., 23 Sep 2025).

The three hidden-layer classification experiments compare a fixed-size MLP baseline trained from scratch, Firefly, SWE+RAS, and SWE+SVoD. The reported final test accuracy values (mean $\mathbf{D}$ 4 std, %) are:

MNIST: Baseline $\mathbf{D}$ 5, Firefly $\mathbf{D}$ 6, SWE+RAS $\mathbf{D}$ 7, SWE+SVoD $\mathbf{D}$ 8
FMNIST: Baseline $\mathbf{D}$ 9, Firefly $m_l$ 0, SWE+RAS $m_l$ 1, SWE+SVoD $m_l$ 2
CIFAR-10: Baseline $m_l$ 3, Firefly $m_l$ 4, SWE+RAS $m_l$ 5, SWE+SVoD $m_l$ 6
CIFAR-100: Baseline $m_l$ 7, Firefly $m_l$ 8, SWE+RAS $m_l$ 9, SWE+SVoD $l$ 0 (Chatzis et al., 23 Sep 2025)

These results demonstrate the specific benefit claimed for SVoD over random allocation in deep MLPs. For MNIST and CIFAR-10/CIFAR-100, SWE+SVoD obtains the best or near-best accuracy, outperforming the fixed-size baseline, Firefly expansion, and SWE+RAS. For FMNIST, the baseline is slightly higher, but SWE+SVoD still beats Firefly and SWE+RAS. Figures for the three hidden-layer experiments, including the CIFAR-100 plot, show the stage-wise evolution of accuracy; SWE+SVoD tends to achieve higher accuracy early in expansion and maintains strong performance throughout. This suggests that SVoD’s allocations lead to more effective use of added capacity at each stage (Chatzis et al., 23 Sep 2025).

6. Relation to prior expansion methods, limitations, and future directions

The conceptual justification for SVoD is explicitly connected to earlier gradient-based expansion methods. The paper situates it relative to Splitting Steepest Descent, Firefly, GradMax, and SENN, which examine gradients with respect to candidate new parameters or splitting directions to identify expansions that locally reduce loss (Liu et al., 2019, Piecuch et al., 2023, Diaconis et al., 2020, Kucharski et al., 2022). SVoD adopts this principle but repurposes it from selecting individual candidates to layer-wise voting. Its novelty is therefore a voting-based layer allocator that uses gradient information from all probes only to count votes per layer, together with a strict separation between the Distributor and the Extender (Chatzis et al., 23 Sep 2025).

This separation clarifies several points that are sometimes conflated. SVoD is not a function-preserving split rule, not a particular initialization strategy, and not a mechanism for directly training probe gates. It is a gradient-driven allocation mechanism whose output is the layer-wise vector $l$ 1. SWE supplies the shared weights and coupling mechanism that integrate the actual inserted neurons. A plausible implication is that the empirical behavior of the full method depends on both placement and activation: SVoD addresses the first, SWE the second (Chatzis et al., 23 Sep 2025).

The paper does not enumerate explicit limitations of SVoD separately, but several emerge from the reported design. SVoD requires inserting probe neurons and performing at least one forward–backward pass to compute gate gradients, although this is done only at expansion stages, so the overhead is described as moderate. The method depends on local first-order gradient information, which may not capture long-term benefits of expansion or interactions between layers beyond what current gradient magnitudes indicate. Probe design is another open issue, because the paper does not detail how probe directions are chosen. Finally, the experimental study is restricted to feedforward ReLU MLPs with simple CNN backbones, and the behavior of SVoD for more complex architectures, including transformers and large residual nets, is not studied (Chatzis et al., 23 Sep 2025).

The future directions suggested in the paper are correspondingly broad. They include extending the framework, including SVoD, to continual learning; developing more advanced probing strategies using richer candidate sets or structured directions based on Hessian or Fisher information; and integrating SVoD into neural architecture search or dynamic routing schemes, where gate gradients could guide both width and depth changes. In that sense, SVoD is presented not as a complete theory of growth, but as a simple, gradient-based mechanism for turning temporary probe responses into a global, layer-wise distribution of new neurons (Chatzis et al., 23 Sep 2025).