Random-Set Neural Networks

• RS-NNs are a class of neural networks that incorporate randomization and set-based outputs to enhance model uncertainty and optimization efficiency.
• They employ techniques like post-learning weight perturbations, randomized architecture sampling, and bias-only training to achieve universal approximation and reduce overfitting.
• Applications span uncertainty quantification, quantum many-body simulations, and biologically inspired learning, demonstrating scalability across diverse domains.

Random-Set Neural Networks (RS-NNs) denote a class of machine learning models that leverage the theory of random sets, randomization strategies, and set-based architectures in neural network design and training. The terminology encompasses several threads in contemporary research, including post-learning randomized weight perturbation for optimization, neural architectures outputting belief functions over sets of classes, networks with randomized or randomly sampled structure, and models exploiting the expressive potential of random weights with minimal training. RS-NNs have found application in uncertainty quantification, efficient function approximation, self-supervised modeling of physical systems, and biologically inspired learning paradigms.

1. Core Concepts and Motivations

RS-NNs generalize conventional neural networks by introducing stochasticity and set-based representations into their architecture, outputs, or training routines. In contrast to deterministic, fully optimized networks, RS-NNs often comprise random weights, random selection mechanisms, or operate on randomly sampled data subsets. The approach serves several purposes:

• Mitigating local minima and overfitting by escaping deterministic optimization traps through post-training randomization (Kapanova et al., 2015).
• Generating set-valued predictive outputs for robust uncertainty modeling (via belief functions and convex sets of probabilities), particularly in safety-critical domains (Manchingal et al., 2023).
• Enabling efficient learning by restricting training to a small subset of parameters (e.g., output weights or biases) while the majority of connections remain random (Williams et al., 1 Jul 2024, Gallicchio et al., 2020).
• Facilitating scalable and interpretable ensemble architectures where feature or model subsets are sampled randomly (Biau et al., 2016, Cao et al., 2019).
• Supporting problem domains where training data or system representations are too large for classical full-batch processing, as in quantum many-body simulations using random matrix sampling (Liu et al., 2020).
• Incorporating biological diversity and modularity by optimizing neuro-centric parameters rather than only synaptic weights (Pedersen et al., 2023).

2. Mathematical Foundations and Set-Valued Representations

A distinguishing feature of certain RS-NNs is the output of belief functions over sets of classes, constructed from the mathematics of random sets and Dempster-Shafer theory. Instead of outputting a probability vector $p \in \Delta^N$ for $N$ classes, RS-NNs predict a mass function $m : 2^\mathcal{C} \to [0,1]$ over the power set of classes $\mathcal{C}$ (Manchingal et al., 2023). This enables a principled quantification of epistemic uncertainty:

$$\text{Bel}(A) = \sum_{B \subseteq A} m(B) \qquad \forall\, A \subseteq \mathcal{C}$$

For decision making, the "pignistic" probability is derived from the belief function:

$$\text{BetP}(c) = \sum_{c \in A} \frac{m(A)}{|A|} \qquad \forall\, c \in \mathcal{C}$$

The set-valued nature of outputs allows modelling ignorance and hedging in ambiguous scenarios, with the associated convex credal set quantifying the size and nature of uncertainty.

Random sampling methods, as in quantum many-body modeling (Liu et al., 2020), entail constructing a reduced representation:

$H_S^{(m)}[\lambda] \in \mathbb{C}^{M\times M}$

where $M$ is the sample size and $\lambda$ indexes system parameters. The neural network function $\mathcal{F}_{\textrm{RSNN}}$ approximates physical observables by learning mappings from these sampled patches.

3. Randomization Strategies in Learning and Optimization

Several RS-NN paradigms exploit randomness to enhance optimization and generalization:

• Post-learning randomization (Kapanova et al., 2015): After classical training (gradient descent/simulated annealing), hidden layer weights are perturbed by small amounts of random noise. This mimics quantum tunneling phenomena, enabling escape from local minima with minimal computational overhead:

$$\text{weights}[j] \gets \text{weights}[j] + \text{randomDouble()} \cdot \text{weightNoise}$$

• Randomized neural architectures (Gallicchio et al., 2020): Networks where input-to-hidden and/or recurrent connections are fixed randomly, with only output (readout) or bias parameters optimized. Theoretical results guarantee universal approximation for sufficiently wide networks (Williams et al., 1 Jul 2024).
• Random subset and feature sampling (Cao et al., 2019): Neural modules operate on permuted or randomly selected feature subsets, often aggregated via group convolutions, preserving efficient coverage and nonlinearity.
• Randomized ensembles and subnetworks: In tree-based models recast as neural networks ("Neural Random Forests"), each tree’s architecture generates a sparse, trainable subnetwork, reducing parameter count and enabling flexible decision boundaries (Biau et al., 2016).

4. Universal Approximation and Theoretical Guarantees

Recent work rigorously establishes the expressive power of RS-NNs under various randomization regimes:

• Bias-only training (Williams et al., 1 Jul 2024): Feedforward and recurrent networks with fixed random weights and learned biases can universally approximate any continuous function or dynamical system over compact domains. The proof employs masking arguments—learned biases act as gates to select functional subnetworks—which provides a theoretical foundation for parameter-efficient fine-tuning.
• Mutual complexity and separation theory (Dirksen et al., 2021): A two-layer random ReLU network with Gaussian weights and biases can separate two $\Delta$-separated sets $X^-, X^+$ if the layer width exceeds an instance-specific function of their "mutual complexity," quantified via covering numbers and Gaussian mean width. This overcomes worst-case dimensionality barriers in "low-complexity" or manifold-structured data.
• Randomized kernels and Gaussian processes (Gallicchio et al., 2020): In the infinite-width limit, function spaces realized by random weights converge to Gaussian processes. The architecture defines the corresponding kernel, illustrating deep links between model structure and learnability.
• Spline regularization equivalence (Heiss et al., 2023): Wide randomized ReLU networks with output layer training behave as generalized additive models (GAMs) in function space; the learned solution corresponds to a mixture of spline-regularized univariate functions across directions in input space, yielding smooth generalization bias.

5. Practical Applications, Scalability, and Empirical Results

RS-NNs have broad empirical validation and practical relevance across domains:

Application	Core Mechanism	Empirical Result
Uncertainty quantification	Belief function output (random set)	Superior OoD detection, robust uncertainty (Manchingal et al., 2023)
Quantum many-body problems	Random sampling of Hamiltonians	High accuracy in energy spectrum and critical exponents, linear time scaling (Liu et al., 2020)
Classification/regression	Bias-only training in random networks	Comparable accuracy to full training, with >1000× parameter reduction (Williams et al., 1 Jul 2024)
Deep ensemble learning	Random subspace neural modules	Improved accuracy/efficiency over random forests, effective CNN integration (Cao et al., 2019)
Reinforcement learning	Evolved neuro-centric parameters	Solves continuous control with only neuron evolution (Pedersen et al., 2023)

Scalability of RS-NNs is addressed via limiting the size of focal sets (random-set output budget), modular architecture design, and efficient randomization strategies. Models have been deployed on large-scale architectures such as WideResNet, VGG, Inception, EfficientNet, and ViT, demonstrating competitive accuracy and robust performance under adversarial, noisy, and out-of-distribution regimes (Manchingal et al., 2023).

6. Connections to Biological Inspiration and Future Directions

RS-NNs reflect and motivate several lines of research in neuroscience and biologically inspired AI:

• Bias modulation and non-synaptic mechanisms underline the potential for dynamic adaptation without synaptic connectivity changes (Williams et al., 1 Jul 2024).
• Neuro-centric diversity, with each neuron parameterized independently, recapitulates cellular heterogeneity and enables efficient internal state dynamics (Pedersen et al., 2023).
• Co-evolving synaptic and neuro-centric parameters, activity-dependent plasticity, and modular hybridization (e.g. with weight-agnostic neural networks) are identified as active research themes seeking further gains in adaptability, compactness, and generalization (Pedersen et al., 2023).

Among open questions and suggested future work are:

• Minimizing required network width in bias-only universal approximation (Williams et al., 1 Jul 2024).
• Extending convergence guarantees for recurrent models beyond pointwise approximation.
• Optimizing sampling and aggregation strategies for physical system modeling (e.g. higher-dimensional RS-NNs for quantum mechanics) (Liu et al., 2020).
• Systematic comparison and integration of random-set, bias, and gain modulation approaches for efficient model adaptation.

7. Limitations and Controversies

RS-NN methods must be carefully calibrated; randomization can risk the loss of representative power if sampling strategies or random connection patterns are not sufficiently diverse or informative. The reliance on minimal parameter tuning may limit performance in domains requiring fine structural adaptation or where learning is nontrivial across tasks or distributions. Scalability concerns persist when the power set of classes is large in belief-function outputs, necessitating budgeting or rational selection of focal sets (Manchingal et al., 2023). The underlying theory divides between approaches emphasizing randomization for expressivity and those using set-based outputs for uncertainty quantification, requiring context-specific assessment for optimal application.

RS-NNs represent an evolving convergence of probabilistic reasoning, ensemble modeling, function approximation theory, and biologically motivated computation in neural network research. Their integration of randomness and set-theoretic structures enables new solutions for uncertainty quantification, scaling, and efficient learning, with continued theoretical and empirical developments broadening their applicability and impact.