Random Projection Neural Networks

• Random Projection Neural Networks are architectures that replace standard weight matrices with fixed, randomly initialized projection matrices to enable efficient dimension reduction.
• They leverage theoretical guarantees like the Johnson–Lindenstrauss lemma to nearly preserve pairwise distances, enhancing function approximation and representation learning.
• RPNNs offer significant model compression and computational gains, enabling resource-constrained inference and effective deployment in scientific computing and classification tasks.

Random Projection Neural Networks (RPNNs) constitute a framework in which neural architectures incorporate fixed, randomly initialized projections—typically at the level of input, intermediate, or hidden layers—to achieve efficient, compact, and distance-preserving representations. These networks leverage the theory of random feature maps, the Johnson–Lindenstrauss lemma, and random matrix design to achieve reduced parameterization, computational gains, and occasionally enhanced expressive power, with direct applications in compression, resource-limited deployment, function approximation, scientific computing, and high-dimensional learning.

1. Architectural Principles and Random Projection Layer Design

The prototypical RPNN replaces one or more weight matrices in a standard neural network with randomly sampled and fixed matrices, decoupling feature mixing from trainable adaptation. The random projection (RP) layer accepts an input $x\in\mathbb{R}^D$ and generates a lower-dimensional representation:

• Dense Gaussian RP: $R_{ij} \sim \mathcal{N}(0,1/k)$, with $k \ll D$.
• Sparse and Fast RP alternatives: Achlioptas, Li, Count-Sketch, and SRHT (Subsampled Randomized Hadamard Transform) matrices, which are computationally efficient and preserve pairwise distances with high probability (Wójcik, 2018).
• Elementwise modulated random matrices: The PRP layer modulates a fixed $R\in\mathbb{R}^{d_\text{in}\times d_\text{out}}$ with learned scaling vectors $\alpha, w$ and bias $b$:

$y = (R^\top (\alpha \odot x)) \odot w + b$

reducing trainable parameters to $d_\text{in} + 2d_\text{out}$ (Szorc, 15 Dec 2025).

The forward computation through the RP layer is typically:

$h = R^\top x + b$

optionally followed by batch normalization, activation functions (linear or ReLU), and further layers.

Crucially, in all these constructs, $R$ is fixed at initialization and never updated by backpropagation, enabling both computational savings and theoretical analysis grounded in random matrix theory.

2. Theoretical Guarantees: Johnson–Lindenstrauss and Expressive Power

RPNNs derive much of their justification from the Johnson–Lindenstrauss lemma, which establishes the existence of almost distance-preserving maps from high- to low-dimensional spaces via random projections:

• Pairwise distance preservation:

$$(1-\varepsilon)\|x-y\|^2 \leq \|R(x-y)\|^2 \leq (1+\varepsilon)\|x-y\|^2$$

with $k = O(\varepsilon^{-2} \log n)$ for $n$ datapoints and high-probability bounds (Wójcik, 2018, Fabiani, 2024).

• Expressive power on structured data:

For inputs from $s$-sparse subsets or $k$-dimensional manifolds, RP reduces expressive requirements from ambient dimension to intrinsic structure, yielding networks where the number of trainable parameters and required hidden units scale with $O(s \log(d/s))$ or $O(k \log d)$, not $d$ (Cai et al., 2018).

• Function approximation:

For any infinitely smooth target $f$, a one-hidden-layer RPNN with $N$ random basis functions achieves exponential convergence in $L^p$:

$\|f - f_N\|_p = O(e^{-\alpha N})$

for appropriate non-polynomial, $C^\infty$ activations and random internal parameters, provided the output layer is optimally fit (Fabiani, 2024).

These guarantees underpin the parameter efficiency and stability of RPNN-based models in both high-dimensional learning and scientific computation contexts.

3. Compression, Resource-Constrained Inference, and Training Dynamics

RPNNs provide model compression by drastically reducing the number of trainable parameters:

• PRP Layer Compression:

The parametrized random projection (PRP) reduces trainable weights in a fully connected layer from $O(d_\text{in} d_\text{out})$ to $O(d_\text{in}+d_\text{out})$, while empirical performance on classification (MNIST, CIFAR-10, TinyImageNet) typically retains over $90\%$ accuracy, with some large-scale cases even showing improvements over dense layers (Szorc, 15 Dec 2025).

• Fixed vs. learnable RP layers:

Some RPNN variants fix $R$ entirely (zero backprop); others restrict learning to the nonzero entries of a sparse $R$ for flexible adaptation without full parameterization (Wójcik, 2018).

• ProjectionNet:

A distinct bit-sketching approach uses multiple random projections to binarize input or hidden vectors. These are processed in a distillation framework, jointly training a large "teacher" and a compact "projection network" via a combined loss:

$$L(\theta,p) = \lambda_1 L_\theta + \lambda_2 L^p + \lambda_3 \hat{L}^p$$

with straight-through estimator (STE) for back-propagation through the sign() quantization operation. This configuration enables highly compressed models for on-device inference (Ravi, 2017).

4. RPNNs in Representation Learning, Similarity Search, and Language Modelling

RPNNs have been employed to address representation efficiency, similarity-based learning, and out-of-vocabulary handling:

• Orthogonality-regularized NNs (Savushkin et al., 2020):

Enforcing orthogonal hidden layer matrices during training, combined with modified BatchNorm and dropout, yields embeddings with near-zero neuron correlation and distance-preservation properties, as validated by improved downstream $k$-NN accuracy on MNIST and medical datasets. The required width for the final hidden layer is directly connected to Johnson–Lindenstrauss bounds, $k \gtrsim C \log n / \varepsilon^2$ (Savushkin et al., 2020).

• Language modelling via sparse random indices:

Encoding word types with sparse, nearly orthogonal random codes, then projecting to a learnable embedding, allows for compact handling of large (or evolving) vocabularies and is empirically competitive on Penn Treebank, reducing parameter count by 20–50% at the cost of $10\%$ higher perplexity vs. standard embeddings (Nunes et al., 2018).

5. Scientific and Numerical Computing: Function Approximation and PDE/ODE Solvers

RPNNs serve as surrogate models and collocation-based solvers in scientific computing:

• ODE solvers and Physics-Informed RPNNs:

For stiff ODEs, a single-hidden-layer RPNN with RBF or analytic activation and random internal parameters serves as a meshfree, collocation-based surrogate. Output weights are fit via (regularized) least squares or Gauss–Newton; adaptivity in interval width addresses local solution stiffness (Galaris et al., 2021). Compared to traditional solvers, such as ode45/ode15s, RPNN-based approaches are competitive in accuracy and computational cost, especially when analytic surrogate evaluation across dense grids is required.

• PDE stability and bifurcation analysis:

RPNNs parameterize the solution space for physics-informed neural solutions to PDEs. The collocation matrix built from random features is typically exponentially ill-conditioned, yet the network admits a mathematically regular generalized eigenvalue problem—for stability and bifurcation analysis—solved efficiently via a matrix-free Krylov–Arnoldi shift-invert algorithm in weight space (Fabiani et al., 23 Mar 2026).

• Parallel-in-time methods:

In the Parareal ODE solution framework, a two-layer RPNN acts as an efficient, accurate coarse propagator, fitted by least squares per sub-step, requiring drastically less compute than offline-trained global neural surrogates for each interval (Betcke et al., 2024).

6. Training Regimes and Practical Variants

A broad spectrum of training methodologies supports RPNN deployment:

• Fixed random projection:

Pre-compute $X_\text{proj} = XR$, normalize, and train the remaining layers with standard optimization, often with improved stability and interpretability—no gradients flow through $R$.

• Learnable-sparse projection:

Only nonzero entries of a sparse $R$ are adapted, maintaining computational efficiency while allowing data-driven flexibility.

• Bit-sketching and quantization:

Use of sign-hash transforms with straight-through estimators for efficient back-propagation (Ravi, 2017).

• Parameter selection and conditioning:

Construction of internal RP parameters may be naive uniform, function-informed, or agnostic. Robust training mandates close monitoring of the collocation matrix's singular value decay and may require function-informed sampling for exponential convergence in numerical approximation tasks (Fabiani, 2024).

• Batch normalization and regularization:

To control the amplification of correlations, modified batch-normalization (shared scale, zero mean, no bias) and dropout are integrated with RP layers (Savushkin et al., 2020).

7. Empirical Benchmarks and Quantitative Outcomes

Selected key results across domains:

Task	RPNN Variant	Params (RPNN/FC)	Accuracy/Metric	Reference
MNIST classification	PRP layer, MLP	3,108 / 535,818	91.66% / 97.79%	(Szorc, 15 Dec 2025)
CIFAR-10 (VGG classifier head)	PRP layer + feature extractor	4,372 / 1,182,986	87.4% / 86.2%	(Szorc, 15 Dec 2025)
Stiff ODEs (ROBER, van der Pol, HIRES)	Physics-Informed RPNN	20 hidden (adaptive)	Error $\sim$ 1e-3	(Galaris et al., 2021)
Neural episodic control	RP layer in Q memory	0 (RP) / 32d	4/5 games better RP	(Nishio et al., 2019)
Function Approximation	One-hidden-layer RP, best fit	N varied	Exponential $\|f-f_N\|$ decay	(Fabiani, 2024)
Language modeling (PTB)	Sparse RP embedding	5M / 6.4M	PPL 129 / 118	(Nunes et al., 2018)

Empirical findings indicate that with appropriate design, RP-induced parameter reductions of $10-100\times$ can be achieved at modest cost in accuracy; in some large-scale classification regimes, performance even improves.

---

Random Projection Neural Networks, by grounding high-dimensional function approximation and representation in random-matrix theory, enable robust dimension reduction, resource-efficient deployment, and principled parameter reduction. Their applicability spans high-dimensional learning, scientific computation, and compact on-device inference, with their practical efficacy largely determined by architectural choices, regularization, and parameterization strategies (Szorc, 15 Dec 2025, Ravi, 2017, Fabiani, 2024, Galaris et al., 2021, Wójcik, 2018, Savushkin et al., 2020).