Random Nonlinear Projections

• Random nonlinear projections are explicit mappings that reveal nonlinear structure from data by approximating kernel functions.
• They combine randomized linear projections with nonlinear transformations, enabling efficient binary, quantized, and layered feature representations.
• These methods reduce computational and memory demands, transforming high-dimensional nonlinear learning into tractable linear problems.

Random nonlinear projections are randomized mappings that make nonlinear structure accessible through explicit finite-dimensional representations. In the kernel setting, they replace an implicit feature map $\Phi$ by an explicit embedding $\phi(x)\in\mathbb{R}^D$ such that $k(x,y)\approx \phi(x)^\top \phi(y)$; in hashing and binary embedding they apply nonlinearities such as $\operatorname{sign}$ or quantization after projection so that Hamming or coded similarities estimate angular or other nonlinear similarities; and in layered architectures they alternate randomized linear operators with explicit nonlinear transforms to produce compact discriminative features (Chung et al., 2016, Ghojogh et al., 2021, Choromanska et al., 2015).

1. Conceptual scope

Kernel methods learn linear predictors in an implicit feature space through kernels of the form $k(x,y)=\langle \Phi(x),\Phi(y)\rangle$. Random feature methods replace the implicit $\Phi$ by an explicit randomized map $\phi(x)$, reducing dependence on $n\times n$ Gram matrices and enabling linear learning in the embedded space. In this sense, a random nonlinear projection is a randomized embedding whose nonlinear stage is explicit rather than hidden inside a kernel evaluation (Chung et al., 2016).

The term is used heterogeneously across the literature. In the tutorial-and-survey account, linear random projections are separated from nonlinear random projections: the former are justified by the Johnson–Lindenstrauss lemma and preserve Euclidean geometry, whereas the latter use a random linear map followed by a nonlinear transformation such as cosine, sigmoid, ReLU, or sign (Ghojogh et al., 2021). In contrast, the neural-network study on random projections states that its projection operators are linear and that nonlinearity enters through the activation function and the nonlinear structure of sparse or manifold data rather than through a nonlinear random feature map (Cai et al., 2018). A further extension appears in the theory of random oblique projections induced by sampling, where the nonlinearity arises from pseudoinversion, so that the projection operator is nonlinear in the sketch even when the sketch itself is linear (Niu et al., 24 May 2026).

A common misconception is therefore that “random nonlinear projection” always refers to a single formalism. The literature instead covers several related constructions: explicit kernel feature maps, binary and quantized embeddings, layered random networks, random-projection modules inside trainable architectures, and nonlinear oblique projections generated by sketch-and-solve procedures.

2. Kernel approximation and explicit random feature maps

The canonical construction is Random Fourier Features (RFF). For a continuous, positive definite, shift-invariant kernel $k(x-y)$ with spectral density $p(w)$, one samples $\phi(x)\in\mathbb{R}^D$0 and $\phi(x)\in\mathbb{R}^D$1, and sets

$\phi(x)\in\mathbb{R}^D$2

so that $\phi(x)\in\mathbb{R}^D$3. The survey also gives the cosine–sine variant and reports pointwise concentration

$\phi(x)\in\mathbb{R}^D$4

together with a uniform approximation bound over compact sets (Ghojogh et al., 2021).

Random Kitchen Sinks (RKS) generalize this template by taking a random linear projection followed by a bounded nonlinear activation and training only the linear readout. In the notation of the survey, $\phi(x)\in\mathbb{R}^D$5, and the learned predictor is a regularized linear combination of the random basis functions. The same survey gives an empirical/true risk decomposition whose dominant terms scale as $\phi(x)\in\mathbb{R}^D$6, separating estimation and approximation effects (Ghojogh et al., 2021).

Several other explicit random-feature constructions target specific analytic kernel classes. Random Maclaurin and Tensor Sketch approximate polynomial kernels; Fastfood replaces dense Gaussian matrices with structured Hadamard- and diagonal-based transforms and achieves $\phi(x)\in\mathbb{R}^D$7 time with reduced memory; Nyström methods are closely related low-rank kernel approximations, although they were not evaluated in the LaRP paper (Chung et al., 2016).

Family	Core randomized map	Primary role
RFF	Random frequencies plus sinusoidal features	Shift-invariant kernel approximation
RKS	Random projection plus nonlinear activation, linear readout	General random nonlinear basis expansion
Random Maclaurin / Tensor Sketch	Randomized polynomial expansions or CountSketch hashing	Polynomial-kernel approximation
Fastfood	Structured Hadamard-diagonal transform	Fast RFF-like features with reduced memory
Hyperparameter-randomized RFF	Randomly sample kernel hyperparameters and average	Randomized surrogate to large kernel grid searches

The dimension-reduction survey and the matrix/kernel study both emphasize that these embeddings turn nonlinear learning into linear learning on $\phi(x)\in\mathbb{R}^D$8, with feature construction typically scaling as $\phi(x)\in\mathbb{R}^D$9 and storage as $k(x,y)\approx \phi(x)^\top \phi(y)$0, rather than requiring explicit $k(x,y)\approx \phi(x)^\top \phi(y)$1 kernel matrices (Ghojogh et al., 2021, Advani et al., 2020). The latter paper also proposes sampling kernel hyperparameters $k(x,y)\approx \phi(x)^\top \phi(y)$2 from a prior $k(x,y)\approx \phi(x)^\top \phi(y)$3 on a Borel set and averaging the resulting RFF maps, which suggests a randomized alternative to exhaustive kernel-scale search (Advani et al., 2020).

3. Binary, stable, and quantized random projections

A second major line replaces smooth nonlinearities by discrete ones. In binary embeddings, one starts from

$k(x,y)\approx \phi(x)^\top \phi(y)$4

and compares points through normalized Hamming distance. For fully random hyperplanes, the classical identity gives $k(x,y)\approx \phi(x)^\top \phi(y)$5. The structured-hashing analysis extends this unbiasedness to short and extended $k(x,y)\approx \phi(x)^\top \phi(y)$6-regular hashing, where the pseudo-random projection matrix shares a fixed budget of Gaussian randomness across rows. It further derives concentration bounds whose constants depend on the overlap parameter $k(x,y)\approx \phi(x)^\top \phi(y)$7 and the chromatic number $k(x,y)\approx \phi(x)^\top \phi(y)$8; Toeplitz and circulant constructions satisfy $k(x,y)\approx \phi(x)^\top \phi(y)$9, use $\operatorname{sign}$0 randomness and memory, and admit near-linear multiplication through FFT-friendly structure (Choromanska et al., 2015).

Stable-sign projections interpolate among several nonlinear similarities. With $\operatorname{sign}$1 and $\operatorname{sign}$2, the induced kernel depends on $\operatorname{sign}$3: for $\operatorname{sign}$4, the expected sign inner product is an angular arc-cosine kernel; for $\operatorname{sign}$5, under nonnegative unit-sum histogram conditions, the collision probability is approximated by an arc-cos-$\operatorname{sign}$6 kernel; and as $\operatorname{sign}$7, the method converges to the resemblance kernel, equivalently binary Jaccard in the $\operatorname{sign}$8 coding view. The same study reports that $\operatorname{sign}$9 behaves as a tuning parameter across dense, histogram, and sparse regimes (Li, 2015).

Quantization adds another nonlinear stage. The 2-bit random projection framework partitions each Gaussian projection into four bins using thresholds $k(x,y)=\langle \Phi(x),\Phi(y)\rangle$0, forms joint counts over the resulting $k(x,y)=\langle \Phi(x),\Phi(y)\rangle$1 regions, and estimates cosine similarity by maximum likelihood from the multinomial log-likelihood. The paper derives closed forms for the region probabilities and their derivatives, shows that the 2-bit MLE improves substantially over 1-bit and linear 2-bit estimators, and quantifies the low-similarity variance reduction ratio $k(x,y)=\langle \Phi(x),\Phi(y)\rangle$2, which attains approximately $k(x,y)=\langle \Phi(x),\Phi(y)\rangle$3 at $k(x,y)=\langle \Phi(x),\Phi(y)\rangle$4 (Li et al., 2016).

These binary and quantized constructions serve two distinct purposes. First, they provide explicit kernels or similarity estimators suitable for linear downstream models. Second, they furnish locality-sensitive hash families for approximate nearest-neighbor search. For the 2-bit scheme, the recommended uniform quantization widths are $k(x,y)=\langle \Phi(x),\Phi(y)\rangle$5 to $k(x,y)=\langle \Phi(x),\Phi(y)\rangle$6 for high target similarity and $k(x,y)=\langle \Phi(x),\Phi(y)\rangle$7 to $k(x,y)=\langle \Phi(x),\Phi(y)\rangle$8 for more moderate similarity (Li et al., 2016).

4. Layered random nonlinear mappings: the LaRP framework

The Layered Random Projection (LaRP) framework departs from single-layer random kitchen sinks by alternating localized random linear projections with global nonlinearities. If $k(x,y)=\langle \Phi(x),\Phi(y)\rangle$9, each layer applies

$\Phi$0

where the LRPE stage $\Phi$1 is an ensemble of sparse banded Toeplitz projections and the NONL stage $\Phi$2 consists of absolute value rectification followed by sliding-window median regularization (Chung et al., 2016).

Each LRPE projection uses a matrix

$\Phi$3

with a short kernel $\Phi$4 sampled from a learned uniform distribution $\Phi$5. Only the bounds $\Phi$6 are trained, so the number of trainable parameters is $\Phi$7. The NONL stage applies $\Phi$8 and then a median filter on a $\Phi$9 window, producing spatially regularized feature maps (Chung et al., 2016).

This separation of linear kernels and nonlinearities is central to LaRP’s design. The linear stage remains sparse and localized, while the nonlinearity and regularization are parameter-free. The reported per-example complexity is

$\phi(x)$0

dominated by the sparse LRPE transforms. In the paper’s configuration, $\phi(x)$1, $\phi(x)$2, $\phi(x)$3, $\phi(x)$4, $\phi(x)$5, and the final feature dimensionality used for classification is $\phi(x)$6 (Chung et al., 2016).

The empirical study reports the following test errors:

Dataset	LaRP	Best reported baseline
MNIST	1.30% at $\phi(x)$7	1.44% for CM-TS at $\phi(x)$8
COIL-100	0.36% at $\phi(x)$9	3.96% for CM-TS at $n\times n$0

On MNIST, LaRP improves by $n\times n$1 over the best $n\times n$2 baseline and by $n\times n$3 over the best $n\times n$4 baseline; on COIL-100, the improvements are $n\times n$5 and $n\times n$6, respectively. The paper does not report runtime or memory benchmarks, and it does not provide formal bias or variance bounds to a target kernel class, so its claims are empirical and architectural rather than analytic kernel-approximation guarantees (Chung et al., 2016).

5. Random projections inside neural, operator, and probabilistic models

One line of work inserts fixed random projections into otherwise trainable neural networks. In the fully connected case, the projected layer is

$n\times n$7

where $n\times n$8 is drawn once and frozen, while $n\times n$9 and $k(x-y)$0 are trainable. The theoretical motivation comes from stable embeddings of sparse vectors and smooth manifolds: the number of neurons needed to approximate a Lipschitz function depends on sparsity or manifold dimension and only weakly on ambient dimension. Empirically, the paper reports substantial parameter and FLOP reductions with moderate or task-dependent accuracy loss; for example, an FCNN on MNIST drops from $k(x-y)$1M parameters and $k(x-y)$2M FLOPs to $k(x-y)$3M parameters and $k(x-y)$4M FLOPs when the projection dimension is $k(x-y)$5, with error increasing from $k(x-y)$6 to $k(x-y)$7 (Cai et al., 2018).

RandONet applies the same general idea to operator learning. It uses random branch features $k(x-y)$8 for discretized input functions and random trunk features $k(x-y)$9 for output locations, and represents the operator by

$p(w)$0

The branch can be either a linear Johnson–Lindenstrauss embedding or a nonlinear Random Fourier Feature Network embedding, while the trunk is a nonlinear random-feature map, typically with $p(w)$1. Only the matrix $p(w)$2 is trained, using least-squares solvers such as Tikhonov regularization, tSVD, or complete orthogonal decomposition. The paper proves a universal approximation theorem for nonlinear operators and reports large empirical gains over “vanilla” DeepONets on several PDE-related tasks; for the 1D viscous Burgers operator, RandONet-RFFN attains MSE $p(w)$3 and median $p(w)$4 error $p(w)$5, compared with DeepONet MSE $p(w)$6 and median $p(w)$7 error $p(w)$8 (Fabiani et al., 2024).

A different interpretation appears in Bayesian nonlinear PCA using random fields. There the observation model is

$p(w)$9

with $\phi(x)\in\mathbb{R}^D$00 an orthonormal matrix that varies with latent position $\phi(x)\in\mathbb{R}^D$01. The collection $\phi(x)\in\mathbb{R}^D$02 is coupled by a Markov-random-field prior

$\phi(x)\in\mathbb{R}^D$03

where $\phi(x)\in\mathbb{R}^D$04 decays as a Gaussian kernel of latent distance. In this model, nonlinear projection arises from a smooth field of local linear projections rather than from a fixed random feature map. The reported MNIST experiment reduces nearest-neighbor mismatches in latent space from $\phi(x)\in\mathbb{R}^D$05 under PCA to $\phi(x)\in\mathbb{R}^D$06 under the proposed model (0802.1258).

6. Trade-offs, limitations, and broader extensions

Across these families, three trade-offs recur. The first is analytic specificity versus flexibility: RFF provides strong guarantees for shift-invariant kernels via Bochner’s theorem, whereas LaRP constructs hierarchical discriminative embeddings without targeting a specific analytic kernel and therefore lacks corresponding approximation bounds (Ghojogh et al., 2021, Chung et al., 2016). The second is structural efficiency versus concentration: structured sign embeddings reduce randomness and memory to $\phi(x)\in\mathbb{R}^D$07, but the strongest guarantees require balancing transforms such as $\phi(x)\in\mathbb{R}^D$08; without them, concentration is weaker (Choromanska et al., 2015). The third is model compression versus assumption strength: random-projection neural architectures gain parameter efficiency when data are sparse or lie on low-dimensional manifolds, but those structural assumptions are explicit in the theory (Cai et al., 2018). RandONet similarly benefits from aligned outputs and cheap least-squares solvers, while its unaligned formulation has much higher computational complexity (Fabiani et al., 2024). For sign $\phi(x)\in\mathbb{R}^D$09-stable features, closed-form kernel identifications are available only for $\phi(x)\in\mathbb{R}^D$10, $\phi(x)\in\mathbb{R}^D$11 under histogram conditions, and $\phi(x)\in\mathbb{R}^D$12 (Li, 2015).

Recent work extends the notion of nonlinear random projection beyond classical feature maps. In subsampled least squares and fast CUR, the random oblique projector $\phi(x)\in\mathbb{R}^D$13 is nonlinear because of pseudoinversion, and standard sampling schemes can therefore be statistically biased even when the sketched Gram matrix is unbiased. The debiasing framework based on $\phi(x)\in\mathbb{R}^D$14 corrects this bias at the matrix level and yields sharper bias–variance guarantees for subsampled OLS and fast CUR (Niu et al., 24 May 2026). In quantum information, random projection onto lower-dimensional subspaces before a multi-copy collective measurement creates a tunable trade-off between coherent quantum resources and sample complexity; after compressing an $\phi(x)\in\mathbb{R}^D$15-qubit state to a $\phi(x)\in\mathbb{R}^D$16-qubit subspace, estimating $\phi(x)\in\mathbb{R}^D$17 requires approximately $\phi(x)\in\mathbb{R}^D$18 copies, so each projected-out qubit multiplies the sampling cost by $\phi(x)\in\mathbb{R}^D$19 (Liu et al., 18 Jun 2026).

Taken together, these results show that random nonlinear projections are not a single algorithmic object but a family of randomized mechanisms for exposing nonlinear structure in explicit form. The unifying principle is that a cheap randomized transform, followed by a nonlinear map, estimator, or reconstruction step, can replace more expensive implicit nonlinear machinery while preserving enough geometry, similarity, or operator structure for downstream inference.