Orthogonal Random Features (ORF)

• ORF are random-feature techniques that use mutually orthogonal projection vectors with tailored radial scaling to lower kernel approximation variance.
• They replace i.i.d. Gaussian sampling with a structured orthogonal ensemble, improving Monte Carlo kernel estimates for Gaussian kernels.
• Variants like SORF and GORF extend ORF to fast transforms and generalized kernels, offering computational efficiency without sacrificing approximation quality.

Searching arXiv for Orthogonal Random Features and closely related papers to ground the article in current literature. Orthogonal Random Features (ORF) are a random-feature mechanism for kernel approximation in which the projection vectors are constructed to be mutually orthogonal, rather than sampled independently as in standard Random Fourier Features (RFF). In the Gaussian-kernel setting, ORF replaces the random Gaussian matrix by a properly scaled random orthogonal matrix and thereby significantly decreases kernel approximation error (Yu et al., 2016). The method preserves the random-feature paradigm $\hat{K}(x,y)=\phi(x)^\top \phi(y)$ while altering the sampling geometry of the projection matrix to reduce estimator variance. Subsequent work has connected ORF to structured orthogonal embeddings, structured fast transforms, indefinite-kernel approximations, positive random features for softmax and Gaussian kernels, and explicit bias–variance analyses based on Haar orthogonal matrices (Choromanski et al., 2017, Luo et al., 2021, Likhosherstov et al., 2022, Demni et al., 2023).

1. Origins and problem setting

ORF arose from the observation that standard RFF, although unbiased for shift-invariant kernels such as the Gaussian kernel, can exhibit substantial kernel approximation error because the frequency vectors are sampled i.i.d. from the target spectral distribution (Yu et al., 2016). The motivating claim is that replacing these i.i.d. Gaussian directions with orthogonal directions yields lower-variance Monte Carlo estimates of the kernel for the same number of features (Yu et al., 2016).

Within the random-feature framework, the Gaussian kernel is written as

$K(x, y) = e^{-\frac{||x - y||^2}{2\sigma^2}}$

and approximated by a feature map built from sampled projection vectors $w_i$. In the standard RFF construction for the Gaussian kernel, $w \sim N(0,\frac{1}{\sigma^2}I)$, and

$$\phi(x) = \sqrt{\frac{1}{D}} \left[ \sin(w_1^T x), \ldots, \sin(w_D^T x), \cos(w_1^T x), \ldots, \cos(w_D^T x) \right]^T,$$

with kernel estimate

$\hat{K}(x,y) = \phi(x)^T \phi(y).$

ORF keeps the same trigonometric map but changes the law of the matrix $W$ that contains the projection vectors (Yu et al., 2016).

The basic intuition given in the literature is geometric: random Gaussian directions are already almost orthogonal in high dimensions, but enforcing exact orthogonality eliminates redundancy among sampled directions and improves approximation quality (Yu et al., 2016). Related work on structured random orthogonal embeddings describes this as reducing covariance between random projections and making the projections “repel” one another in regions that matter for kernel estimation (Choromanski et al., 2017). This suggests that ORF should be understood not merely as a sampling trick, but as a variance-reduction principle within Monte Carlo kernel approximation.

2. Core construction and relation to RFF

The canonical ORF construction for the Gaussian kernel is

$W_{\mathrm{ORF}} = \frac{1}{\sigma} S Q,$

where $Q \in \mathbb{R}^{d \times d}$ is a random orthogonal matrix drawn from the Haar measure and $S \in \mathbb{R}^{d \times d}$ is diagonal with i.i.d. $K(x, y) = e^{-\frac{||x - y||^2}{2\sigma^2}}$0-distributed entries (Yu et al., 2016). The role of $K(x, y) = e^{-\frac{||x - y||^2}{2\sigma^2}}$1 is to supply orthogonal directions, while the role of $K(x, y) = e^{-\frac{||x - y||^2}{2\sigma^2}}$2 is to match the norm distribution of Gaussian random vectors so that the marginal radial law agrees with that of the Gaussian ensemble (Yu et al., 2016).

This scaling is central. The literature states that the $K(x, y) = e^{-\frac{||x - y||^2}{2\sigma^2}}$3 factors ensure that the norm of each row of $K(x, y) = e^{-\frac{||x - y||^2}{2\sigma^2}}$4 matches the distribution of norms in the Gaussian ensemble, thereby preserving the intended marginals (Yu et al., 2016). In the survey treatment, the same construction is presented as

$K(x, y) = e^{-\frac{||x - y||^2}{2\sigma^2}}$5

with the accompanying interpretation that ORF randomizes the direction using an orthogonal set on the sphere and randomizes the length according to the appropriate radial distribution for the target Fourier law (Liu et al., 2020).

For $K(x, y) = e^{-\frac{||x - y||^2}{2\sigma^2}}$6, one uses the first $K(x, y) = e^{-\frac{||x - y||^2}{2\sigma^2}}$7 rows of the orthogonal construction; for $K(x, y) = e^{-\frac{||x - y||^2}{2\sigma^2}}$8, one concatenates multiple independent $K(x, y) = e^{-\frac{||x - y||^2}{2\sigma^2}}$9 blocks (Yu et al., 2016). In later formulations for positive random features and related mechanisms, the same block-orthogonal strategy is described as partitioning the features into $w_i$0 independent orthogonal blocks when $w_i$1 (Likhosherstov et al., 2022).

The distinction from RFF is therefore precise. In standard RFF, the frequencies are sampled i.i.d. from $w_i$2; in ORF, the directions are mutually orthogonal while the radial scaling is adjusted so that each row still has the desired marginal law (Liu et al., 2020). The estimator retains the same random-feature form, but the joint dependence structure among the rows is changed from independence to negative dependence induced by orthogonality (Likhosherstov et al., 2022). A common misconception is that ORF changes the feature map itself; in the original Gaussian-kernel formulation, it changes the sampling mechanism for the frequency matrix, not the trigonometric feature template (Yu et al., 2016).

3. Variance reduction, bias, and refined theory

The classical result associated with ORF is variance reduction relative to RFF in Gaussian-kernel approximation. For RFF, with $w_i$3 features and $w_i$4,

$w_i$5

For ORF,

$w_i$6

and the variance ratio is given approximately by

$w_i$7

The stated implication is that ORF offers lower variance than RFF, especially for small $w_i$8, i.e. when points are close in the input space (Yu et al., 2016). The survey restates the same phenomenon, emphasizing that the reduction is especially pronounced for large $w_i$9 and small distance between inputs (Liu et al., 2020).

A broader theory of orthogonal random embeddings reaches an analogous conclusion through mean-squared-error analysis for dot-product and angular-kernel estimation. For Gaussian orthogonal estimators,

$w \sim N(0,\frac{1}{\sigma^2}I)$0

and for structured estimators based on $w \sim N(0,\frac{1}{\sigma^2}I)$1-product matrices, sub-sampling without replacement always yields lower MSE than the traditional iid setting when $w \sim N(0,\frac{1}{\sigma^2}I)$2 (Choromanski et al., 2017). This work frames orthogonality as a general statistical mechanism rather than a Gaussian-kernel-specific curiosity.

Later theory complicates the standard narrative. An explicit analysis of ORF based on Haar orthogonal matrices shows that the expected ORF kernel is not the Gaussian kernel but a Bessel kernel: $w \sim N(0,\frac{1}{\sigma^2}I)$3 where

$w \sim N(0,\frac{1}{\sigma^2}I)$4

The same work gives the variance formula

$w \sim N(0,\frac{1}{\sigma^2}I)$5

with $w \sim N(0,\frac{1}{\sigma^2}I)$6 (Demni et al., 2023). It also derives sharp exponential upper and lower bounds and supports the view that ORF is less dispersed than RFF over a wide range of distances (Demni et al., 2023).

This creates a noteworthy interpretive tension in the literature. Earlier descriptions emphasize that with correct scaling ORF remains unbiased for the Gaussian kernel (Yu et al., 2016, Liu et al., 2020), whereas the later Haar-based analysis states that ORF does not approximate the Gaussian kernel but a Bessel kernel (Demni et al., 2023). A plausible implication is that what counts as “unbiased” depends sensitively on the exact ensemble and finite-dimensional formulation under consideration. The literature supplied here does not reconcile these positions completely, but it clearly establishes that the variance-reduction story is robust even as the precise bias characterization has become more nuanced.

4. Structured and fast variants

A major practical limitation of dense ORF is the cost of generating and applying a dense random orthogonal matrix. The original ORF paper therefore introduced Structured Orthogonal Random Features (SORF), which replace the generic orthogonal transform with a product of Walsh–Hadamard and diagonal sign matrices: $w \sim N(0,\frac{1}{\sigma^2}I)$7 where $w \sim N(0,\frac{1}{\sigma^2}I)$8 is the normalized Walsh–Hadamard matrix and $w \sim N(0,\frac{1}{\sigma^2}I)$9 are diagonal Rademacher sign matrices (Yu et al., 2016).

The computational claim is explicit: SORF reduces the time cost from $$\phi(x) = \sqrt{\frac{1}{D}} \left[ \sin(w_1^T x), \ldots, \sin(w_D^T x), \cos(w_1^T x), \ldots, \cos(w_D^T x) \right]^T,$$0 to $$\phi(x) = \sqrt{\frac{1}{D}} \left[ \sin(w_1^T x), \ldots, \sin(w_D^T x), \cos(w_1^T x), \ldots, \cos(w_D^T x) \right]^T,$$1, with almost no compromise in kernel approximation quality compared to ORF (Yu et al., 2016). Each multiplication by $$\phi(x) = \sqrt{\frac{1}{D}} \left[ \sin(w_1^T x), \ldots, \sin(w_D^T x), \cos(w_1^T x), \ldots, \cos(w_D^T x) \right]^T,$$2 is implemented by the Fast Walsh–Hadamard Transform in $$\phi(x) = \sqrt{\frac{1}{D}} \left[ \sin(w_1^T x), \ldots, \sin(w_D^T x), \cos(w_1^T x), \ldots, \cos(w_D^T x) \right]^T,$$3, while the sign-flip matrices cost $$\phi(x) = \sqrt{\frac{1}{D}} \left[ \sin(w_1^T x), \ldots, \sin(w_D^T x), \cos(w_1^T x), \ldots, \cos(w_D^T x) \right]^T,$$4, and the memory overhead is negligible because only the random sign patterns need to be stored (Yu et al., 2016). The survey summarizes the same point more generally: dense ORF remains $$\phi(x) = \sqrt{\frac{1}{D}} \left[ \sin(w_1^T x), \ldots, \sin(w_D^T x), \cos(w_1^T x), \ldots, \cos(w_D^T x) \right]^T,$$5 for feature application but requires expensive orthogonal generation, whereas structured variants such as SORF reduce cost to $$\phi(x) = \sqrt{\frac{1}{D}} \left[ \sin(w_1^T x), \ldots, \sin(w_D^T x), \cos(w_1^T x), \ldots, \cos(w_D^T x) \right]^T,$$6 (Liu et al., 2020).

The structured-random-embedding literature places SORF into a larger class of Random Ortho-Matrices (ROMs). One main family is the $$\phi(x) = \sqrt{\frac{1}{D}} \left[ \sin(w_1^T x), \ldots, \sin(w_D^T x), \cos(w_1^T x), \ldots, \cos(w_D^T x) \right]^T,$$7-product matrix

$$\phi(x) = \sqrt{\frac{1}{D}} \left[ \sin(w_1^T x), \ldots, \sin(w_D^T x), \cos(w_1^T x), \ldots, \cos(w_D^T x) \right]^T,$$8

where $$\phi(x) = \sqrt{\frac{1}{D}} \left[ \sin(w_1^T x), \ldots, \sin(w_D^T x), \cos(w_1^T x), \ldots, \cos(w_D^T x) \right]^T,$$9 is a structured orthogonal matrix such as a Hadamard matrix and $\hat{K}(x,y) = \phi(x)^T \phi(y).$0 are random diagonal sign matrices (Choromanski et al., 2017). In this broader perspective, Hadamard-based orthogonal embeddings are useful for kernel approximation, dimensionality reduction, and angular-kernel estimation, while also yielding computational efficiency through $\hat{K}(x,y) = \phi(x)^T \phi(y).$1 multiplication and $\hat{K}(x,y) = \phi(x)^T \phi(y).$2 storage (Choromanski et al., 2017).

The same line of work also studies complex-valued variants. The hybrid complex-structured estimator satisfies

$\hat{K}(x,y) = \phi(x)^T \phi(y).$3

which is a factor-of-two reduction in MSE over the real-valued structured estimator (Choromanski et al., 2017). This does not alter the basic definition of ORF, but it locates ORF within a family of orthogonality-based constructions whose statistical efficiency can be further improved by structural design.

5. Generalizations beyond the original Gaussian-kernel setting

Although ORF was introduced for Gaussian-kernel approximation, later work generalizes the orthogonality principle to other kernel classes and feature mechanisms. For stationary indefinite kernels, the paper “Towards Unbiased Random Features with Lower Variance For Stationary Indefinite Kernels” introduces generalized orthogonal random features (GORF) (Luo et al., 2021). Starting from a stationary indefinite kernel with signed spectral measure $\hat{K}(x,y) = \phi(x)^T \phi(y).$4, the method uses the Jordan decomposition

$\hat{K}(x,y) = \phi(x)^T \phi(y).$5

normalizes the positive and negative parts, and defines a generalized random feature map built from positive and negative spectral samples (Luo et al., 2021). Orthogonality is then imposed not only within the positive and negative blocks but also across them. The resulting method is stated to be unbiased with lower variance, and the variance difference satisfies

$\hat{K}(x,y) = \phi(x)^T \phi(y).$6

where all terms are negative or zero, so GORF always reduces or, in the worst case, matches the variance of i.i.d.-sampled generalized random features (Luo et al., 2021).

For positive random features used in Gaussian and softmax kernel approximation, the Chefs’ Random Tables framework reports that orthogonal random features applied in optimal positive random features provide additional variance reduction for any dimensionality $\hat{K}(x,y) = \phi(x)^T \phi(y).$7, not only asymptotically for sufficiently large $\hat{K}(x,y) = \phi(x)^T \phi(y).$8 as for RKS (Likhosherstov et al., 2022). In that setting, with orthogonal ensembles,

$\hat{K}(x,y) = \phi(x)^T \phi(y).$9

for some $W$0 (Likhosherstov et al., 2022). Here orthogonality is combined with positivity and boundedness, which the paper treats as crucial for stronger tail bounds and uniform convergence results in softmax-attention approximation (Likhosherstov et al., 2022).

A different extension appears in Simplex Random Features, where ORF serves as an intermediate baseline rather than the endpoint. In that work, ORF is formulated as

$W$1

with $W$2 a random orthogonal matrix and $W$3 a diagonal matrix of i.i.d. $W$4 norms, and is shown to have lower MSE than IIDRFs (Reid et al., 2023). However, the simplex construction

$W$5

is proved to achieve still lower MSE than ORF among weight-independent geometrically coupled positive random features (Reid et al., 2023). This suggests that orthogonality is an important but not universally optimal geometric coupling: more strongly “spread” configurations can outperform mere orthogonality in specific positive-random-feature families.

6. Empirical behavior and applications

Empirically, ORF and SORF are consistently reported to improve kernel approximation quality relative to standard random features. The original ORF paper states that experiments on several datasets verify the effectiveness of ORF and SORF over existing methods, with SORF and ORF yielding almost identical and significantly lower kernel approximation error than RFF and other fast approximations (Yu et al., 2016). Using approximate features in kernel SVMs, SORF and ORF are reported to match or outperform RFF for a given number of features (Yu et al., 2016).

The survey on random features likewise states that ORF achieves consistently lower approximation error than RFF in most cases, particularly for Gaussian kernels and with a moderate-to-large number of features, and that ORF and structured variants occasionally improve classification accuracy on practical large-scale datasets (Liu et al., 2020). At the same time, the survey explicitly notes an open question: better kernel approximation does not always guarantee better generalization in downstream tasks (Liu et al., 2020). This is an important corrective to overly direct readings of variance-reduction results.

The structured-embedding literature broadens the empirical picture. Gaussian orthogonal matrices and Hadamard-based ROMs consistently outperform unstructured Gaussians on MSE for inner-product estimation and Gram-matrix approximation across tested datasets such as g50c, LETTER, and USPS (Choromanski et al., 2017). For the angular kernel, $W$6 and Hadamard-based methods are described as practically indistinguishable and best (Choromanski et al., 2017).

In indefinite-kernel learning, GORF is reported to achieve lower variance and approximation error than GRFF and several competing methods, with higher classification accuracy in SVM experiments and lower RMSE in kernel ridge regression (Luo et al., 2021). In scalable attention, orthogonal random features within OPRF-based Chefs’ Random Tables are reported to yield much lower variance than earlier softmax random features and to improve performance in text, speech, and image Transformer tasks while retaining linear space and time complexity (Likhosherstov et al., 2022). In chemistry, structured orthogonal random features are used to rewrite kernel ridge regression expressions for conformer ensembles, producing physics-motivated trigonometric neural networks and prediction errors comparable to state-of-the-art machine learning approaches on oxidation-potential and hydration-energy datasets (Karandashev, 27 May 2025). These uses reflect the migration of ORF from a kernel-approximation technique into a broader design pattern for scalable models.

7. Conceptual status, misconceptions, and related directions

ORF is often presented as a direct replacement for RFF, but the literature indicates a more layered status. In the survey taxonomy, ORF is a data-independent variance-reduction method that is simpler than leverage-score-based data-dependent sampling while still delivering strong improvements (Liu et al., 2020). It is therefore best viewed as one point in a wider design space that includes structured transforms, positive feature maps, generalized signed-measure constructions, and more elaborate geometric couplings.

Several misconceptions are explicitly or implicitly corrected by the cited papers. One is that orthogonality is useful only asymptotically. In the Chefs’ Random Tables framework, orthogonalization provides variance reduction for any dimensionality $W$7 for OPRFs, in contrast with asymptotic large-$W$8 statements associated with trigonometric RKS (Likhosherstov et al., 2022). Another is that ORF is necessarily the optimal unbiased geometric design. Simplex Random Features prove a strict MSE ordering,

$W$9

within the specified weight-independent geometrically coupled positive-random-feature class (Reid et al., 2023).

A third misconception is that orthogonality alone determines downstream success. The survey stresses that improved kernel approximation need not entail improved generalization (Liu et al., 2020). A plausible implication is that ORF’s benefits are strongest when approximation error is the dominant bottleneck, and weaker when model misspecification, regularization, or task-level inductive bias dominate performance.

Finally, recent work has expanded the meaning of “orthogonal features” beyond random matrix sampling. Physics-Driven Orthogonal Feature Method (PD-OFM) uses physics-informed objectives together with orthogonality regularization to learn nearly orthogonal feature bases for PDEs, rather than sampling them from a random orthogonal ensemble (Jia et al., 3 Feb 2026). The paper contrasts this with standard ORF, which it describes as generating features orthogonally with respect to random distributions but without operator or domain adaptivity (Jia et al., 3 Feb 2026). This suggests an emerging distinction between orthogonality as a probabilistic sampling device and orthogonality as a learned structural constraint.

In summary, Orthogonal Random Features denote a family of kernel-approximation methods centered on replacing i.i.d. random projections with orthogonal ones, typically combined with radial scaling that matches the target ensemble. Their original significance lies in lower kernel approximation variance for Gaussian kernels (Yu et al., 2016). Their later significance lies in the way orthogonality became a reusable principle across structured embeddings, indefinite kernels, positive random features for softmax attention, and fast structured approximations (Choromanski et al., 2017, Luo et al., 2021, Likhosherstov et al., 2022). The subsequent literature both confirms the practical utility of the idea and sharpens its theoretical interpretation, especially regarding finite-dimensional bias, the role of Haar orthogonal ensembles, and the fact that orthogonality, while powerful, is not always the final word in random-feature design (Demni et al., 2023, Reid et al., 2023).