Structured Orthogonal Random Features (SORF)

• SORF is a structured method for kernel approximation that replaces dense orthogonal transforms with products of Hadamard matrices and random sign flips.
• The technique achieves asymptotically unbiased Gaussian-kernel approximation with a bounded error of O(τ/√d) while reducing estimator variance compared to standard RFF.
• Its efficient feature mapping enables scalable kernel ridge regression and ensemble molecular learning by converting kernel evaluations into fast, explicit trigonometric features.

Searching arXiv for the cited SORF and related papers to ground the article in current literature. Structured Orthogonal Random Features (SORF) are a structured random-feature construction for approximating kernels by explicit finite-dimensional feature maps. In the Gaussian-kernel setting, SORF replaces dense random or dense orthogonal transforms with products of Walsh–Hadamard matrices and random sign diagonals, so that kernel methods can be evaluated through fast trigonometric features rather than through full kernel matrices. In the recent chemical-machine-learning literature, SORF has also been used as the central approximation device for rewriting kernel ridge regression over conformer ensembles into explicit multilevel cosine-feature models, yielding what is described as a physics-motivated trigonometric neural network (Yu et al., 2016, Karandashev, 27 May 2025).

1. Position within random-feature methods

SORF arises from the random-feature approach to shift-invariant kernels. For a Gaussian kernel,

$$K(x,y)=\exp\!\left(-\frac{\|x-y\|^2}{2\sigma^2}\right),$$

standard Random Fourier Features (RFF) sample frequencies independently from the Gaussian spectral density and use sinusoidal features so that the kernel is approximated by an inner product in feature space. Orthogonal Random Features (ORF) modify this construction by replacing the independent Gaussian directions with properly scaled orthogonal directions, motivated by the observation that independent directions can cluster while orthogonal directions are more uniformly spread. SORF is the fast structured variant of this orthogonality-based idea: it seeks the variance-reduction benefits of orthogonality without the cost of explicitly forming or multiplying by dense orthogonal matrices (Yu et al., 2016, Liu et al., 2020).

In the survey literature, SORF is placed in the family of Monte Carlo random features with orthogonality-based variance reduction. Within that taxonomy, RFF, ORF, and SORF differ chiefly in how they sample feature directions. RFF uses independent isotropic Gaussian sampling; ORF uses explicit orthogonalization; SORF uses structured approximate orthogonalization via repeated Hadamard transforms and random sign flips. This places SORF at the intersection of two design objectives: lower estimator dispersion through orthogonality and lower computational cost through structure (Liu et al., 2020).

The broader structured-embedding literature treats SORF-like transforms as a special case of $\mathbf{S}\mathbf{D}$-product random orthogonal embeddings. In that view, SORF is a particular Hadamard-based instance of a wider class of structured random orthogonal matrices used for kernel approximation, angular kernels, and dimensionality reduction (Choromanski et al., 2017).

2. Core construction

The canonical SORF transform is

$$\bm{W}_{\text{SORF}}=\frac{\sqrt d}{\varsigma}\,\bm H\bm D_1\bm H\bm D_2\bm H\bm D_3,$$

where $\bm H$ is the normalized Walsh–Hadamard matrix and $\bm D_1,\bm D_2,\bm D_3$ are diagonal sign-flipping matrices with i.i.d. Rademacher entries. In the original ORF-to-SORF development, the analogous form is written with $\sigma$ in place of $\varsigma$; the structural content is the same: a Hadamard–Diagonal product scaled to mimic Gaussian-kernel random features (Yu et al., 2016, Liu et al., 2020).

This construction is intended as a structured substitute for the dense orthogonal matrix used in ORF. Because each $D_i$ is orthogonal and $H$ is orthogonal after normalization, their product is orthogonal up to the scaling convention. The use of three blocks is not arbitrary. The survey describes the roles of the blocks as follows: $\bm H\bm D_1$ makes vectors balanced, $\mathbf{S}\mathbf{D}$0 makes vectors closer to orthogonal, and $\mathbf{S}\mathbf{D}$1 controls the overall capacity of the structured transform. For $\mathbf{S}\mathbf{D}$2, the balancing step is accompanied by the bound

$\mathbf{S}\mathbf{D}$3

The same survey relates SORF to the generalized ROM construction

$\mathbf{S}\mathbf{D}$4

and states that odd $\mathbf{S}\mathbf{D}$5 tends to work better than even $\mathbf{S}\mathbf{D}$6, which helps explain the conventional choice $\mathbf{S}\mathbf{D}$7 in SORF (Liu et al., 2020).

In the chemical-learning formulation, the corresponding random-feature layer is written as a trigonometric map built from Hadamard-structured transforms and random phase shifts. Its defining approximation is

$\mathbf{S}\mathbf{D}$8

so the Gaussian kernel is replaced by the inner product of explicit cosine features (Karandashev, 27 May 2025).

3. Approximation properties, bias, and variance

The original ORF analysis for Gaussian kernels states that ORF remains unbiased when the feature directions are constructed as $\mathbf{S}\mathbf{D}$9, with $$\bm{W}_{\text{SORF}}=\frac{\sqrt d}{\varsigma}\,\bm H\bm D_1\bm H\bm D_2\bm H\bm D_3,$$0 taken from a random orthonormal basis and $$\bm{W}_{\text{SORF}}=\frac{\sqrt d}{\varsigma}\,\bm H\bm D_1\bm H\bm D_2\bm H\bm D_3,$$1 sampled so that the row norms match Gaussian norms. In that setting, each $$\bm{W}_{\text{SORF}}=\frac{\sqrt d}{\varsigma}\,\bm H\bm D_1\bm H\bm D_2\bm H\bm D_3,$$2, and the variance reduction comes from negative covariance between different cosine terms. The resulting comparison with RFF is summarized by

$$\bm{W}_{\text{SORF}}=\frac{\sqrt d}{\varsigma}\,\bm H\bm D_1\bm H\bm D_2\bm H\bm D_3,$$3

and

$$\bm{W}_{\text{SORF}}=\frac{\sqrt d}{\varsigma}\,\bm H\bm D_1\bm H\bm D_2\bm H\bm D_3,$$4

with the strongest gain for small $$\bm{W}_{\text{SORF}}=\frac{\sqrt d}{\varsigma}\,\bm H\bm D_1\bm H\bm D_2\bm H\bm D_3,$$5 (Yu et al., 2016).

For SORF itself, the survey literature is more conservative. It states that SORF is not an unbiased estimator of the Gaussian kernel, but satisfies the asymptotic unbiasedness bound

$$\bm{W}_{\text{SORF}}=\frac{\sqrt d}{\varsigma}\,\bm H\bm D_1\bm H\bm D_2\bm H\bm D_3,$$6

The same source emphasizes that SORF’s variance-reduction effect is aligned with the general orthogonalization principle, while also labeling its “lower variance than RFF” status as unknown in the summary table. In this formulation, the main theoretical guarantee is asymptotic Gaussian-kernel fidelity rather than a closed-form variance formula comparable to the ORF case (Liu et al., 2020).

A sharper theoretical distinction appears in the analysis of Haar-orthogonal directions. “Orthogonal Random Features: Explicit Forms and Sharp Inequalities” studies orthogonal random features built from columns of a Haar orthogonal matrix and shows that, in that regime,

$$\bm{W}_{\text{SORF}}=\frac{\sqrt d}{\varsigma}\,\bm H\bm D_1\bm H\bm D_2\bm H\bm D_3,$$7

so the expected kernel is a normalized Bessel kernel rather than the Gaussian kernel. The same work gives explicit variance formulas in terms of normalized Bessel functions and proves sharp inequalities showing reduced dispersion relative to RFF on a dimension-dependent interval (Demni et al., 2023).

These results are often read as contradictory, but the data support a narrower conclusion: the exact mean-kernel behavior depends on the precise orthogonal-feature construction being analyzed. The scaled-ORF treatment, the structured Hadamard-diagonal approximation, and the pure Haar-column formulation are not identical objects. This suggests that “orthogonality helps” is robust, whereas the exact unbiasedness claim is model-dependent (Yu et al., 2016, Demni et al., 2023).

4. Computational role in kernel methods

The central computational idea of SORF is to replace expensive kernel evaluations with explicit features. In the notation used for molecular learning,

$$\bm{W}_{\text{SORF}}=\frac{\sqrt d}{\varsigma}\,\bm H\bm D_1\bm H\bm D_2\bm H\bm D_3,$$8

so kernel ridge regression (KRR) can be written as a linear model in feature space rather than as a full kernel expansion. The standard KRR estimator is expressed as

$$\bm{W}_{\text{SORF}}=\frac{\sqrt d}{\varsigma}\,\bm H\bm D_1\bm H\bm D_2\bm H\bm D_3,$$9

with $\bm H$0. In the SORF version, the kernel is replaced by a feature inner product,

$\bm H$1

with $\bm H$2 (Karandashev, 27 May 2025).

Because the feature dimension $\bm H$3 can be chosen smaller than the number of training points $\bm H$4, the memory footprint and linear solve can be reduced substantially relative to a full kernel matrix. The cited chemical-learning work emphasizes two gains. First, when $\bm H$5, the regression solve can be carried out by SVD with $\bm H$6 cost. Second, for local and orbital kernels, evaluating explicit features scales linearly with molecular size rather than bilinearly in the number of atoms or conformers. This is particularly important for ensemble models, where direct kernel evaluation would otherwise require pairwise comparisons over all conformers of both molecules (Karandashev, 27 May 2025).

At the transform level, the fast Walsh–Hadamard transform yields $\bm H$7 multiplication for a single Hadamard stage, and the survey lists SORF with extra memory $\bm H$8 and time $\bm H$9 for computing $\bm D_1,\bm D_2,\bm D_3$0. This is substantially better than dense ORF, which is described as needing $\bm D_1,\bm D_2,\bm D_3$1 memory and time in the survey’s complexity comparison (Liu et al., 2020).

5. Multilevel SORF in molecular and conformer learning

In ensemble-based molecular learning, SORF has been reinterpreted as a structured trigonometric neural architecture. The basic SORF construction is treated as a layer $\bm D_1,\bm D_2,\bm D_3$2 producing cosine activations; sum, switch, weighting, normalization, and mixed-extensive layers are then composed to mirror conformers, atoms, and localized orbitals. This formulation is referred to as Multilevel SORF (MSORF), and it is explicitly described as a physics-motivated trigonometric neural network rather than a generic black-box network (Karandashev, 27 May 2025).

For local atomic kernels, the feature construction uses a sum layer over atom-wise SORF features,

$\bm D_1,\bm D_2,\bm D_3$3

which reproduces the local Gaussian kernel. For element-specific local kernels, a switch layer initializes separate SORF parameters for each nuclear charge, yielding $\bm D_1,\bm D_2,\bm D_3$4 and approximating the different-atom-type local kernel $\bm D_1,\bm D_2,\bm D_3$5. For orbital-based FJK kernels, localized orbital vectors are combined by a weighted sum layer, followed by normalization and another SORF map, reproducing the nested orbital Gaussian structure used in FJK (Karandashev, 27 May 2025).

The conformer-ensemble extension is a particularly prominent use case. Molecules are represented by multiple conformers $\bm D_1,\bm D_2,\bm D_3$6 with Boltzmann weights $\bm D_1,\bm D_2,\bm D_3$7. A direct ensemble kernel is naturally written as a weighted double sum over conformers, but that is expensive. The SORF reformulation replaces the double sum by aggregated explicit features. For extensive properties, conformer-level features are summed. For intensive properties, normalization layers are introduced to ensure invariance under molecule multiplication or duplicated copies of the system. A mixed-extensive layer $\bm D_1,\bm D_2,\bm D_3$8 is introduced to interpolate between a purely linear extensive contribution and a nonlinear SORF contribution, so that extensive observables can be modeled while deviations from strict additivity remain learnable (Karandashev, 27 May 2025).

6. Empirical behavior

In the general random-feature literature, SORF is reported to behave very similarly to ORF and often among the best methods for Gaussian kernel approximation. On benchmark classification datasets, it is grouped with ORF and QMC among the methods that often perform best for larger feature sizes in approximation quality. On MNIST and CIFAR-10, ORF and SORF are reported to yield the best approximation quality on the Gaussian kernel, although test-accuracy differences across methods are often small. On the ultra-large MNIST-8M benchmark, the survey reports Gaussian-kernel approximation error $\bm D_1,\bm D_2,\bm D_3$9 for both ORF and SORF, and test error around $\sigma$0 for SORF in that experiment (Liu et al., 2020).

In chemical machine learning, MSORF has been evaluated on the “literature electrochemical stability” dataset of oxidation potentials in acetonitrile (LES) and on FreeSolv hydration free energies. It is combined with CM, SLATM, aSLATM, SOAP, FCHL19, cMBDF, and the orbital-based FJK representation. For LES, the best-performing geometric descriptors under MSORF, especially FCHL19, cMBDF, SLATM, and aSLATM, reach near-experimental accuracy with a few hundred training molecules and are described as comparable to state-of-the-art results. For FreeSolv, the same general pattern is reported, with aSLATM, FCHL19, and SLATM performing best and reaching chemical accuracy with training sizes in the low hundreds (Karandashev, 27 May 2025).

The same study reports that FJK, especially with a pair of Slater determinants for redox-like changes, performs less well on these datasets, suggesting that the chosen approximate electronic structure level was not sufficiently accurate for the target observables. It also reports that using an entire conformer ensemble versus only the lowest-energy conformer made little difference for the tested datasets and representations, with differences mostly within statistical uncertainty. This shifts the empirical emphasis toward representation choice, even though SORF is what makes the full ensemble treatment computationally practical and lossless in principle (Karandashev, 27 May 2025).

7. Implementation practice, related constructions, and limitations

The implementation workflow described for ensemble molecular learning is explicit. Conformers are generated with RDKit/MMFF94 via Morfeus, low-weight conformers are discarded through a cutoff, the SORF feature size $\sigma$1 is chosen as a power of two, and the Hadamard dimension is padded as needed. The workflow also uses a cut Boltzmann weighting that discards negligible conformers while normalizing the remaining weights. Hyperparameters include kernel widths $\sigma$2, regularization $\sigma$3, and, for mixed-extensive and shifted-target variants, additional shift parameters. Hyperparameter optimization is performed by minimizing leave-one-out cross-validation errors efficiently computed from the SORF design matrix. To make this practical, the same work introduces self-consistent Huber and self-consistent LogCosh losses, described as smoothed, MAE-like objectives with a tunable relative-error parameter $\sigma$4 (Karandashev, 27 May 2025).

SORF also sits within a wider family of structured orthogonal methods. The structured random orthogonal embedding literature formulates Hadamard–Rademacher products as a broader $\sigma$5-product framework and reports that increasing the number of blocks gives diminishing returns, that odd $\sigma$6 is better than even $\sigma$7, and that $\sigma$8 is often a practical sweet spot. A complex hybrid extension is further reported to reduce MSE by a factor of $\sigma$9 relative to the real-valued $\varsigma$0-product estimator, but that result belongs to the broader ROM setting rather than to the standard real-valued SORF construction (Choromanski et al., 2017).

Several limitations are explicit in the cited literature. The survey notes that structured methods such as Fastfood, SORF, and ROM do not always show large runtime savings in MATLAB implementations because the fast Walsh–Hadamard transform is not especially efficient in that environment. It also states that ORF and SORF do not show a clear advantage over RFF for arc-cosine kernels, and that the theoretical variance-reduction results for ORF and SORF are only established for the Gaussian kernel in the cited discussion (Liu et al., 2020).

A common misconception is that SORF is simply “Gaussian RFF but faster.” The literature gives a more qualified picture. SORF is motivated by Gaussian-kernel approximation and by the variance-reduction principle of orthogonality, but exact theoretical statements depend on the sampling model being used. The survey frames SORF as asymptotically unbiased for the Gaussian kernel, whereas the Haar-orthogonal analysis shows that exact orthogonal directions induce a normalized Bessel kernel in expectation. The stable point across these perspectives is that orthogonality and structure jointly define SORF’s identity: orthogonality alters the statistical geometry of the feature map, and structure makes that geometry computationally usable at scale (Liu et al., 2020, Demni et al., 2023).