Subspace Learning Machine (SLM)

• SLM is a supervised learning algorithm that selects optimal one-dimensional discriminant subspaces to maximize class purity in classification and minimize error in regression.
• It employs both probabilistic and adaptive particle swarm optimization methods to efficiently search for high-quality projection weights and achieve diverse, powerful splits.
• By integrating parallel computing and ensemble strategies, SLM constructs shallower trees that deliver state-of-the-art performance over traditional decision trees.

The Subspace Learning Machine (SLM) is a hierarchical, decision tree–style algorithm for supervised classification and regression. It distinguishes itself from traditional decision trees by seeking oblique, data-adaptive splits: at each node, SLM identifies an optimal one-dimensional discriminant subspace—a linear combination of features—that maximizes class purity gains (for classification) or minimizes target error (for regression). This enables shallower and broader trees with few, but powerful, recursive partitions. SLM’s methodology comprises discriminant subspace selection, probabilistic or particle-swarm-based weight search for projections, flexible node partitioning, and ensemble augmentation. Advances in computational acceleration via adaptive particle swarm optimization (APSO) and parallelism make SLM practically viable for moderate to high-dimensional settings (Fu et al., 2022, Fu et al., 2022).

1. Discriminant Subspace Identification and Projection

SLM begins by quantifying the discriminant power of each input feature using the Discriminant Feature Test (DFT). For each feature $d$, a series of thresholds $t_b$ are tested to find the split minimizing a loss,

$$L_{d,\,opt} = \min_{t_b}\left[ \frac{N_+}{N_+ + N_-} H(F_{d, t_b, +}) + \frac{N_-}{N_+ + N_-} H(F_{d, t_b, -}) \right], \quad H(S) = -\sum_{c=1}^K p_c \log p_c,$$

where $H$ is the entropy (or Gini, MSE), and $p_c$ the class proportion. Features are ranked by $L_{d, opt}$; only the most discriminant subset (dimension $D_0 \ll D$) is retained as $S^0$ for all subsequent splits.

At each node, SLM generates multiple candidate projection vectors $a \in \mathbb{R}^{D_0}, \ \|a\|_2 = 1$. Projections are not uniformly random; instead, a probabilistic scheme prioritizes high-quality features, where the $R$ nonzero coefficients, dynamic ranges $A_i$, and activation probabilities $P_i$ are

$$P_i \propto \exp(-\beta i), \quad A_i = \alpha_0 \exp(-\alpha i),$$

over the ranked feature axes (Fu et al., 2022).

2. Node Partitioning and Tree Construction

For each projection vector $a_j$, SLM determines a threshold $t_{j, opt}$ minimizing the DFT or regression loss along $a_j^\top x$. Out of $p$ candidate projections, $q$ are chosen: the first as the best by loss, the subsequent $q-1$ with minimal pairwise correlations (minimax decorrelation), ensuring diverse splits.

Each selected $(a_j, t_{j, opt})$ divides the node’s sample set, forming $2q$ child subspaces as intersections of the split hyperplanes. SLM recurses on each child using the active subspace, terminating when minimum purity, sample, or depth criteria are met. The resulting SLM trees are typically wider (2$q$-way) and shallower than conventional decision trees, yet with higher discriminative power per split (Fu et al., 2022).

In SLM regression (SLR), entropy is replaced with mean squared error, and leaf nodes predict the mean response over their samples.

3. Probabilistic and Particle Swarm Optimization for Projection Search

The original SLM employed a probabilistic search for projection weights, repeating $I$ times (typically $1000$–$2000$) per node. Each iteration sampled $R$ feature axes via $P_d$, drew integer coefficients within $[-\lfloor A_d \rfloor, \lfloor A_d \rfloor]$, normalized $a$, and evaluated the corresponding 1D split.

Recognizing the substantial computational cost, an adaptive particle swarm optimization (APSO) is introduced (Fu et al., 2022). In APSO, a swarm of $M$ particles explores the $n_f$-dimensional subspace (top $n_f$ features by DFT), updating positions $\{x_i\}$ and velocities $\{v_i\}$ per

$$v_i(t+1) = \omega v_i(t) + c_1 r_1 \odot [p_{i, \text{best}} - x_i(t)] + c_2 r_2 \odot [g_{\text{best}} - x_i(t)],$$

where $(\omega, c_1, c_2)$ adapt to swarm dispersion modes: exploration, exploitation, convergence, and jump-out. APSO reliably reduces the projection-search iteration count an order of magnitude (classification: $1000 \rightarrow 110$, regression: $2000 \rightarrow 110$).

Binary SLM trees are used in the APSO variant, reflecting the fact that the global best projection per node can be effectively identified due to the optimizer’s robustness (Fu et al., 2022).

4. Parallelization and Computational Acceleration

The main computational bottleneck—evaluating the split criterion over numerous candidate thresholds—is addressed through both CPU multithreading and GPU acceleration. The C++/Cython core spawns $T$ threads (matching core count) to test subsets of thresholds in parallel, while a CUDA kernel launches one GPU thread per threshold for massively-parallel evaluation. Tree-building, APSO iterations, and node management remain orchestrated on the CPU (Fu et al., 2022).

This design achieves dramatic empirical speedups:

• Maximal observed training acceleration of up to $577\times$ (Python/probabilistic $\rightarrow$ C++/multithreaded/APSO).
• $40$–$100\times$ (Python $\rightarrow$ C++) and further $2$–$3\times$ (C++ $\rightarrow$ multithreaded or GPU).
• Both SLM Forest and SLM Boost ensemble variants benefit identically, with ensemble training shrinking from thousands to tens of seconds for medium datasets.

APSO-accelerated SLM matches or exceeds the predictive accuracy of the original formulation, with classification accuracy and regression MSE remaining within $\pm 1$\% across benchmarks, even at drastically reduced iteration budgets (Fu et al., 2022).

5. Ensemble Methods and Comparative Performance

SLM can serve as a base learner for bagging (SLM Forest) or boosting (SLM Boost). In bagging, $M$ SLM trees are trained on independent bootstrap samples; ensemble prediction is the majority vote (classification) or mean (regression). In boosting—formulated analogously to XGBoost—each SLM tree fits the negative gradient of the loss, optionally using sample weights given by second derivatives.

Empirically, SLM Forest converges faster and to higher accuracy than random forest, while SLM Boost outperforms XGBoost and support vector regression with RBF kernels (SVR-RBF) for a variety of real and synthetic tasks. SLM and SLR trees are uniformly shallower—fewer depth levels and parameters—than standard decision trees or multilayer perceptrons, yet achieve equal or better accuracy (Fu et al., 2022).

6. Computational Complexity and Practical Deployment

The asymptotic per-node complexity of the original probabilistic SLM is

$$\mathcal{O}(I_{\text{orig}} \cdot N \cdot D + \text{pruning} \cdot p^2),$$

while the APSO-accelerated variant is

$\mathcal{O}(I_{\text{pso}} \cdot N \cdot n_f),$

with $I_{\text{pso}} \ll I_{\text{orig}}$ and $n_f \ll D$. Precompilation against SIMD instruction sets (AVX2/AVX-512) is advised for throughput; GPU kernel deployment is recommended in high-dimensional, large-$N$ regimes. SLM integrates directly with scikit-learn APIs via Cython wrappers, supporting drop-in replacement for decision trees in standard pipelines. Hyperparameters such as $n_f$, swarm size $M$, and APSO thresholds are best tuned based on impurity-convergence diagnostics (Fu et al., 2022).

7. Significance and Research Context

SLM synthesizes principles from decision trees, feedforward neural networks, and discriminant analysis, offering the interpretability and recursive logic of trees alongside expressive, learned hyperplane partitions. Its recursive subspace splitting procedure allows both more efficient learning and improved generalization relative to axis-aligned trees. The introduction of APSO- and parallelism-based acceleration resolves the main computational bottleneck, extending SLM’s applicability to moderate and high-dimensional supervised learning tasks.

A plausible implication is that SLM and its variants offer an attractive tradeoff in domains where tree interpretability and projection-based flexibility are both valued. The ensemble SLM approach produces state-of-the-art results on several benchmark datasets for both classification and regression efficiency (Fu et al., 2022, Fu et al., 2022).