Selective Adaptive Learning (SAL)

• SAL is a machine learning paradigm characterized by selective parameter updates and active data querying to minimize computational cost and interference.
• It decomposes network parameters into modular areas, updating only relevant regions per input to achieve sparse, efficient learning in deep architectures.
• SAL methods excel in adaptive filtering, online prediction, and Bayesian optimization, resulting in improved scalability, energy efficiency, and label efficiency.

Selective Adaptive Learning (SAL) encompasses a family of machine learning methodologies that combine probabilistic adaptivity, data-efficient sample selection, local or modular network updates, and dynamic partitioning of parameter spaces or data streams. SAL achieves efficient, scalable, and biologically-inspired learning through selective updating mechanisms and domain-tailored sample querying. This survey focuses on leading developments, underlying theory, algorithmic implementations, variants in deep learning, adaptive filtering, online learning, Bayesian optimization, and active learning, as consolidated from the published literature (Liu et al., 29 Jan 2026, Castro et al., 2023, Yazdanpanah, 2019, Hvarfner et al., 2023, Bu et al., 2018).

1. Core Principles of Selective Adaptive Learning

SAL is characterized by its selective activation or update of parameters, dynamic adaptation to data or environment, and principled sparsification of learning steps. The general principle is to update only a subset of parameters or query only informative data points, based on data-driven criteria, selection mechanisms, or performance-driven thresholds.

• Selective parameter update: Adaptive learning algorithms frequently constrain updates to parameters or sub-networks directly implicated by a sample, minimizing interference and computational cost (Liu et al., 29 Jan 2026, Yazdanpanah, 2019).
• Adaptive data querying: In active and online learning, SAL methodologies identify which samples to label or query to maximize information gain or minimize risk (Castro et al., 2023, Bu et al., 2018, Hvarfner et al., 2023).
• Partitioned parameter spaces: In deep networks, parameters are grouped into mutually exclusive areas or blocks, and each sample activates only one area per layer, effecting local learning (Liu et al., 29 Jan 2026).
• Feedback alignment: For backpropagation-free training, SAL deploys fixed, asymmetric feedback pathways instead of weight-symmetric transport, addressing biological plausibility (Liu et al., 29 Jan 2026).

A plausible implication is that these techniques collectively enable improved scalability, label/energy efficiency, and robustness to drift compared to monolithic or passive training protocols.

2. SAL in Deep Networks: Architecture and Training Protocol

The deep-learning instantiation of SAL decomposes each weight matrix into $N$ disjoint areas, with routing networks activating a single area per input sample. The main steps per (Liu et al., 29 Jan 2026):

• Parameter decomposition: For each layer $l$, parameters are provided as $\{W^{(l,k)}\}_{k=1}^N$.
• Sample-dependent routing:
  • For input $x_i$, routing features $z_i^{(l)} = x_i W_s^{(l)}$ are projected onto fixed prototype vectors $W_{\text{fix}}^{(l)}$.
  • Cluster assignment $k_i^{(l)}$ is determined by $\arg\max_j p_{i,j}^{(l)}$.
• Sparse forward and feedback: Activation propagates only through $W^{(l,k_i)}$; error is backpropagated via fixed feedback matrix $B^{(l)}$, with auxiliary local loss for selector network.
• Localized parameter update: Only the area-activated parameters are updated:

$$\nabla W^{(l,k)} = \sum_{i:K_i^{(l)}=k}(H^{(l-1)}_{i,:})^\top \delta_i^{(l)}, \quad W^{(l,k)} \leftarrow W^{(l,k)} - \eta_{net} \nabla W^{(l,k)}$$

• Mitigation of gradient interference: The isolation of parameter regions corresponding to distinct semantic clusters prevents conflicting gradient flows. The use of fixed feedback matrices obviates biologically implausible weight-transport requirements.

This protocol enforces an effective $1/N$ parameter sparsity per sample and demonstrates stability and competitive performance on large-scale and deep architectures (up to $128$ layers and $1$B parameters) (Liu et al., 29 Jan 2026).

3. Selective Adaptive Learning in Data-Selective Adaptive Filtering

SAL, implemented via set-membership (SM) filtering, controls adaptive filter updates via error thresholding. Key aspects (Yazdanpanah, 2019):

• Update criterion: The filter is updated only when the error $|e(n)| > \gamma(n)$.
• Projection update:

$$w(n+1) = w(n) + \frac{e(n) - \text{sgn}\{e(n)\}\gamma(n)}{\|x(n)\|^2} x(n)$$

• Algorithms: SAL operates across SM-NLMS, SM-AP, and SM-RLS, and is extended to trinion and quaternion domains for multidimensional adaptive filtering.
• Partial update and sparsity: Only a subset of filter taps is updated; sparsity-aware regularization further reduces computational complexity.
• Performance: SAL variants achieve $5\%$–$20\%$ update rates versus $100\%$ in standard LMS/AP, with comparable MSE and up to $90\%$ multiplication reduction in certain tasks.

This selective updating framework yields superior energy efficiency and is robust to noise and signal sparsity.

4. Online Selective Adaptive Learning: Active Expert Aggregation and Label-Efficiency

In online prediction, the SAL framework achieves worst-case regret guarantees while minimizing label queries through selective sampling (Castro et al., 2023):

• Exponentially-weighted forecaster: Each expert $i$ is assigned weight $w_{i,t}$, updated via importance-weighted losses.
• Selective query probability:

$q_t = \min\{4A_{1,t}(1-A_{1,t}) + \eta/3, 1\}$

where $A_{1,t}$ is the weighted fraction of experts predicting 1.

• Regret and query complexity:
  • Adversarial regime: $O(\sqrt{T \ln N})$ expected regret.
  • Benign stochastic regime: $O((\sqrt{T}/\Delta^2) \log T)$ query complexity for gap $\Delta>0$.
• Active learning rates: Empirically, SAL recovers minimax active learning rates for pool-based learning under Tsybakov-type noise, matching optimal label efficiency without knowing underlying distributional parameters.

SAL thus effectively interpolates between adversarial full-information and label-limited benign regimes.

5. Bayesian and Sequential Active SAL: Statistical Distance-Based and Adaptive Tracking

SAL is generalized in Gaussian process active learning and Bayesian optimization as an acquisition criterion maximizing expected disagreement, measured via a statistical distance, between posterior predictive distributions conditioned on different hyperparameters (Hvarfner et al., 2023):

• SAL acquisition function:

$$\alpha_{\text{SAL}}(x) = \mathbb{E}_{\theta \sim p(\theta|D)} [ d(p(y|x, D, \theta), p(y|x, D)) ]$$

with $d$ being Hellinger, 2-Wasserstein, or KL divergence.

• Theoretical properties: With $d = \mathrm{KL}$, SAL is equivalent to the mutual information criterion BALD.
• Empirical findings: SAL variants achieve superior or state-of-the-art uncertainty calibration, faster hyperparameter convergence, and improved final log-likelihood on several standard Bayesian benchmarks.
• Joint BO+hyperparameter adaptation: SCoreBO extends SAL to simultaneously optimize for function maximization and hyperparameter learning by conditioning on 'fantasy' observations drawn from the fully Bayesian posterior.

Adaptive sequential learning for drifting parameter estimation is also structured as an instance of SAL (Bu et al., 2018):

• Active sample selection: At time $t$, SAL solves an SDP to optimize sample selection over the Fisher information ratio, adapts the number of label queries $K_t$ to guarantee excess risk, and tracks parameter drift via sliding-window estimation.
• Guarantees: The SAL algorithm converges to targeted excess risk with minimal query complexity and proven drift-tracking.

6. Empirical Performance and Comparative Benchmarks

SAL methods have been rigorously benchmarked in diverse domains:

• Deep learning classification: SAL matches or exceeds BP and Mixture-of-Experts on 8/10 datasets, with pronounced gains in deep or wide architectures and superior depth stability ($128$ layers: $65\%$ vs. $38\%$ for BP) (Liu et al., 29 Jan 2026).
• Adaptive filtering: In wind-profile prediction and system identification, SAL achieves $>80\%$ update-rate reduction with comparable MSE and up to $50\%$ lower multiplication counts (Yazdanpanah, 2019).
• Online prediction: SAL achieves label complexity and regret bounds commensurate with full-information EWA or active learning, empirically tracking theoretical rates in both adversarial and benign regimes (Castro et al., 2023).
• Bayesian optimization and active learning: SAL acquisition functions lead to State-of-the-Art (SoTA) negative held-out Marginal Log Likelihood, rapid hyperparameter convergence, and overall ranking improvement on classic GP tasks (Hvarfner et al., 2023).
• Sequential tracking: In dynamic parameter estimation tasks, SAL consistently meets excess risk targets and reduces queries by $\sim50\%$ relative to adaptive baselines (Bu et al., 2018).

A summary table for classification performance in deep learning SAL (16 areas, 2 layers) vs. BP baseline (Liu et al., 29 Jan 2026):

Dataset	BP Baseline (%)	SAL-16 (%)
CIFAR-10	30.81 ± 0.24	36.60 ± 0.48
Digits	38.53 ± 5.78	71.63 ± 5.75
MNIST	90.64 ± 0.18	94.71 ± 0.32
Semeion	35.16 ± 2.72	72.03 ± 3.97

7. Biological Motivations and Implications

SAL’s design is informed by biological principles:

• Sparse local learning: The exclusive activation of a local path mirrors the sparse firing of cortical neurons for specific perceptual patterns.
• Asymmetric feedback: The feedback alignment mechanism relaxes weight transport constraints, resembling biological neural pathways where learning signals need not be symmetric.
• Modular architecture: Parameter partitioning resonates with columnar or regional specialization in the cortex.

A plausible implication is that SAL offers a scalable architectural blueprint for neuromorphic hardware paradigms, emphasizing efficiency and local computation (Liu et al., 29 Jan 2026).

---

In synthesis, Selective Adaptive Learning presents a theoretically grounded, empirically validated, and biologically plausible paradigm for efficient, scalable learning across deep networks, adaptive filters, online learning, and Bayesian optimization, unifying disparate strands of selective updating, active data acquisition, and adaptive control (Liu et al., 29 Jan 2026, Castro et al., 2023, Yazdanpanah, 2019, Hvarfner et al., 2023, Bu et al., 2018).