Structured Basis Function Networks (s-BFN)

• Structured Basis Function Networks (s-BFN) are a unified framework that integrates multi-hypothesis modeling, ensemble learning, and basis function approximations to enforce smoothness and diversity.
• The paradigm leverages RBF feature maps and Bregman divergence for structured aggregation, enabling efficient closed-form and iterative learning methods.
• s-BFN supports robust compression and uncertainty quantification, demonstrating superior performance in tasks ranging from regression to image classification.

Structured Basis Function Networks (s-BFN) synthesize geometric principles from multi-hypothesis modeling, ensemble learning, and basis function approximations into a unified framework. The s-BFN paradigm encompasses approaches ranging from continuous-in-depth parameterizations in neural ODEs to loss-centric multi-hypothesis ensembles with explicit diversity control. The key insight underlying all s-BFN models is the construction of a structured, basis-functionified space—where functional, statistical, or ensemble smoothness is induced for generalization, compression, or uncertainty quantification. This entry surveys the mathematical definitions, learning procedures, diversity mechanisms, theoretical characterizations, and empirical findings associated with s-BFN, as developed in both continuous-in-depth ODE architectures (Queiruga et al., 2021) and centroidal ensemble frameworks (Dominguez et al., 2 Sep 2025, Dominguez et al., 2023).

1. Mathematical Foundations and Formal Framework

The structured basis function network framework begins by defining a collection of $M$ base predictors, or hypotheses, $h_j(x)=f_{\theta_j}(x)$, each parameterized by $\theta_j$. For a given datum $x_i$, these outputs are aggregated into a structured vector $$D_i=[h_1(x_i)^\top,\ldots,h_M(x_i)^\top]^\top\in\mathbb{R}^{d_D}$$. This vector forms the input domain for the basis function expansion and ensemble aggregation (Dominguez et al., 2 Sep 2025).

A central unifying component is the use of radial basis function (RBF) feature maps $\Phi:\mathbb{R}^{d_D}\to\mathbb{R}^K$ (often with Gaussian kernels), parameterized by centers $C_k$ and widths $\gamma_k$. The s-BFN output is then

$$\hat{y}_i = G(D_i; \alpha, \vartheta) = \Phi(D_i; \vartheta)\alpha,$$

for learnable weights $\alpha\in\mathbb{R}^{K\times C}$ (with $h_j(x)=f_{\theta_j}(x)$0 in regression and $h_j(x)=f_{\theta_j}(x)$1 in classification).

The ensemble combiner is made consistent with the geometry of the loss through the choice of a Bregman divergence $h_j(x)=f_{\theta_j}(x)$2 (for strictly convex $h_j(x)=f_{\theta_j}(x)$3), yielding, in the canonical case, a weighted Bregman centroid:

$h_j(x)=f_{\theta_j}(x)$4

which specializes to either Euclidean means for squared loss or probability centroids for cross-entropy (Dominguez et al., 2 Sep 2025).

An alternative s-BFN instantiation, realized in continuous-depth neural ODEs, expresses each weight tensor $h_j(x)=f_{\theta_j}(x)$5 (for depth $h_j(x)=f_{\theta_j}(x)$6) as

$h_j(x)=f_{\theta_j}(x)$7

with basis functions $h_j(x)=f_{\theta_j}(x)$8 (e.g., piecewise-constant, piecewise-linear, or higher-order) and coefficient tensors $h_j(x)=f_{\theta_j}(x)$9. All $\theta_j$0 coefficients for all parameters are collected into a global parameter tensor, and the model is trained directly on this lower-dimensional, smooth parameter manifold (Queiruga et al., 2021).

2. Learning Algorithms: Closed-Form and Iterative Procedures

In the regression setting with squared loss, closed-form ridge regression can be applied at the ensemble level. Collect the RBF features into $\theta_j$1 for $\theta_j$2 samples. With regularization parameter $\theta_j$3, the aggregation weights are optimized by minimizing

$\theta_j$4

with analytic solution

$\theta_j$5

This closed-form estimator offers computational efficiency and convexity (Dominguez et al., 2 Sep 2025, Dominguez et al., 2023).

For general losses (e.g., cross-entropy) and in regimes where stochastic gradient optimization is required, s-BFN admits an end-to-end iterative algorithm. Each mini-batch proceeds by

1. Forward-passing to obtain all base outputs and losses,
2. Computing diversity-modulated update weights,
3. Updating each base $\theta_j$6 with a diversity-weighted loss,
4. Constructing the structured ensemble input,
5. Calculating ensemble predictions and aggregate loss,
6. Updating combiner weights $\theta_j$7 and kernel parameters $\theta_j$8 (Dominguez et al., 2 Sep 2025).

In the continuous-in-depth setting, basis function coefficients are learned for parameterized $\theta_j$9-dependence of weights and normalization statistics, with the integrator traversing $x_i$0 during both forward and backward passes. Compression is realized by parameter-space projection: after training with a high-rank ($x_i$1) basis, the parameters are projected or interpolated onto a reduced-rank ($x_i$2) basis, minimizing the $x_i$3 error between the high-rank and low-rank expansions with no need to revisit data (Queiruga et al., 2021).

3. Diversity Regulation and Centroidal Aggregation

s-BFN introduces a parametric diversity mechanism via a relaxation of winner-takes-all (WTA) assignment. On each $x_i$4, let $x_i$5. Update weights are specified by:

$x_i$6

where $x_i$7 controls the allocation of the loss signal. $x_i$8 enforces pure WTA with maximal specialization (diversity) and potentially higher variance. As $x_i$9 increases, the model interpolates towards uniform updates, reducing diversity and increasing bias. Empirically, optimal test error is achieved at intermediate $$D_i=[h_1(x_i)^\top,\ldots,h_M(x_i)^\top]^\top\in\mathbb{R}^{d_D}$$0 (Dominguez et al., 2 Sep 2025, Dominguez et al., 2023).

This formalizes and mitigates mode collapse in multi-hypothesis learning, ensuring that predictors meaningfully partition the label space (centroidal Voronoi tessellation) while avoiding convergence to degenerate, low-diversity solutions.

For basis function ODE-Nets, diversity takes the form of enforcing smooth weight evolution along the depth axis, leading to stable high-order integration and robust compression (Queiruga et al., 2021).

4. Theoretical Properties and Loss Geometry Alignment

The s-BFN ensemble combiner is theoretically supported via Bregman geometry. For any strictly convex $$D_i=[h_1(x_i)^\top,\ldots,h_M(x_i)^\top]^\top\in\mathbb{R}^{d_D}$$1, the prediction minimizes a sum of divergences to each base output, coinciding with the centroid in the geometry dictated by the loss. Theoretical results establish that

$$D_i=[h_1(x_i)^\top,\ldots,h_M(x_i)^\top]^\top\in\mathbb{R}^{d_D}$$2

gives the unique combiner under the induced loss (Dominguez et al., 2 Sep 2025).

A bias-variance-diversity decomposition under general losses is available:

$$D_i=[h_1(x_i)^\top,\ldots,h_M(x_i)^\top]^\top\in\mathbb{R}^{d_D}$$3

where the negative sign for diversity quantifies error cancellation across hypotheses. This yields a principled way to trade off accuracy against ensemble diversity.

Further, the PAC-Bayes C-bound for the majority vote links ensemble disagreement (diversity) to generalization error, underscoring s-BFN's relevance for stability and robust uncertainty quantification.

5. Empirical Evaluation and Practical Performance

Empirical validation spans both tabular regression and image classification. On regression benchmarks (Air Quality, Appliances Energy Prediction), s-BFN achieves lowest root mean squared error (RMSE) compared to SVM-RBF, random forest, gradient boosting, and arithmetic combiners. For $$D_i=[h_1(x_i)^\top,\ldots,h_M(x_i)^\top]^\top\in\mathbb{R}^{d_D}$$4 and $$D_i=[h_1(x_i)^\top,\ldots,h_M(x_i)^\top]^\top\in\mathbb{R}^{d_D}$$5, RMSE values of $$D_i=[h_1(x_i)^\top,\ldots,h_M(x_i)^\top]^\top\in\mathbb{R}^{d_D}$$6 and $$D_i=[h_1(x_i)^\top,\ldots,h_M(x_i)^\top]^\top\in\mathbb{R}^{d_D}$$7 were achieved on Air and Energy tasks, exceeding standard ensemble and single-predictor baselines (Dominguez et al., 2 Sep 2025, Dominguez et al., 2023).

For classification (MNIST, CIFAR-10), s-BFN improves accuracy over mean/logit averaging and mixture-of-experts, with the benefit amplified in heterogeneous ensembles and at moderate $$D_i=[h_1(x_i)^\top,\ldots,h_M(x_i)^\top]^\top\in\mathbb{R}^{d_D}$$8. On CIFAR-10, heterogeneous s-BFN ensembles reached accuracies up to $$D_i=[h_1(x_i)^\top,\ldots,h_M(x_i)^\top]^\top\in\mathbb{R}^{d_D}$$9, outperforming base models ($\Phi:\mathbb{R}^{d_D}\to\mathbb{R}^K$0) and other ensemble techniques.

In continuous-depth neural ODEs, s-BFN achieves $\Phi:\mathbb{R}^{d_D}\to\mathbb{R}^K$1 (CIFAR-10) and $\Phi:\mathbb{R}^{d_D}\to\mathbb{R}^K$2 (CIFAR-100) in image classification, matching or exceeding deep ResNet baselines. A posteriori compression via basis projection reduces parameters and inference time by $\Phi:\mathbb{R}^{d_D}\to\mathbb{R}^K$3 with minimal accuracy loss ($\Phi:\mathbb{R}^{d_D}\to\mathbb{R}^K$4 absolute) (Queiruga et al., 2021).

6. Structured Compression and Memory Efficiency

A salient practical property of s-BFN is its support for post-training compression. In the continuous depth context, projection from a high-rank to a lower-rank basis function expansion is performed entirely in parameter space using Gaussian quadrature and normal equations. No additional data or retraining is required; memory and compute are reduced (e.g., $\Phi:\mathbb{R}^{d_D}\to\mathbb{R}^K$5 space savings), while test accuracy drops by less than $\Phi:\mathbb{R}^{d_D}\to\mathbb{R}^K$6. This property is particularly advantageous for memory-constrained deployment (Queiruga et al., 2021).

For s-BFN in the ensemble context, the aggregation and prediction stage costs are dominated by the evaluation of $\Phi:\mathbb{R}^{d_D}\to\mathbb{R}^K$7 base models. The final RBF aggregation is significantly less expensive than full joint training and exhibits low variance across runs.

7. Extensions and Applications

The s-BFN paradigm is instantiated in diverse architectures:

• Continuous-in-depth image classification and transformers for sequence tagging, employing neural ODE-blocks and basis expansions (Queiruga et al., 2021).
• Structured multi-hypothesis regression ensembles with Voronoi partitioning and RBF aggregation for tabular prediction (Dominguez et al., 2023).
• General-purpose, loss-centric, multi-hypothesis deep ensembles with controllable diversity for both regression and classification, incorporating both convex and stochastic training (Dominguez et al., 2 Sep 2025).

A key thread across these applications is the unification of generalized centroid-based aggregation, smooth/structured parameterization, closed-form and iterative learning, and explicit diversity–bias–variance trade-off mechanisms. This positions s-BFN as a robust framework for both predictive modeling and uncertainty quantification across a spectrum of data and model complexities.