Super-Ensemble Optimization Overview

• Super-Ensemble Optimization is a design principle that treats ensemble construction as an optimization variable by dynamically tuning weights, hyperparameters, and fusion rules.
• Methods range from convex stacking and deep meta-learning to adaptive operator allocation and fine-grained confidence weighting, enhancing ensemble diversity and performance.
• Applications include supervised prediction, hyperparameter search, evolutionary optimization, and specialized tasks like image super-resolution and mammographic retrieval.

Searching arXiv for the cited papers and nearby work on ensemble optimization to ground the article. Super-Ensemble Optimization denotes a family of methods in which the ensemble itself is the optimization object: rather than fixing aggregation a priori, these methods optimize ensemble weights, meta-functions, operator-allocation probabilities, feature-subset distributions, sparse support patterns, or output-fusion coefficients. In the literature, the term covers convex super learning for prediction, hierarchical “deep” super learners, probabilistic multi-method evolutionary search, model-based hyperparameter search that returns ensembles, fine-grained confidence weighting, and training-free output-level fusion (Lacoste et al., 2014, Shahhosseini et al., 2019, Pérez-Aracil et al., 2022, Yuan et al., 2024, Chang et al., 13 Apr 2026).

1. Scope and conceptual definition

Across the cited works, Super-Ensemble Optimization is not a single algorithmic template but a recurrent design principle: ensemble performance is improved by optimizing the mechanism that allocates influence among constituent learners, search operators, or representation branches. In supervised prediction, this often appears as convex stacking or meta-learning over out-of-fold predictions. In black-box optimization, it appears as adaptive control of operator usage or as hyperparameter search procedures that produce ensembles rather than a single selected model. In sparse modeling and feature-subspace methods, it appears as joint optimization over multiple supports or over distributions on random subsets. In output-space fusion, it can reduce to a constrained search over a small number of combination parameters (Young et al., 2018, Huynh-Thu et al., 2021, Christidis et al., 2022, Rahman et al., 6 Aug 2025).

Historically, one early formulation integrated ensemble construction directly into sequential model-based optimization. “Sequential Model-Based Ensemble Optimization” replaced the usual goal of selecting a single best hyperparameter configuration with an Agnostic Bayesian ensemble constructed through bootstrap-based posterior sampling over the best learner (Lacoste et al., 2014). Later work broadened the idea in several directions: hierarchical super learners for classification (Young et al., 2018), bi-level optimization of ensemble weights and base-model hyperparameters for regression (Shahhosseini et al., 2019), adaptive operator mixtures in evolutionary search (Pérez-Aracil et al., 2022), and learnable confidence structures that assign class-specific trust to individual ensemble members (Yuan et al., 2024, Yuan et al., 2024).

Taken together, these works suggest that Super-Ensemble Optimization is best understood functionally: it refers to methods that optimize not only the members of an ensemble, but also the rules by which ensemble diversity is generated, weighted, constrained, or exploited.

2. Canonical optimization formulations

A central feature of the area is the diversity of optimization variables. Some methods optimize simplex-constrained weights over fixed base predictions, some optimize hyperparameters and weights jointly, some optimize probabilities over operators or features, and some optimize structured matrices or tensors encoding class-conditional trust.

Family	Representative formulation	Optimized quantity
Convex stacking	$\hat f(x)=\sum_m w_m f_m(x)$	Nonnegative weights with $\sum_m w_m=1$
Deep stacking	$$\hat f^{(\ell)}(x)=\sum_m w_m^{(\ell)} f_m^{(\ell)}(x)$$	Layer-specific weights and stopping depth
Joint weight–hyperparameter tuning	$$\min_{w,\theta}\frac{1}{N}\sum_i \bigl(y_i-\sum_j w_j f_j(x_i;\theta_j)\bigr)^2$$	Ensemble weights and base hyperparameters
Operator-allocation optimization	$$p_i(t+1)=\exp(S_i(t)/\tau)\big/\sum_j \exp(S_j(t)/\tau)$$	Time-varying probabilities over search operators
Parametric random subspace	$z^{(t)}\sim \mathrm{Bern}(p)$	Feature-selection probabilities $p_j$
Sparse multi-model ensembles	$$\sum_g\|y-X\beta^{(g)}\|_2^2+\lambda_s\sum_g\|\beta^{(g)}\|_0+\lambda_d\sum_{g<h}\sum_j \mathbf 1\{\beta_j^{(g)}\neq0,\beta_j^{(h)}\neq0\}$$	Multiple sparse supports and their overlap
Confidence-weighted fusion	$L(\Theta)=\mathcal C-\gamma\mathcal M$ or $L(C)$ with margin terms	Class-specific confidence tensor or matrix
Output-space fusion	$\sum_m w_m=1$0	Scalar fusion weight

In convex stacking, the standard objective is empirical risk minimization over out-of-fold predictions under simplex constraints. Deep Super Learner minimizes multiclass log-loss at each layer, while daily streamflow super learning minimizes a cross-validated squared-error criterion over nonnegative weights summing to one (Young et al., 2018, Tyralis et al., 2019). GEM-ITH extends this into a bi-level setting by nesting convex weight optimization inside an outer search over hyperparameter tuples (Shahhosseini et al., 2019).

Adaptive search ensembles use probabilities as control variables. In DPCRO-SL, operator-selection probabilities are updated from performance statistics through a Boltzmann rule, with a minimum $\sum_m w_m=1$1 to prevent permanent starvation of any operator (Pérez-Aracil et al., 2022). Parametric Random Subspace instead optimizes a Bernoulli vector $\sum_m w_m=1$2 governing feature inclusion, using score-function gradients, baselines for variance reduction, and importance sampling to reuse pre-trained base models (Huynh-Thu et al., 2021).

Structured confidence methods optimize richer objects than scalar weights. The tensor-based ensemble method introduced a confidence tensor $\sum_m w_m=1$3 describing how confidently the $\sum_m w_m=1$4-th base classifier predicts class $\sum_m w_m=1$5 when the true class is $\sum_m w_m=1$6, then optimized a smooth, convex margin-based objective in the unfolded parameter $\sum_m w_m=1$7 (Yuan et al., 2024). A related fine-grained method optimized a learnable confidence matrix $\sum_m w_m=1$8 together with a log-sum-exp surrogate for the margin (Yuan et al., 2024).

3. Hierarchical stacking and meta-learning

The most direct line of work treats super-ensemble optimization as an extension of stacking. Deep Super Learner (DSL) defines a layer-$\sum_m w_m=1$9 ensemble prediction

$$\hat f^{(\ell)}(x)=\sum_m w_m^{(\ell)} f_m^{(\ell)}(x)$$0

with the next layer receiving the original features concatenated with the ensemble probabilities from the previous layer. At each layer, the weights solve a convex multiclass log-loss minimization over out-of-fold predictions, and layers are added until validation loss ceases to decrease. Using five base learners—logistic regression, $$\hat f^{(\ell)}(x)=\sum_m w_m^{(\ell)} f_m^{(\ell)}(x)$$1-NN$$\hat f^{(\ell)}(x)=\sum_m w_m^{(\ell)} f_m^{(\ell)}(x)$$2, random forest, extra-trees, and XGBoost—the method converged in $$\hat f^{(\ell)}(x)=\sum_m w_m^{(\ell)} f_m^{(\ell)}(x)$$3–$$\hat f^{(\ell)}(x)=\sum_m w_m^{(\ell)} f_m^{(\ell)}(x)$$4 layers on IMDB and MNIST. On IMDB it achieved log-loss $$\hat f^{(\ell)}(x)=\sum_m w_m^{(\ell)} f_m^{(\ell)}(x)$$5 and accuracy $$\hat f^{(\ell)}(x)=\sum_m w_m^{(\ell)} f_m^{(\ell)}(x)$$6; on MNIST it achieved log-loss $$\hat f^{(\ell)}(x)=\sum_m w_m^{(\ell)} f_m^{(\ell)}(x)$$7 and accuracy $$\hat f^{(\ell)}(x)=\sum_m w_m^{(\ell)} f_m^{(\ell)}(x)$$8. In both domains it outperformed the one-layer super learner and the MLP, and on text data it also outperformed the CNN (Young et al., 2018).

Large-scale streamflow forecasting provides a classical super-learning formulation at application scale. With $$\hat f^{(\ell)}(x)=\sum_m w_m^{(\ell)} f_m^{(\ell)}(x)$$9 machine-learning algorithms, $$\min_{w,\theta}\frac{1}{N}\sum_i \bigl(y_i-\sum_j w_j f_j(x_i;\theta_j)\bigr)^2$$0-fold CV on the training period, and one-step-ahead forecasting over $$\min_{w,\theta}\frac{1}{N}\sum_i \bigl(y_i-\sum_j w_j f_j(x_i;\theta_j)\bigr)^2$$1 basins, the super learner improved over linear regression by $$\min_{w,\theta}\frac{1}{N}\sum_i \bigl(y_i-\sum_j w_j f_j(x_i;\theta_j)\bigr)^2$$2, while the equal-weight combiner improved by $$\min_{w,\theta}\frac{1}{N}\sum_i \bigl(y_i-\sum_j w_j f_j(x_i;\theta_j)\bigr)^2$$3. The best individual learner was neural networks at $$\min_{w,\theta}\frac{1}{N}\sum_i \bigl(y_i-\sum_j w_j f_j(x_i;\theta_j)\bigr)^2$$4, followed by extremely randomized trees at $$\min_{w,\theta}\frac{1}{N}\sum_i \bigl(y_i-\sum_j w_j f_j(x_i;\theta_j)\bigr)^2$$5 and XGBoost at $$\min_{w,\theta}\frac{1}{N}\sum_i \bigl(y_i-\sum_j w_j f_j(x_i;\theta_j)\bigr)^2$$6. Weight distributions adapted across basins, and poor learners such as lasso and SVR often received zero weight under the nonnegative simplex constraint (Tyralis et al., 2019).

GEM-ITH generalizes weighted stacking by coupling the combination weights to base-learner hyperparameters. For fixed hyperparameters, the inner problem is a convex quadratic program over out-of-bag predictions. The outer loop searches over joint hyperparameter tuples, with Bayesian search used to restrict the candidate set. In the reported setup, four base learners were selected by a diversity heuristic, $$\min_{w,\theta}\frac{1}{N}\sum_i \bigl(y_i-\sum_j w_j f_j(x_i;\theta_j)\bigr)^2$$7 Bayesian-selected settings were used per model, and $$\min_{w,\theta}\frac{1}{N}\sum_i \bigl(y_i-\sum_j w_j f_j(x_i;\theta_j)\bigr)^2$$8 combinations were evaluated. GEM-ITH achieved the lowest test MSE in $$\min_{w,\theta}\frac{1}{N}\sum_i \bigl(y_i-\sum_j w_j f_j(x_i;\theta_j)\bigr)^2$$9 of $$p_i(t+1)=\exp(S_i(t)/\tau)\big/\sum_j \exp(S_j(t)/\tau)$$0 public regression datasets, with relative improvements over GEM ranging from $$p_i(t+1)=\exp(S_i(t)/\tau)\big/\sum_j \exp(S_j(t)/\tau)$$1 to $$p_i(t+1)=\exp(S_i(t)/\tau)\big/\sum_j \exp(S_j(t)/\tau)$$2 (Shahhosseini et al., 2019).

Meta-HAL pushes the meta-learning viewpoint further by replacing linear or simplex aggregation with a highly adaptive cadlag meta-function $$p_i(t+1)=\exp(S_i(t)/\tau)\big/\sum_j \exp(S_j(t)/\tau)$$3 having bounded sectional variation norm. The Meta Highly Adaptive Lasso Minimum Loss Estimator minimizes cross-validated empirical risk over this class, and the final M-HAL super-learner averages the resulting fold-specific fitted functions. Its excess risk converges to the oracle ensemble at rate $$p_i(t+1)=\exp(S_i(t)/\tau)\big/\sum_j \exp(S_j(t)/\tau)$$4 up to a $$p_i(t+1)=\exp(S_i(t)/\tau)\big/\sum_j \exp(S_j(t)/\tau)$$5 factor, and under weak conditions the undersmoothed estimator yields asymptotically linear target-feature estimators with an efficient or potentially super-efficient influence curve (Wang et al., 2023).

4. Adaptive operator allocation and automated search

In optimization rather than prediction, Super-Ensemble Optimization often means adaptive allocation of search effort across multiple procedures. PCRO-SL and DPCRO-SL reformulate Coral Reefs Optimization with Substrate Layers by replacing fixed substrate zones with per-individual tags. A coral carries a tag $$p_i(t+1)=\exp(S_i(t)/\tau)\big/\sum_j \exp(S_j(t)/\tau)$$6, indicating which operator acts during broadcast spawning. In PCRO-SL the assignment is uniform, $$p_i(t+1)=\exp(S_i(t)/\tau)\big/\sum_j \exp(S_j(t)/\tau)$$7. In DPCRO-SL the probabilities are updated from substrate performance statistics through

$$p_i(t+1)=\exp(S_i(t)/\tau)\big/\sum_j \exp(S_j(t)/\tau)$$8

followed by an $$p_i(t+1)=\exp(S_i(t)/\tau)\big/\sum_j \exp(S_j(t)/\tau)$$9 floor and renormalization. The intended effect is an exploitation–exploration balance in which productive operators are reinforced without completely eliminating the rest (Pérez-Aracil et al., 2022).

Empirically, the adaptive scheme was tested on $z^{(t)}\sim \mathrm{Bern}(p)$0 classic continuous benchmark functions with $z^{(t)}\sim \mathrm{Bern}(p)$1 fitness evaluations and $z^{(t)}\sim \mathrm{Bern}(p)$2 runs each, and on wind-farm layout optimization. PCRO-SL gave uniformly better means and best-found values than vanilla CRO-SL on almost all benchmark functions, while DPCRO-SL improved best and mean values by orders of magnitude on hard functions such as Rosenbrock F5 and multimodal F9. In the NREL/IEA Task 37 wind-farm case with $z^{(t)}\sim \mathrm{Bern}(p)$3 turbines and radius $z^{(t)}\sim \mathrm{Bern}(p)$4 m, DPCRO-SL achieved best AEP $z^{(t)}\sim \mathrm{Bern}(p)$5 MWh and ranked $z^{(t)}\sim \mathrm{Bern}(p)$6 in the official AIAA 2019 competition report. Performance differences passed nonparametric tests at $z^{(t)}\sim \mathrm{Bern}(p)$7, and the dynamic ensemble reduced standard deviation of results (Pérez-Aracil et al., 2022).

“Sequential Model-Based Ensemble Optimization” integrates ensemble construction into Bayesian hyperparameter search. A Gaussian-process surrogate models validation risk over hyperparameters, and Expected Improvement drives proposals. Instead of returning only the single best configuration, the method uses an Agnostic Bayesian bootstrap approximation over the validation set to sample from the posterior over the best learner, thereby producing an ensemble. On $z^{(t)}\sim \mathrm{Bern}(p)$8 regression datasets, ESMBO attained expected rank $z^{(t)}\sim \mathrm{Bern}(p)$9 versus $p_j$0 for ERS, $p_j$1 for SMBO, and $p_j$2 for random search. On $p_j$3 classification datasets, it attained expected rank $p_j$4 versus $p_j$5 for ERS and $p_j$6 for SMBO, and over all $p_j$7 datasets it was never significantly outperformed (Lacoste et al., 2014).

Optimization over ensembles can also invert the usual direction of the problem: instead of learning an ensemble for prediction, one optimizes a decision vector against an ensemble-valued surrogate objective. For an ensemble of $p_j$8 ReLU networks, the objective is

$p_j$9

embedded in a mixed-integer linear program using standard big-$$\sum_g\|y-X\beta^{(g)}\|_2^2+\lambda_s\sum_g\|\beta^{(g)}\|_0+\lambda_d\sum_{g<h}\sum_j \mathbf 1\{\beta_j^{(g)}\neq0,\beta_j^{(h)}\neq0\}$$0 ReLU constraints. The proposed two-phase method combines targeted bound tightening on critical neurons, Benders-type cuts, and a Lagrangian-relaxation-based branch-and-bound. Over approximately $$\sum_g\|y-X\beta^{(g)}\|_2^2+\lambda_s\sum_g\|\beta^{(g)}\|_0+\lambda_d\sum_{g<h}\sum_j \mathbf 1\{\beta_j^{(g)}\neq0,\beta_j^{(h)}\neq0\}$$1 instances with a $$\sum_g\|y-X\beta^{(g)}\|_2^2+\lambda_s\sum_g\|\beta^{(g)}\|_0+\lambda_d\sum_{g<h}\sum_j \mathbf 1\{\beta_j^{(g)}\neq0,\beta_j^{(h)}\neq0\}$$2 s limit, the direct big-$$\sum_g\|y-X\beta^{(g)}\|_2^2+\lambda_s\sum_g\|\beta^{(g)}\|_0+\lambda_d\sum_{g<h}\sum_j \mathbf 1\{\beta_j^{(g)}\neq0,\beta_j^{(h)}\neq0\}$$3 baseline solved approximately $$\sum_g\|y-X\beta^{(g)}\|_2^2+\lambda_s\sum_g\|\beta^{(g)}\|_0+\lambda_d\sum_{g<h}\sum_j \mathbf 1\{\beta_j^{(g)}\neq0,\beta_j^{(h)}\neq0\}$$4 to optimality, whereas the proposed E-NN solved approximately $$\sum_g\|y-X\beta^{(g)}\|_2^2+\lambda_s\sum_g\|\beta^{(g)}\|_0+\lambda_d\sum_{g<h}\sum_j \mathbf 1\{\beta_j^{(g)}\neq0,\beta_j^{(h)}\neq0\}$$5; average solve times on solved instances were approximately $$\sum_g\|y-X\beta^{(g)}\|_2^2+\lambda_s\sum_g\|\beta^{(g)}\|_0+\lambda_d\sum_{g<h}\sum_j \mathbf 1\{\beta_j^{(g)}\neq0,\beta_j^{(h)}\neq0\}$$6 s and approximately $$\sum_g\|y-X\beta^{(g)}\|_2^2+\lambda_s\sum_g\|\beta^{(g)}\|_0+\lambda_d\sum_{g<h}\sum_j \mathbf 1\{\beta_j^{(g)}\neq0,\beta_j^{(h)}\neq0\}$$7 s, respectively (Wang et al., 2021).

5. Fine-grained confidence, feature-subspace, and sparse-structure methods

A major branch of the literature treats super-ensemble optimization as learning where diversity should occur. Parametric Random Subspace introduces a Bernoulli feature-selection vector $$\sum_g\|y-X\beta^{(g)}\|_2^2+\lambda_s\sum_g\|\beta^{(g)}\|_0+\lambda_d\sum_{g<h}\sum_j \mathbf 1\{\beta_j^{(g)}\neq0,\beta_j^{(h)}\neq0\}$$8, samples masks $$\sum_g\|y-X\beta^{(g)}\|_2^2+\lambda_s\sum_g\|\beta^{(g)}\|_0+\lambda_d\sum_{g<h}\sum_j \mathbf 1\{\beta_j^{(g)}\neq0,\beta_j^{(h)}\neq0\}$$9, and optimizes the expected ensemble loss plus differentiable regularization by gradient descent. Importance sampling allows reuse of already trained base models until the effective sample size

$L(\Theta)=\mathcal C-\gamma\mathcal M$0

drops below a threshold such as $L(\Theta)=\mathcal C-\gamma\mathcal M$1. The learned $L(\Theta)=\mathcal C-\gamma\mathcal M$2 values are interpretable as feature-importance scores. On simulated data, PRS outperformed standard Random Subspace, Random Forest, and GB-Trees, especially with kNN or SVM bases; on $L(\Theta)=\mathcal C-\gamma\mathcal M$3 tabular datasets it improved over RS and RaSE in most settings; on MNIST $L(\Theta)=\mathcal C-\gamma\mathcal M$4 vs. $L(\Theta)=\mathcal C-\gamma\mathcal M$5 with fused-lasso regularization, PRS-tree and GBDT achieved approximately $L(\Theta)=\mathcal C-\gamma\mathcal M$6 accuracy (Huynh-Thu et al., 2021).

Fine-grained class-specific weighting appears in two related proposals. The tensor-optimization-powered ensemble method defines a confidence tensor $L(\Theta)=\mathcal C-\gamma\mathcal M$7, where $L(\Theta)=\mathcal C-\gamma\mathcal M$8 is the probability that learner $L(\Theta)=\mathcal C-\gamma\mathcal M$9 predicts class $L(C)$0 when the true class is $L(C)$1, and optimizes a loss $L(C)$2 combining cross-entropy with a smooth large-margin term. A key proposition states that every column of $L(C)$3 sums to zero, so gradient descent preserves the linear constraint $L(C)$4. On $L(C)$5 real datasets, the method with only $L(C)$6 trees tied or beat RF100 on $L(C)$7 and, on the toy two-ring data, reached $L(C)$8 versus $L(C)$9 for RF100 (Yuan et al., 2024). The “Margin-Maximizing Fine-Grained Ensemble Method” uses a learnable confidence matrix $\sum_m w_m=1$00 and proves Lipschitz continuity of the loss with constant $\sum_m w_m=1$01. In experiments on a moon-toy dataset and eight real-world benchmarks, it used only $\sum_m w_m=1$02 decision trees yet improved test accuracies by up to $\sum_m w_m=1$03–$\sum_m w_m=1$04 points over RF100 (Yuan et al., 2024).

Sparse, interpretable super-ensembles optimize not only combination but also support diversity. Multi-Model Subset Selection (BSpS) learns $\sum_m w_m=1$05 sparse linear models jointly with an $\sum_m w_m=1$06 sparsity control per model and an overlap control across models. The bias–variance–covariance decomposition is explicit: $\sum_m w_m=1$07 which motivates separate control of within-model sparsity and between-model overlap. The Projected Subsets Gradient Descent algorithm updates one model at a time and projects onto the top-$\sum_m w_m=1$08 entries consistent with the overlap limit $\sum_m w_m=1$09. In simulation studies, Fast-BSpS achieved the best MSPE rank at $\sum_m w_m=1$10, with recall rank $\sum_m w_m=1$11 and precision rank $\sum_m w_m=1$12. In the BBS gene-expression example, Fast-BSpS attained ensemble MSPE $\sum_m w_m=1$13 with only $\sum_m w_m=1$14 interpretable models (Christidis et al., 2022).

Random Subset Averaging (RSA) uses binomial random subsets and a two-round Mallows weighting scheme. Within each group, $\sum_m w_m=1$15 random-subset predictions are aggregated by convex weights; then the $\sum_m w_m=1$16 group-level predictors are aggregated again. Under general conditions, RSA is asymptotically optimal relative to the best convex combination of group predictors, and under orthogonal design it admits explicit finite-sample risk bounds. In simulations, RSA achieved the lowest or joint lowest MSFE in almost all scenarios, especially under high correlation or moderate density. In financial return forecasting with $\sum_m w_m=1$17 PCA-extracted factors, it attained the lowest MSFE across all $\sum_m w_m=1$18 horizons in both pre- and post-crisis subperiods (Cui et al., 27 Dec 2025).

For imbalanced binary classification, the superensemble classifier based on HDDT and RBFN is structurally simpler but conceptually aligned. HDDT supplies Hellinger-distance-based feature selection and an imbalance-insensitive class estimate; these selected features plus the HDDT label are then fed into a one-hidden-layer RBFN whose parameters minimize empirical risk. Under the stated HDDT and RBFN regularity conditions, the combined procedure is distribution-free and consistent, and no resampling is needed (Chakraborty et al., 2018).

6. Applications, empirical patterns, and limitations

Super-Ensemble Optimization is also used in application-specific systems where the optimization variable is deliberately minimal. In BIRADS-based mammographic image retrieval, the super-ensemble stage is a selective feature-fusion step built on fine-tuned DenseNet121 and ResNet50 embeddings. The optimized subset is chosen by exhaustive evaluation of $\sum_m w_m=1$19 candidate combinations under the constraint $\sum_m w_m=1$20 ms, and the selected pair is concatenated into a $\sum_m w_m=1$21-dimensional super-feature indexed by FAISS FlatL2. The resulting system achieved $\sum_m w_m=1$22 with $\sum_m w_m=1$23 CI $\sum_m w_m=1$24, an average query latency of $\sum_m w_m=1$25 ms, and a highly significant improvement with $\sum_m w_m=1$26 and Cohen’s $\sum_m w_m=1$27 (Rahman et al., 6 Aug 2025).

A different low-parameter formulation appears in single-image super-resolution. The training-free dual-branch system combines a Hybrid attention network with TLC inference and a MambaIRv2 branch with geometric self-ensemble using $\sum_m w_m=1$28 transforms. Fusion is performed directly in image space: $\sum_m w_m=1$29 On $\sum_m w_m=1$30 DIV2K $\sum_m w_m=1$31 images, the best PSNR occurred at $\sum_m w_m=1$32, yielding PSNR/SSIM $\sum_m w_m=1$33 versus $\sum_m w_m=1$34 for the base branch and $\sum_m w_m=1$35 for the strong branch. The method requires approximately $\sum_m w_m=1$36 the cost of a single branch, but no retraining and no extra trainable parameters (Chang et al., 13 Apr 2026).

Several empirical regularities recur across the literature. First, gains often come from optimized complementarity rather than from maximizing the raw number of models. This is explicit in the confidence-tensor and fine-grained confidence-matrix methods, which report performance beyond much larger random forests using only one-tenth as many trees, and in mammographic retrieval, where the selected pair outperformed larger fusion alternatives such as mega-all or weighted averaging (Yuan et al., 2024, Yuan et al., 2024, Rahman et al., 6 Aug 2025). Second, many methods use constraints to stabilize optimization: simplex constraints in stacking, $\sum_m w_m=1$37 floors in adaptive operator allocation, overlap constraints in sparse multi-model ensembles, and linear equality constraints preserved by zero-sum gradients in tensor-based methods (Young et al., 2018, Pérez-Aracil et al., 2022, Christidis et al., 2022).

A common misconception is to equate the topic with learned convex averaging alone. The cited literature includes cases in which no gradient-based meta-weight learning is performed at the final stage. In the mammographic retrieval framework, the “optimization” is subset selection under a latency constraint rather than continuous weight fitting, and in the super-resolution framework the ensemble is training-free and controlled by a single validation-tuned scalar $\sum_m w_m=1$38 (Rahman et al., 6 Aug 2025, Chang et al., 13 Apr 2026). Conversely, some methods optimize much richer objects than weights, including hyperparameter tuples, cadlag meta-functions, Bernoulli feature-selection distributions, or $\sum_m w_m=1$39-constrained support matrices (Shahhosseini et al., 2019, Wang et al., 2023, Huynh-Thu et al., 2021, Christidis et al., 2022).

The limitations are correspondingly varied. Some methods have high search cost: GEM-ITH requires $\sum_m w_m=1$40 model fits and evaluates $\sum_m w_m=1$41 hyperparameter combinations in the reported setup; PRS periodically retrains ensembles when importance weights degenerate; RSA requires cross-validation over $\sum_m w_m=1$42 and can be costly when $\sum_m w_m=1$43 and $\sum_m w_m=1$44 are huge (Shahhosseini et al., 2019, Huynh-Thu et al., 2021, Cui et al., 27 Dec 2025). Some methods explicitly acknowledge incomplete theory: DPCRO-SL provides no formal convergence proof, and exact $\sum_m w_m=1$45 BSpS guarantees remain an area of ongoing research (Pérez-Aracil et al., 2022, Christidis et al., 2022). Others note task dependence: DSL may further improve with task-specific tuning, and in very small samples or extremely sparse targets, simple variable selection may outperform RSA (Young et al., 2018, Cui et al., 27 Dec 2025).

In this sense, Super-Ensemble Optimization is less a single method than a general optimization doctrine for ensembles: diversity is not merely generated, but explicitly parameterized, constrained, selected, or learned.