Dynamic Boosted Annealing (DBA)

• DBA is a method that uses annealing to control optimization, applied both in compact tree ensembles via RET and in LLM fine-tuning frameworks.
• In the RET formulation, DBA generates a diverse pool of candidate trees through multi-threaded boosting and applies Feature Selection with Annealing (FSA) to jointly optimize leaf weights.
• In LLM fine-tuning, DBA decouples general capability preservation from domain specialization by combining a fixed global gradient with dynamic training correction and annealed learning rates.

Dynamic Boosted Annealing (DBA) denotes two distinct methodological constructs in the literature. In one usage, DBA is a retrospective label applied to the 2017 method "Relevant Ensemble of Trees" (RET), which combines multi-threaded gradient-boosted tree pool generation with Feature Selection with Annealing (FSA) to obtain compact tree ensembles (Dawer et al., 2017). In a later and explicit usage, DBA names a 2025 two-stage fine-tuning framework for LLMs that decouples general capability preservation from domain specialization through a reusable global gradient, gradient boosting during domain training, dynamic training step correction, and annealed learning (Tang et al., 30 Sep 2025). The commonality is therefore not a shared implementation lineage, but the use of annealing as a control mechanism inside a broader optimization procedure.

1. Terminological scope and historical usage

The term DBA is not uniform across the two cited works. In "Generating Compact Tree Ensembles via Annealing" (Dawer et al., 2017), the method is named Relevant Ensemble of Trees and relies on Feature Selection with Annealing. The accompanying technical description explicitly states that what is called DBA there is a mapping onto RET: a dynamic, multi-threaded gradient-boosting pool generation followed by annealing-based subset selection and leaf-weight optimization via FSA. By contrast, "Finetune Once: Decoupling General & Domain Learning with Dynamic Boosted Annealing" introduces DBA as the formal name of an LLM fine-tuning pipeline (Tang et al., 30 Sep 2025).

This terminological divergence is important because the two methods address different hypothesis classes, objectives, and computational regimes. RET operates over additive decision-tree ensembles and enforces sparsity over trees or tree leaves (Dawer et al., 2017). The later DBA operates over neural parameters $\theta$ in full-parameter or adapter-compatible LLM fine-tuning, with optimization guided by a fixed global gradient estimated from general data (Tang et al., 30 Sep 2025).

Usage	Paper	Core mechanism
DBA as mapped term	"Generating Compact Tree Ensembles via Annealing" (Dawer et al., 2017)	Multi-chain boosting pool generation plus FSA-based annealing and reweighting
DBA as formal method name	"Finetune Once: Decoupling General & Domain Learning with Dynamic Boosted Annealing" (Tang et al., 30 Sep 2025)	Global-gradient precomputation, boosted domain gradients, dynamic correction, annealed LR

A common misconception is to treat these as successive versions of the same algorithm. The available evidence does not support that interpretation. The 2017 work is a tree-ensemble method centered on RET and FSA, whereas the 2025 work is an LLM fine-tuning framework with its own mathematical formulation and empirical evaluation.

2. RET as an annealing-based method for compact tree ensembles

In the 2017 formulation, the underlying problem is that boosting can exhibit cyclic behavior in which “the same or a very similar weak learner is selected again and again at later boosting iterations,” and Random Forest is “not based on loss optimization and obtains a more complex and less interpretable model” (Dawer et al., 2017). RET addresses this by generating a large pool of candidate trees and then selecting a small subset while jointly updating leaf weights through loss optimization.

The learning task is formulated through a loss-regularized objective

$L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$

with shrinkage such as

$$s(\boldsymbol{\beta})=\rho \|\boldsymbol{\beta}\|^2.$$

The paper lists square loss for regression,

$\ell(u,y)=(u-y)^2,$

logistic loss for binary classification,

$\ell(u,y)=\log(1+\exp(-uy)),$

and hinge loss,

$\ell(u,y)=\max(1-uy,0).$

The tree-ensemble model is additive:

$$f(\mathbf{x})= \sum_{j=1}^k \boldsymbol{\beta}_j^T \mathbf{i}_j(\mathbf{x}),$$

where each tree is represented as

$$T_j(\mathbf{x})=\boldsymbol{\beta}_j^T\mathbf{i}_j(\mathbf{x}),$$

with $\mathbf{i}_j(\mathbf{x})$ a one-hot index function indicating the reached leaf (Dawer et al., 2017).

The RET pipeline begins with pool generation by GradientBoost under three regimes: Single Chain Single Depth (SCSD), Multi Chain Single Depth (MCSD), and Multi Chain Multi Depth (MCMD). The pool size is typically $M \approx 3000$. In multi-chain variants, random initialization replaces the usual constant bias initialization by base margins sampled i.i.d. from $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$0 per training instance, diversifying gradient and Hessian statistics and thereby producing structurally diverse trees. Because this initialization is random per instance, these trees cannot be used directly; only their structure is retained, and leaf weights must be retrained later by loss minimization (Dawer et al., 2017).

Annealing enters through FSA rather than simulated annealing. The objective is

$L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$1

and FSA alternates a stochastic gradient descent update with deterministic pruning. The core update is

$L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$2

followed by retaining only the $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$3 variables with highest magnitude. The annealing schedule is

$L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$4

with $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$5 used in practice in $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$6 (Dawer et al., 2017).

Two FSA instantiations are described. "FSA on trees" treats complete tree outputs as features,

$L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$7

whereas "FSA on leaves" is the primary approach and optimizes the leaf-weight matrix

$L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$8

with prediction

$L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$9

To ensure whole-tree selection while correcting bias toward deeper trees, the group criterion is

$$s(\boldsymbol{\beta})=\rho \|\boldsymbol{\beta}\|^2.$$0

where $$s(\boldsymbol{\beta})=\rho \|\boldsymbol{\beta}\|^2.$$1 is the number of leaves of tree $$s(\boldsymbol{\beta})=\rho \|\boldsymbol{\beta}\|^2.$$2 (Dawer et al., 2017).

3. Optimization logic, diversification, and relation to boosting

The central rationale of RET is that greedy stage-wise boosting adds one tree at a time conditional on the current ensemble and does not revisit leaf weights of earlier trees. This suggests why cyclic behavior can arise: newly added trees may be correlated with previous trees, and the only way to compensate for earlier leaf weights is to add another similar tree (Dawer et al., 2017). RET replaces that greediness with a two-part mechanism: diversification during pool generation and joint reweighting during annealed selection.

Diversification is achieved by multiple independent boosting threads and, in MCMD, by multiple maximum depths in a prescribed set $$s(\boldsymbol{\beta})=\rho \|\boldsymbol{\beta}\|^2.$$3. Different random base margins across chains create different gradient and Hessian profiles at the first boosting iteration and therefore different split preferences; successive trees inherit these early differences. This suggests a pool less dominated by near-duplicate trees than a single long boosting chain (Dawer et al., 2017).

Joint optimization arises in FSA on leaves, where leaf weights are updated together under the chosen loss. The update remains first-order rather than Newton-style:

$$s(\boldsymbol{\beta})=\rho \|\boldsymbol{\beta}\|^2.$$4

and the shrinkage term contributes the standard $$s(\boldsymbol{\beta})=\rho \|\boldsymbol{\beta}\|^2.$$5 component to the gradient. In logistic loss, gradients are computed at $$s(\boldsymbol{\beta})=\rho \|\boldsymbol{\beta}\|^2.$$6, and leaf weights receive contributions according to the one-hot index functions $$s(\boldsymbol{\beta})=\rho \|\boldsymbol{\beta}\|^2.$$7 (Dawer et al., 2017).

An additional efficiency enhancement is "FSA with multiple sparsity levels," which outputs models at several target sparsities in one run. The paper specifies sparsity levels $$s(\boldsymbol{\beta})=\rho \|\boldsymbol{\beta}\|^2.$$8, computes the iterations $$s(\boldsymbol{\beta})=\rho \|\boldsymbol{\beta}\|^2.$$9 at which the annealing schedule reaches those levels, stores snapshots, prunes to each $\ell(u,y)=(u-y)^2,$0, and then performs additional updates for $\ell(u,y)=(u-y)^2,$1 times on each stored model (Dawer et al., 2017). This mechanism supports cross-validation over ensemble size without restarting optimization from scratch.

The resulting interpretation is not that RET abandons boosting, but that it uses boosting as a structure generator and relocates the decisive loss minimization step to a later annealing stage. A plausible implication is that the method should be understood as post-boosting structure selection plus re-estimation, rather than as a variant of classical stage-wise boosting.

4. DBA as an LLM fine-tuning framework

The 2025 DBA formulation addresses a different optimization problem: preserving general capabilities while specializing an LLM to a domain without repeatedly tuning a general/domain data mixture (Tang et al., 30 Sep 2025). Vanilla mixed-data fine-tuning is written as

$\ell(u,y)=(u-y)^2,$2

with mixed gradient

$\ell(u,y)=(u-y)^2,$3

where $\ell(u,y)=(u-y)^2,$4 and $\ell(u,y)=(u-y)^2,$5.

DBA removes $\ell(u,y)=(u-y)^2,$6 from the training loop by estimating a fixed global gradient from general data in a dedicated pass with learning rate set to zero. In that stage, parameters remain at $\ell(u,y)=(u-y)^2,$7, Adam or AdamW moments are updated, $\ell(u,y)=(u-y)^2,$8 is set close to $\ell(u,y)=(u-y)^2,$9—the appendix uses $\ell(u,y)=\log(1+\exp(-uy)),$0—and the first moment $\ell(u,y)=\log(1+\exp(-uy)),$1 approximates the expected gradient over $\ell(u,y)=\log(1+\exp(-uy)),$2:

$\ell(u,y)=\log(1+\exp(-uy)),$3

No parameter updates or weight decay are applied in this stage, and no gradient clipping is required in the reported experiments (Tang et al., 30 Sep 2025).

To reduce storage, $\ell(u,y)=\log(1+\exp(-uy)),$4 is compressed by SVD and stored as a rank-$\ell(u,y)=\log(1+\exp(-uy)),$5 artifact with $\ell(u,y)=\log(1+\exp(-uy)),$6:

$\ell(u,y)=\log(1+\exp(-uy)),$7

During domain training, DBA reconstructs a low-rank approximation $\ell(u,y)=\log(1+\exp(-uy)),$8 and forms a boosted gradient

$\ell(u,y)=\log(1+\exp(-uy)),$9

with annealed boost magnitude

$\ell(u,y)=\max(1-uy,0).$0

Because $\ell(u,y)=\max(1-uy,0).$1 is fixed, the paper gives the variance relation

$\ell(u,y)=\max(1-uy,0).$2

presenting boosting as a form of gradient regularization and variance reduction (Tang et al., 30 Sep 2025).

DBA also introduces dynamic training step correction through the absolute cosine similarity

$\ell(u,y)=\max(1-uy,0).$3

and correction coefficient

$\ell(u,y)=\max(1-uy,0).$4

with $\ell(u,y)=\max(1-uy,0).$5. Under AdamW, using bias-corrected moments $\ell(u,y)=\max(1-uy,0).$6 and $\ell(u,y)=\max(1-uy,0).$7 computed on $\ell(u,y)=\max(1-uy,0).$8, the update is

$\ell(u,y)=\max(1-uy,0).$9

Larger $$f(\mathbf{x})= \sum_{j=1}^k \boldsymbol{\beta}_j^T \mathbf{i}_j(\mathbf{x}),$$0 increases the denominator and shrinks steps, which the paper interprets as stabilizing updates when domain and general gradients strongly interact (Tang et al., 30 Sep 2025).

The annealing component is an annealed learning rate

$$f(\mathbf{x})= \sum_{j=1}^k \boldsymbol{\beta}_j^T \mathbf{i}_j(\mathbf{x}),$$1

with $$f(\mathbf{x})= \sum_{j=1}^k \boldsymbol{\beta}_j^T \mathbf{i}_j(\mathbf{x}),$$2 in the experiments and $$f(\mathbf{x})= \sum_{j=1}^k \boldsymbol{\beta}_j^T \mathbf{i}_j(\mathbf{x}),$$3 equal to the number of domain steps in one epoch. Combining the components yields the final DBA update

$$f(\mathbf{x})= \sum_{j=1}^k \boldsymbol{\beta}_j^T \mathbf{i}_j(\mathbf{x}),$$4

where $$f(\mathbf{x})= \sum_{j=1}^k \boldsymbol{\beta}_j^T \mathbf{i}_j(\mathbf{x}),$$5 and $$f(\mathbf{x})= \sum_{j=1}^k \boldsymbol{\beta}_j^T \mathbf{i}_j(\mathbf{x}),$$6 are Adam or AdamW bias-corrected moments computed on $$f(\mathbf{x})= \sum_{j=1}^k \boldsymbol{\beta}_j^T \mathbf{i}_j(\mathbf{x}),$$7 (Tang et al., 30 Sep 2025).

5. Empirical performance and computational profile

The empirical record of RET emphasizes compactness and lower loss relative to standard boosting. Across six public datasets, the loss-minimization plots show that RET variants generally achieve lower test loss than GB for the same number of trees, with MCMD dominating in test loss on all but one dataset, madelon (Dawer et al., 2017). In the XOR simulation with $$f(\mathbf{x})= \sum_{j=1}^k \boldsymbol{\beta}_j^T \mathbf{i}_j(\mathbf{x}),$$8 trees of depth $$f(\mathbf{x})= \sum_{j=1}^k \boldsymbol{\beta}_j^T \mathbf{i}_j(\mathbf{x}),$$9 under SCSD, "FSA on leaves" selects one tree. Averaged over $$T_j(\mathbf{x})=\boldsymbol{\beta}_j^T\mathbf{i}_j(\mathbf{x}),$$0 runs, GradientBoost uses $$T_j(\mathbf{x})=\boldsymbol{\beta}_j^T\mathbf{i}_j(\mathbf{x}),$$1 trees with Train AUC $$T_j(\mathbf{x})=\boldsymbol{\beta}_j^T\mathbf{i}_j(\mathbf{x}),$$2 and Test AUC $$T_j(\mathbf{x})=\boldsymbol{\beta}_j^T\mathbf{i}_j(\mathbf{x}),$$3, whereas RET uses $$T_j(\mathbf{x})=\boldsymbol{\beta}_j^T\mathbf{i}_j(\mathbf{x}),$$4 tree with Train AUC $$T_j(\mathbf{x})=\boldsymbol{\beta}_j^T\mathbf{i}_j(\mathbf{x}),$$5 and Test AUC $$T_j(\mathbf{x})=\boldsymbol{\beta}_j^T\mathbf{i}_j(\mathbf{x}),$$6 (Dawer et al., 2017).

On real datasets, averaged over $$T_j(\mathbf{x})=\boldsymbol{\beta}_j^T\mathbf{i}_j(\mathbf{x}),$$7 runs, RET achieves the best reported test performance in all six cases. The classification test errors are: gisette $$T_j(\mathbf{x})=\boldsymbol{\beta}_j^T\mathbf{i}_j(\mathbf{x}),$$8, madelon $$T_j(\mathbf{x})=\boldsymbol{\beta}_j^T\mathbf{i}_j(\mathbf{x}),$$9, miniboone $\mathbf{i}_j(\mathbf{x})$0, and wilt $\mathbf{i}_j(\mathbf{x})$1. The regression test $\mathbf{i}_j(\mathbf{x})$2 values are: abalone $\mathbf{i}_j(\mathbf{x})$3 and online news $\mathbf{i}_j(\mathbf{x})$4 (Dawer et al., 2017). Training times show that RET is more computationally demanding in these settings, although the reported comparison is mixed: for gisette, RET is $\mathbf{i}_j(\mathbf{x})$5 min versus GB $\mathbf{i}_j(\mathbf{x})$6 min and XGB $\mathbf{i}_j(\mathbf{x})$7 min; for miniboone, RET is $\mathbf{i}_j(\mathbf{x})$8 min versus GB $\mathbf{i}_j(\mathbf{x})$9 min and XGB $M \approx 3000$0 min (Dawer et al., 2017).

The LLM-oriented DBA is evaluated across four domains—finance, medicine, law, and News QA—and three base models: Llama3.1-8B, Phi4-14B, and Qwen3-8B (Tang et al., 30 Sep 2025). Metrics are domain score $M \approx 3000$1, general score $M \approx 3000$2 defined as the average normalized change on MMLU, MMLU-Pro, GSM8K, MATH, and M3Exam, and harmonic mean $M \approx 3000$3. Averaged over settings, DBA improves joint performance by $M \approx 3000$4 over vanilla fine-tuning and reduces GPU hours by approximately $M \approx 3000$5, with reported per-domain costs of approximately $M \approx 3000$6 GPU-hours for DBA versus approximately $M \approx 3000$7 GPU-hours for vanilla fine-tuning on $M \approx 3000$8A100 (Tang et al., 30 Sep 2025).

The paper reports several concrete highlights for Qwen3-8B. In finance, DBA achieves $M \approx 3000$9, with $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$00 and $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$01. In medicine, DBA achieves $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$02, with $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$03 and $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$04. In law, DBA achieves $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$05, with $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$06. In News QA, DBA achieves $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$07 versus direct fine-tuning collapsing $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$08 to $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$09; DBA preserves $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$10 while matching $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$11 (Tang et al., 30 Sep 2025). The News QA ablation further reports that AL only yields $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$12 for Llama3.1/Phi4/Qwen3, AL + GGB increases performance, AL + DC yields larger gains with Qwen3 $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$13, and full DBA reaches $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$14 (Tang et al., 30 Sep 2025).

6. Hyperparameters, implementation practice, and limitations

For RET, the key hyperparameters include pool size $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$15 typically $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$16, number of chains $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$17 such as $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$18 chains with $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$19 iterations each, a depth set $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$20 such as $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$21 or $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$22 for madelon, learning rate $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$23 with examples of $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$24 for classification and $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$25 for regression, annealing parameter $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$26 in $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$27, number of FSA iterations $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$28 such as $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$29 or $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$30, target ensemble size $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$31, and L2 shrinkage parameter $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$32 tuned by cross-validation (Dawer et al., 2017). The paper recommends MCMD pool generation, random initialization in multi-chain boosting followed by FSA-based re-optimization of leaf weights, and cross-validation over $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$33 using the multiple-sparsity FSA variant (Dawer et al., 2017).

RET’s main limitation is computational cost. Large pools, storage of index functions $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$34 or equivalent leaf assignments, the leaf-weight matrix $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$35, and cross-validation over $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$36 and $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$37 all contribute to substantial runtime and memory requirements (Dawer et al., 2017). Applicability also depends on differentiable losses because FSA assumes differentiability; although hinge loss is listed, the reported experiments use square and logistic losses (Dawer et al., 2017). The paper further notes that if the pool is insufficiently diverse or the annealing schedule is too aggressive, performance may degrade.

For the LLM formulation, the core defaults are $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$38, $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$39, $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$40, AdamW with $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$41 and $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$42 in domain training, $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$43 and $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$44 in the global-gradient stage, rank $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$45 for SVD compression, batch size $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$46 in both stages, one epoch over $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$47, and one pass over $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$48 (Tang et al., 30 Sep 2025). The paper states that performance increases up to approximately $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$49 and then saturates, recommending $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$50 (Tang et al., 30 Sep 2025).

The main failure modes are explicitly tied to gradient misalignment and data quality. For far-OOD domains with very small $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$51, overly strong boosting can slow specialization; mitigation consists of reducing $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$52 or recomputing $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$53 on broader general data (Tang et al., 30 Sep 2025). If $L(\boldsymbol{\beta})=\sum_{i=1}^N \ell(f_{\boldsymbol{\beta}(\mathbf{x_i}),y_i)+s(\boldsymbol{\beta}),$54 is too narrow or noisy, the anchor may regularize toward undesirable directions. A frequent misconception is that DBA eliminates the need for general data altogether. The 2025 method does not do that; it removes general data from the domain annealing loop, but still requires a one-time global-gradient precomputation on general data (Tang et al., 30 Sep 2025).

Taken together, the two uses of DBA show annealing deployed in sharply different roles. In RET, annealing controls sparsity over tree structures and enables joint leaf-weight optimization for compact ensembles (Dawer et al., 2017). In the LLM framework, annealing controls learning-rate decay and the fading influence of a fixed global gradient during domain specialization (Tang et al., 30 Sep 2025). The shared terminology therefore reflects an optimization motif rather than a single unified algorithmic family.