BoostTransformer: Enhancing Transformer Models

• The paper introduces BoostTransformer, which integrates gradient boosting with Transformer architectures using sequential additive fitting for robust learning.
• It employs a novel subgrid token selection and importance-weighted sampling strategy to reduce computational cost and prevent overfitting.
• Empirical results show up to 36% FLOPs reduction and significant accuracy gains on NLP benchmarks compared to standard Transformer models.

BoostTransformer is a framework that incorporates boosting principles directly into the training of Transformer models for NLP. It achieves this by combining the sequential learning guarantees of gradient boosting with modern self-attention architectures. BoostTransformer tightly integrates weak Transformer learners via sequential additive fitting, augmented by subgrid (subsequence) token selection and importance-weighted sampling to achieve computational efficiency, robust generalization—particularly under data scarcity—and stable convergence behaviors (Fang et al., 4 Aug 2025).

1. Architectural Foundation and Motivation

BoostTransformer extends classical gradient boosting—where weak learners are combined sequentially over pseudo-residuals—by using Transformer encoder architectures as the weak learners. Let $f_t(x)=\sum_{i=1}^t \nu\,g_i(x)$ denote the ensemble prediction at boosting round $t$, with shrinkage parameter $\nu\in(0,1]$. Each $g_i$ is a Transformer model producing $M$-dimensional logits (one per class).

Pre-trained models such as BERT or RoBERTa provide strong feature learning capabilities but are limited by overfitting, expensive fine-tuning, and hyperparameter search when data are scarce or classes are imbalanced. BoostTransformer addresses these limitations by focusing subsequent ensemble members on mistakes (hard examples and tokens), and by aggressively reducing both sequence and dataset size per iteration through its subgrid and importance sampling mechanisms (Fang et al., 4 Aug 2025).

2. Least-Squares Boosting Objective

BoostTransformer replaces standard cross-entropy minimization with a functional gradient boosting approach, leveraging a multiclass exponential loss:

$$R[f]=\frac{1}{N}\sum_{i=1}^N L(z_i, f(x_i)),\quad L(z, f)=\log\left(\sum_{k=1}^M e^{f_k}-e^{f_z}\right).$$

Pseudo-residuals are computed as

$$w_i^{(t)} = -\nabla_{f}L(z_i, f_{t-1}(x_i)) \in \mathbb{R}^M,$$

and the next Transformer learner is fit by

$$g_t = \arg\min_{g\in\mathcal{G}} \sum_{i=1}^N \|g(x_i) - w_i^{(t)}\|_2^2.$$

The optimal step-size $\alpha_t$ for each round is found via line search:

$$\alpha_t = \arg\min_\alpha R[f_{t-1}+\alpha\,g_t] = \arg\min_{\alpha} \frac{1}{N}\sum_{i} L(z_i,\,f_{t-1}(x_i)+\alpha\,g_t(x_i)).$$

The updated ensemble becomes $t$0. Empirically, using a squared error surrogate for the fit yields stable and fast convergence, comparable to standard boosting fits but with improved numerical behavior (Fang et al., 4 Aug 2025).

3. Subgrid Token Selection Mechanism

BoostTransformer addresses the quadratic computational complexity of self-attention in sequence length $t$1 by pruning "unimportant" tokens at every boosting round using attention signals from the previous round's encoder. Token importance combines self-importance and relational importance:

• Self-importance for token $t$2 at position $t$3 in $t$4:

$t$5

• Relational importance:

$t$6

with $t$7.

The total importance for word $t$8 across the dataset is $t$9. At each round, only the top-$\nu\in(0,1]$0 fraction (e.g., $\nu\in(0,1]$1) of the vocabulary by total importance is retained, yielding truncated inputs for subsequent learners. This not only speeds up each boosting round but also reduces overfitting by eliminating contextually irrelevant tokens (Fang et al., 4 Aug 2025).

4. Importance-Weighted Example Sampling

BoostTransformer further reduces compute and overfitting by importance-sampling training examples at each boosting round. The sampling distribution $\nu\in(0,1]$2 is proportional to the norm of current residuals:

$\nu\in(0,1]$3

Each weak learner fits its residuals using the unbiased sub-sampled loss:

$\nu\in(0,1]$4

where $\nu\in(0,1]$5 (e.g., $\nu\in(0,1]$6). This sampling focuses computational resources on the "harder" or poorly fit examples, maximizing the risk reduction per computational step. Theoretical justification (variance reduction optimality) is detailed in the primary reference (Fang et al., 4 Aug 2025).

5. Unified Algorithm, Training, and Complexity

The integrated BoostTransformer pipeline is as follows:

1. For $\nu\in(0,1]$7 to $\nu\in(0,1]$8:
   • Compute pseudo-residuals.
   • Compute sampling weights $\nu\in(0,1]$9; draw sample $g_i$0 of size $g_i$1.
   • Compute token importances, retain top $g_i$2 fraction by $g_i$3; truncate each $g_i$4.
   • Fit $g_i$5 to minimize the weighted sub-sampled squared loss on truncated sequences.
   • Line-search for $g_i$6.
   • Update ensemble: $g_i$7.

Standard self-attention costs $g_i$8 per sequence. Subgrid pruning reduces the length to $g_i$9, yielding up to a $M$0 speedup per round, and when combined with importance sampling ($M$1), achieves $M$236% FLOPs reduction per iteration and $M$350% overall training time reduction compared to a vanilla Transformer (Fang et al., 4 Aug 2025).

6. Empirical Performance and Comparative Analysis

BoostTransformer was evaluated on IMDB, Yelp polarity, and Amazon polarity datasets. Baselines included standard Transformers, random token-pruned Transformers, and ablated BoostTransformer variants. Results:

• Combined BoostTransformer variants outperform the standard Transformer by +0.87% (IMDB), +0.55% (Yelp), +0.79% (Amazon) in accuracy.
• Subgrid BoostTransformer converges in approximately two-thirds of the wall-clock time, with comparable or higher accuracy.
• Importance-Sampling BoostTransformer halves the training time and is particularly advantageous for small, overfitting-prone datasets.
• Convergence analyses show boosted models surpass vanilla Transformers after 2–3 epochs and remain stable, while baseline Transformers often deteriorate due to overfitting.
• Ablations demonstrate that subgrid pruning mainly accelerates training with modest accuracy gain, while importance sampling significantly stabilizes training and improves results under data scarcity (Fang et al., 4 Aug 2025).

7. Practical Hyperparameters and Implementation Guidance

Key hyperparameters for BoostTransformer include: ensemble size $M$4, shrinkage $M$5, subgrid fraction $M$6, importance sample fraction $M$7. The recommended optimizer is AdamW (learning rate $M$8, weight decay 0.01, batch size 16), trained for 5 epochs per weak learner, with a linear decay learning rate schedule and warmup ratio $M$9. The base architecture is a 6-layer RoBERTa encoder, with weak learners inheriting encoder weights from prior rounds except for the final classification head.

For efficient implementation:

• Warm-start $$R[f]=\frac{1}{N}\sum_{i=1}^N L(z_i, f(x_i)),\quad L(z, f)=\log\left(\sum_{k=1}^M e^{f_k}-e^{f_z}\right).$$0 with previous encoder weights.
• Compute token importances in a single pass over the full batch.
• Employ mixed precision for memory and throughput gains.
• Adjust $$R[f]=\frac{1}{N}\sum_{i=1}^N L(z_i, f(x_i)),\quad L(z, f)=\log\left(\sum_{k=1}^M e^{f_k}-e^{f_z}\right).$$1 or $$R[f]=\frac{1}{N}\sum_{i=1}^N L(z_i, f(x_i)),\quad L(z, f)=\log\left(\sum_{k=1}^M e^{f_k}-e^{f_z}\right).$$2 if residual norms indicate overfitting.

BoostTransformer maintains end-to-end differentiability and compatibility with pre-trained Transformer pipelines, providing both training acceleration and higher predictive performance relative to standard Transformer fine-tuning (Fang et al., 4 Aug 2025).

8. Relationship to Prior Boosting-Transformer Work

BoostingBERT (Huang et al., 2020) similarly integrates boosting into Transformer pipelines, using sequentially fine-tuned BERT (or RoBERTa) base models with weighted example reweighting and ensembling via a fusion MLP. Notable distinctions include the use of weighted cross-entropy rather than least-squares objectives, and the absence of explicit subgrid token selection or importance-weighted data subsampling as in BoostTransformer. BoostingBERT also demonstrates the superiority of boosting over bagging or stacking in text classification, particularly in low-data regimes, and supports distillation into a single student model for practical deployment.

Empirical table (GLUE and Chinese benchmarks) from BoostingBERT:

Dataset/Task	BERT-Base	BoostingBERT	Δ
CoLA	80.72	82.93	+2.21
SST-2	92.55	93.35	+0.80
MRPC	85.29	87.87	+2.58
MNLI	84.39	85.16	+0.77
RTE	67.15	72.92	+5.77

This suggests that incorporating boosting methods into Transformer frameworks yields consistent accuracy improvements across diverse NLP tasks and resource settings (Huang et al., 2020).

---

References:

(Fang et al., 4 Aug 2025): "BoostTransformer: Enhancing Transformer Models with Subgrid Selection and Importance Sampling" (Huang et al., 2020): "BoostingBERT:Integrating Multi-Class Boosting into BERT for NLP Tasks"