VeriBToT: Robust Boosting Verification

• VeriBToT is a framework for training, certifying, and verifying gradient-boosted tree classifiers under adversarial perturbations with exact robustness guarantees.
• It leverages the large-spread ensemble property to enable efficient, linear or pseudo-polynomial time certification against norm-bounded adversarial attacks.
• By imposing structural constraints during training, VeriBToT maintains predictive accuracy with less than 1% loss, while significantly accelerating verification compared to conventional methods.

VeriBToT is a framework for training, certifying, and verifying gradient-boosted tree classifiers under adversarial perturbations, enabling exact robustness guarantees in polynomial or pseudo-polynomial time under specific conditions. Central to VeriBToT is the large-spread ensemble property, which enables efficient certification against norm-bounded adversarial attacks for advanced tree boosting models, including those trained with XGBoost or LightGBM. By imposing structural constraints during training and deploying tailored verification algorithms, VeriBToT addresses the computational complexity of robustness certification without significant degradation to predictive accuracy (Calzavara et al., 22 Feb 2024).

1. Formal Model and Definitions

VeriBToT targets binary classification over a feature space $X \subseteq \mathbb{R}^d$ and label set $Y = \{+1,-1\}$. The fundamental model is a gradient-boosted ensemble $T = \{t_1, \dots, t_m\}$, where each $t_i$ is a regression tree. Each regression tree $t$ is recursively defined as either a leaf node $\lambda(s)$ with real-valued score $s$, or an internal node $\sigma(f,v,t_L,t_R)$ splitting at feature $f$ and threshold $v$. The overall ensemble prediction on input $x$ is:

$\hat T(x) = \sum_{i=1}^m t_i(x)$

Classification applies a monotone link $\iota: \mathbb{R} \to \mathbb{R}$ and threshold $\tau$, returning $T(x) = +1$ if $\iota(\hat T(x)) \geq \tau$ and $-1$ otherwise.

The defining property for enabled verification is "large-spread": for norm $\lVert\cdot\rVert_p$, $T$ is large-spread with respect to adversarial budget $k$ if

$\text{spread}_p(T) > 2k$

where the $p$-spread quantifies the minimal separation (in the $p$-norm) between any pair of thresholds on the same feature across distinct trees.

2. Complexity and Verifiability Results

Verification of classifier robustness under adversarial attack consists of deciding, for instance $x$ and ground-truth $y$, whether every allowed perturbation $z$ with $\lVert z - x \rVert_p \leq k$ leaves classification invariant ($T(z) = y$). For large-spread ensembles:

• For the $\ell_\infty$-norm, exact verification is achieved in linear time ($O(N)$, $N$ = total number of tree nodes). This stems from the lack of interference between trees under large-spread orthogonality.
• For any fixed $p < \infty$, robustness verification is NP-hard. Nonetheless, pseudo-polynomial time algorithms exist using dynamic programming.

These properties extend verifiable learning principles to boosted ensembles.

3. Verification Algorithms

a. $\ell_\infty$-Norm: Linear-Time Verification

The attacker's optimal strategy consists of, for each tree, selecting the highest-gain reachable leaf (with perturbation cost $\leq k$). For all $t_i$:

• Identify leaves $j$ with minimal perturbation $\lVert \delta_{ij}\rVert_\infty \leq k$.
• Compute gain $G_{ij}$ in ensemble score.
• Aggregate maximal non-negative $G_{ij}$ across trees to form $\Gamma$:

$$\Gamma = \sum_{i=1}^m \max_{j \in L_i} \max(0, G_{ij})$$

Then, for raw score $s = \hat T(x)$, robustness is certified if:

• $y=+1$: $\iota(s - \Gamma) \geq \tau$,
• $y=-1$: $\iota(s + \Gamma) < \tau$.

The algorithm is linear in the number of nodes:

function VERIFY_LINF(T, x, y, k): if T(x) != y: return False Γ = 0 for each tree t_i in T: compute reachable leaves L_i and gains G_{ij} Γ += max_{j ∈ L_i} max(0, G_{ij}) s = sum_i t_i(x) if y == +1: return (i(s - Γ) >= τ) else: return (i(s + Γ) < τ)

b. General $\ell_p$-Norm: Pseudo-Polynomial Verification

With $\ell_p$, the adversarial budget is $\sum_i w_i \leq K$, where $w_i = \lVert \delta_i \rVert_p^p$ and $K = k^p$. Each tree presents leaf/gain pairs $(w_{ij}, G_{ij})$. This yields a grouped-knapsack DP:

$$M[i,q] = \max\Bigl\{ M[i-1, q],\ \max_{j: w_{ij}\le q}(M[i-1, q-w_{ij}]+G_{ij}) \Bigr\}$$

Initialize $M[0,q]=0$ for all $q=0,\dots,K$. The answer is $\Gamma=M[m,K]$; the same classification test applies as for $\ell_\infty$.

The complexity is $O(m \cdot K \cdot L_{\text{max}})$, where $L_{\text{max}}$ is the number of leaves per tree. Weight discretization is used when $w_{ij}$ and $K$ are not integers.

4. Practical Construction of Large-Spread Ensembles

During training—e.g., in LightGBM—whenever a tree $t_i$ splits on $f$ at threshold $v$, subsequent trees are barred from choosing $v' \in (v-2k, v+2k)$ for feature $f$. This ensures the large-spread condition throughout the ensemble. Such exclusions marginally reduce model capacity, but empirical results indicate that predictive accuracy loss remains negligible (within $1\%$ of standard boosting).

5. Empirical Performance and Limitations

VeriBToT-certified models have been benchmarked on FMNIST, MNIST, and Webspam. Key findings:

• Accuracy of large-spread models matches that of unconstrained LightGBM within $1\%$.
• Robustness certified by VeriBToT matches or nearly matches the most optimistic bounds of unconstrained GBDT, but is guaranteed for every test point.
• Verification time is accelerated by $10\times$ to $400\times$ relative to MILP or abstract-interpretation approaches, even for ensembles with up to 125 trees (each of depth 8).
• Computation for $\ell_0$ or $\ell_1$ attacks is pseudo-polynomial in $k^p$, but this is practical for small perturbation budgets.

6. Significance and Conclusions

VeriBToT extends verifiable machine learning from majority-vote tree ensembles to more powerful boosted ensemble techniques, retaining provable robustness against norm-bounded adversaries in feasible computational time. Imposing the large-spread constraint during training does not materially compromise accuracy, while enabling efficient and exact certification under $\ell_\infty$ and, with dynamic programming, for general $\ell_p$ attacks. This makes VeriBToT suitable for high-assurance applications requiring both model performance and certified robustness (Calzavara et al., 22 Feb 2024).