---
title: Verifiable Training with Interval Bound Propagation
url: https://www.emergentmind.com/topics/verifiable-training-using-ibp
type: topic
---

# Verifiable Training with Interval Bound Propagation

Verifiable training using Interval Bound Propagation (IBP) refers to a class of neural network training protocols that enable the derivation of certified, provable guarantees on robustness to norm-bounded adversarial perturbations. IBP-based verifiable training methods compute, during optimization, sound upper bounds on the worst-case loss that a model may incur within a specified ℓ∞-ball or, more generally, any axis-aligned input uncertainty set. Through layerwise interval arithmetic, IBP enables certified defenses to scale efficiently to deep and wide architectures, and forms the basis of a range of recent state-of-the-art robust training strategies, including hybrid and regularized variants. The central role of IBP is its extremely efficient propagation of worst-case input intervals through a network, yielding bounds that can (provably) over-approximate any possible adversarial input, and therefore certify model predictions.

## 1. Mathematical Principles of IBP-Based Verifiable Training

Let $f_\theta : \mathbb{R}^d \to \mathbb{R}^K$ denote a $K$-class classifier. The certified robust objective seeks to minimize the expected worst-case loss over inputs $x$ under ℓ∞ perturbation of radius $\epsilon$:
\[
\theta^* = \arg\min_\theta \mathbb{E}_{(x,y)\sim D}\left[ \sup_{\|\delta\|_\infty \leq \epsilon} \ell_\mathrm{CE}(f_\theta(x+\delta), y) \right],
\]
where $\ell_\mathrm{CE}$ is cross-entropy.

Direct computation of the supremum is intractable for nonlinear networks. IBP replaces it by propagating input intervals $[x-\epsilon,x+\epsilon]$ layerwise, producing output intervals $[\ell^K,u^K]$ that enclose all possible outputs given allowed perturbations.

For each affine layer with $z^k = W^k z^{k-1} + b^k$ and input intervals $[\ell^{k-1}, u^{k-1}]$, IBP uses:
\[
\begin{aligned}
W^k_+ &= \max(W^k, 0),\quad W^k_- = \min(W^k, 0),\\
\ell^k &= W^k_+ \ell^{k-1} + W^k_- u^{k-1} + b^k, \\
u^k   &= W^k_+ u^{k-1} + W^k_- \ell^{k-1} + b^k.
\end{aligned}
\]

For monotonic activations (e.g., ReLU), intervals are transformed elementwise: $[\ell^k, u^k] \mapsto [h(\ell^k), h(u^k)]$.

The final logit-difference intervals provide certified bounds for each class. An upper bound on the robust loss is computed, e.g., for cross-entropy:
\[
L_\mathrm{IBP}(x, y; \epsilon) = \log\left(1 + \sum_{i \neq y} \exp(\overline{o}^\Delta_i)\right),
\]
where $\overline{o}^\Delta_i$ is the upper bound on $f_i(x') - f_y(x')$ for $x'$ in the input interval.


## 2. Core Methodologies and Loss Formulations

IBP-based verifiable training optimizes surrogate losses that upper-bound the true worst-case loss via these propagated intervals. The canonical training loss is a mix of the natural (standard) and robust (certified) losses:
\[
L(x, y) = \kappa\,\ell_\mathrm{CE}(f_\theta(x), y) + (1 - \kappa)\,L_\mathrm{IBP}(x, y; \epsilon),
\]
where $\kappa$ is typically scheduled from $1$ (all standard loss) to $0$ (pure certified) over the training run for stability [1810.12715, 2306.10426].

Warmup scheduling is critical. $\epsilon$ is ramped from $0$ to the target value over a predefined number of epochs to prevent large initial interval blow-up and optimization collapse. Likewise, regularization (e.g., gradient norm clipping) and network-specific tricks (e.g., full-layer BatchNorm, IBP-aware initialization) are essential for stable training [2103.17268].


## 3. Algorithmic Enhancements and Hybrid Strategies

Several algorithmic variants are built upon IBP to improve tightness, stability, and accuracy:

- **CROWN-IBP**: Combines the forward IBP pass with a backward dynamic linear relaxation bound (CROWN) to yield hybrid bounds. Early epochs use CROWN for stability, then switch to IBP for efficiency as training progresses. The final certified margin is a convex combination of IBP and CROWN bounds [1906.06316].

- **TAPS/STAPS**: Interleave IBP and PGD-based under-approximations in a split network architecture: IBP is used to enclose early (feature-extraction) layers, while PGD attacks maximize logit differences within the embedding box for later (classification) layers. The total loss multiplicatively combines the (sound) IBP upper bound with the (unsound) TAPS estimate, and gradients are balanced by a tunable mixing coefficient [2305.04574]. TAPS significantly improves certified accuracy compared to pure IBP and adversarial training.

- **IBP-R**: IBP-regularization involves an additional loss term minimizing the area of the convex hull (Planet relaxation) for all ambiguous ReLU activations, thus reducing the integrality gap for branch-and-bound verification post-training. Adversarial training is performed on an enlarged ball, and the IBP regularizer encourages tighter relaxations [2206.14772].

- **Expected Tight Bounds (ETB)**: Replaces layerwise IBP with blockwise bounds that are provably tighter in expectation for Affine–ReLU–Affine blocks, directly improving robustness/accuracy trade-off [1905.12418].

- **QA-IBP**: For quantized neural networks (QNNs), QA-IBP modifies the propagation rules to respect quantization, using fake-quantized operations in interval arithmetic and straight-through gradient estimation for differentiability, thus enabling end-to-end robust quantized model training and verification [2211.16187].


## 4. Implementation Practices and Stability Measures

Empirical experience demonstrates that detailed tuning is mandatory for stable and effective IBP certified training:

- **Weight Initialization**: Standard initializations (Xavier, He) can cause interval (bound) explosion; IBP-aware initialization scales weights to maintain constant expected per-layer interval widths [2103.17268].

- **Full-Layer BatchNorm**: Inserting BatchNorm after every affine layer curtails the tendency of IBP optimization to converge towards “dead” ReLU neurons via over-regularization, maintaining a healthy balance between active and inactive units [2103.17268, 2306.10426].

- **Warmup Regularization**: Additional loss terms penalize loose bounds or imbalanced activations during early epochs. For example, enforcing a maximum mean difference $\hat E(\Delta_i)$ per layer and balancing the contributions of active/inactive neurons [2103.17268].

- **Curriculum Schedules**: A dual schedule for both robust-loss weight $\kappa$ and perturbation radius $\epsilon$ is required to prevent collapse to trivial or poor minima [1810.12715, 2306.10426].

- **Hybrid Verification Regimes**: In safety-critical control, IBP can be embedded into a branch-and-bound verification-in-the-loop pipeline for simultaneous synthesis and certification of neural network control barrier functions (nCBFs). Here, IBP is used for efficient bound propagation on boxes over the input space, and violated regions are iteratively refined [2311.10438].

A summary of recommended practical steps:

| Practice        | Purpose                     | Notable Effect                       |
|-----------------|----------------------------|--------------------------------------|
| IBP init        | Prevent bound explosion     | Stable/efficient learning            |
| Full BN         | Balance ReLU activation    | Improved bound tightness, train acc. |
| Warmup curric.  | Stabilize opt. landscape   | Avoid trivial/frozen solutions       |
| Regularizers    | Tighten bounds/activations | Higher certified accuracy            |


## 5. Empirical Trade-offs: Certified Accuracy versus Standard Accuracy

IBP-based certified training suffers from an inherent trade-off: the tighter (more regularized) the network in terms of IBP-bound tightness, the more heavily the weights are regularized towards monotonicity, which can substantially reduce standard accuracy [2306.10426]. Empirical studies report that pure IBP pushes networks toward near-exactness in the propagated intervals (tightness $\tau \to 1$), often at a pronounced cost to clean accuracy. Hybrid approaches (e.g. partial-IBP, TAPS, IBP-R) attenuate this tension by reducing over-regularization, thus raising both clean and certified accuracy.

Table: Representative certified and standard accuracy results (natural = clean, cert = IBP-verified, all at $\ell_\infty$ radius shown; CIFAR-10, TinyImageNet, MNIST):

| Method    | CIFAR-10 ($\epsilon=2/255$)     | TinyImageNet ($\epsilon=1/255$)   | MNIST ($\epsilon=0.3$)        |
|-----------|-------------------------------|-----------------------------------|-------------------------------|
| IBP       | nat 66.84%, cert 52.85%       | nat 25.92%, cert 17.87%           | nat 97.67%, cert 93.10%       |
| SABR      | nat 79.24%, cert 62.84%       | nat 28.85%, cert 20.46%           | -                             |
| TAPS      | nat 75.09%, cert 61.56%       | nat 28.34%, cert 20.82%           | nat 97.94%, cert 93.62%       |
| STAPS     | nat 79.76%, cert 62.98%       | nat 28.98%, cert 22.16%           | nat 98.53%, cert 93.51%       |

TAPS/STAPS achieve higher certified and standard accuracy versus pure IBP. Adopting regularization and improved initialization significantly reduces the training schedule from thousands to $\mathcal{O}(80{-}160)$ epochs without sacrificing guarantees [2103.17268].


## 6. Theoretical Perspectives and Limitations

IBP-based verifiable training has been rigorously analyzed in terms of convergence, tightness, and expressiveness:

- **Convergence**: For small $\epsilon$ and sufficiently overparameterized two-layer ReLU networks, gradient descent on the IBP robust loss converges linearly to zero robust training error with high probability [2203.08961].

- **Tightness and Regularization**: IBP’s propagation tightness decays polynomially with width and exponentially with depth at initialization. Training automatically increases tightness, but exactness in intervals imposes monotonic sign constraints and may curtail network expressiveness [2306.10426]. Very high tightness ($\tau\approx1$) is neither strictly necessary nor sufficient for strong certified robustness; moderate hybrid levels often yield better accuracy–robustness trade-off.

- **Extensions and Applicability**: QA-IBP demonstrates that IBP can be adapted for robust training and *complete* verification of quantized neural networks, critical for deployment on low-precision hardware [2211.16187]. Similarly, blockwise (ETB) and linearly-relaxed IBP (LBP) provide tighter bounds at moderate computational cost [1905.12418, 2104.00447].


## 7. Future Directions and Open Problems

While IBP underlies the state of the art for scalable certified robust training, open avenues remain:

- **Adaptive Bound Tightening**: Developing mechanisms to propagate bounds only on critical neurons or layers, or to use tighter relaxations selectively, may maintain strong certifiability without incurring over-regularization [2306.10426].

- **Dynamic Region and Weight Scheduling**: Automatically adjusting the fraction of IBP-based training (hybridization), the robust loss weight, or the perturbation size during optimization could strike better accuracy–robustness trade-offs.

- **Activation Diversity**: Using novel activations such as ParamRamp allows escape from the dead ReLU pathology and improves certified performance on challenging datasets [2104.00447].

- **Applications beyond Classification**: Embedding IBP into control barrier function synthesis, or into verification-in-the-loop retraining pipelines, supports certification in sequential, safety-critical tasks [2311.10438].

- **Expressivity vs. Certifiability**: Understanding the fundamental limits imposed by IBP-optimization-induced regularization, and seeking architectures (e.g., via width, customized non-linearities) or objectives that reduce this representational loss, remain significant research targets.

A plausible implication is that further improvements in certifiable robustness may come less from pursuing maximal tightness in IBP bounds, and more from hybrid, adaptive, or architecture-tailored propagation and training regimes that preserve both the expressive capacity and verifiability of the model.

Source: https://www.emergentmind.com/topics/verifiable-training-using-ibp