Verifiable Training with Interval Bound Propagation

• The paper’s main contribution is its detailed method to propagate worst-case input intervals for certifying neural network robustness.
• It introduces algorithmic enhancements like CROWN-IBP and TAPS, which improve the tightness and stability of robustness bounds.
• Practical strategies such as warmup scheduling and IBP-aware initialization are key to balancing certified robustness with standard accuracy.

Verifiable training using Interval Bound Propagation (IBP) refers to a class of neural network training protocols that enable the derivation of certified, provable guarantees on robustness to norm-bounded adversarial perturbations. IBP-based verifiable training methods compute, during optimization, sound upper bounds on the worst-case loss that a model may incur within a specified ℓ∞-ball or, more generally, any axis-aligned input uncertainty set. Through layerwise interval arithmetic, IBP enables certified defenses to scale efficiently to deep and wide architectures, and forms the basis of a range of recent state-of-the-art robust training strategies, including hybrid and regularized variants. The central role of IBP is its extremely efficient propagation of worst-case input intervals through a network, yielding bounds that can (provably) over-approximate any possible adversarial input, and therefore certify model predictions.

1. Mathematical Principles of IBP-Based Verifiable Training

Let $f_\theta : \mathbb{R}^d \to \mathbb{R}^K$ denote a $K$-class classifier. The certified robust objective seeks to minimize the expected worst-case loss over inputs $x$ under ℓ∞ perturbation of radius $\epsilon$: $$\theta^* = \arg\min_\theta \mathbb{E}_{(x,y)\sim D}\left[ \sup_{\|\delta\|_\infty \leq \epsilon} \ell_\mathrm{CE}(f_\theta(x+\delta), y) \right],$$ where $\ell_\mathrm{CE}$ is cross-entropy.

Direct computation of the supremum is intractable for nonlinear networks. IBP replaces it by propagating input intervals $[x-\epsilon,x+\epsilon]$ layerwise, producing output intervals $[\ell^K,u^K]$ that enclose all possible outputs given allowed perturbations.

For each affine layer with $z^k = W^k z^{k-1} + b^k$ and input intervals $[\ell^{k-1}, u^{k-1}]$, IBP uses: $$\begin{aligned} W^k_+ &= \max(W^k, 0),\quad W^k_- = \min(W^k, 0),\ \ell^k &= W^k_+ \ell^{k-1} + W^k_- u^{k-1} + b^k, \ u^k &= W^k_+ u^{k-1} + W^k_- \ell^{k-1} + b^k. \end{aligned}$$

For monotonic activations (e.g., ReLU), intervals are transformed elementwise: $[\ell^k, u^k] \mapsto [h(\ell^k), h(u^k)]$.

The final logit-difference intervals provide certified bounds for each class. An upper bound on the robust loss is computed, e.g., for cross-entropy: $$L_\mathrm{IBP}(x, y; \epsilon) = \log\left(1 + \sum_{i \neq y} \exp(\overline{o}^\Delta_i)\right),$$ where $\overline{o}^\Delta_i$ is the upper bound on $f_i(x') - f_y(x')$ for $x'$ in the input interval.

2. Core Methodologies and Loss Formulations

IBP-based verifiable training optimizes surrogate losses that upper-bound the true worst-case loss via these propagated intervals. The canonical training loss is a mix of the natural (standard) and robust (certified) losses: $$L(x, y) = \kappa\,\ell_\mathrm{CE}(f_\theta(x), y) + (1 - \kappa)\,L_\mathrm{IBP}(x, y; \epsilon),$$ where $\kappa$ is typically scheduled from $1$ (all standard loss) to $0$ (pure certified) over the training run for stability (Gowal et al., 2018, Mao et al., 2023).

Warmup scheduling is critical. $\epsilon$ is ramped from $0$ to the target value over a predefined number of epochs to prevent large initial interval blow-up and optimization collapse. Likewise, regularization (e.g., gradient norm clipping) and network-specific tricks (e.g., full-layer BatchNorm, IBP-aware initialization) are essential for stable training (Shi et al., 2021).

3. Algorithmic Enhancements and Hybrid Strategies

Several algorithmic variants are built upon IBP to improve tightness, stability, and accuracy:

• CROWN-IBP: Combines the forward IBP pass with a backward dynamic linear relaxation bound (CROWN) to yield hybrid bounds. Early epochs use CROWN for stability, then switch to IBP for efficiency as training progresses. The final certified margin is a convex combination of IBP and CROWN bounds (Zhang et al., 2019).
• TAPS/STAPS: Interleave IBP and PGD-based under-approximations in a split network architecture: IBP is used to enclose early (feature-extraction) layers, while PGD attacks maximize logit differences within the embedding box for later (classification) layers. The total loss multiplicatively combines the (sound) IBP upper bound with the (unsound) TAPS estimate, and gradients are balanced by a tunable mixing coefficient (Mao et al., 2023). TAPS significantly improves certified accuracy compared to pure IBP and adversarial training.
• IBP-R: IBP-regularization involves an additional loss term minimizing the area of the convex hull (Planet relaxation) for all ambiguous ReLU activations, thus reducing the integrality gap for branch-and-bound verification post-training. Adversarial training is performed on an enlarged ball, and the IBP regularizer encourages tighter relaxations (Palma et al., 2022).
• Expected Tight Bounds (ETB): Replaces layerwise IBP with blockwise bounds that are provably tighter in expectation for Affine–ReLU–Affine blocks, directly improving robustness/accuracy trade-off (Alsubaihi et al., 2019).
• QA-IBP: For quantized neural networks (QNNs), QA-IBP modifies the propagation rules to respect quantization, using fake-quantized operations in interval arithmetic and straight-through gradient estimation for differentiability, thus enabling end-to-end robust quantized model training and verification (Lechner et al., 2022).

4. Implementation Practices and Stability Measures

Empirical experience demonstrates that detailed tuning is mandatory for stable and effective IBP certified training:

• Weight Initialization: Standard initializations (Xavier, He) can cause interval (bound) explosion; IBP-aware initialization scales weights to maintain constant expected per-layer interval widths (Shi et al., 2021).
• Full-Layer BatchNorm: Inserting BatchNorm after every affine layer curtails the tendency of IBP optimization to converge towards “dead” ReLU neurons via over-regularization, maintaining a healthy balance between active and inactive units (Shi et al., 2021, Mao et al., 2023).
• Warmup Regularization: Additional loss terms penalize loose bounds or imbalanced activations during early epochs. For example, enforcing a maximum mean difference $\hat E(\Delta_i)$ per layer and balancing the contributions of active/inactive neurons (Shi et al., 2021).
• Curriculum Schedules: A dual schedule for both robust-loss weight $\kappa$ and perturbation radius $\epsilon$ is required to prevent collapse to trivial or poor minima (Gowal et al., 2018, Mao et al., 2023).
• Hybrid Verification Regimes: In safety-critical control, IBP can be embedded into a branch-and-bound verification-in-the-loop pipeline for simultaneous synthesis and certification of neural network control barrier functions (nCBFs). Here, IBP is used for efficient bound propagation on boxes over the input space, and violated regions are iteratively refined (Wang et al., 2023).

A summary of recommended practical steps:

Practice	Purpose	Notable Effect
IBP init	Prevent bound explosion	Stable/efficient learning
Full BN	Balance ReLU activation	Improved bound tightness, train acc.
Warmup curric.	Stabilize opt. landscape	Avoid trivial/frozen solutions
Regularizers	Tighten bounds/activations	Higher certified accuracy

5. Empirical Trade-offs: Certified Accuracy versus Standard Accuracy

IBP-based certified training suffers from an inherent trade-off: the tighter (more regularized) the network in terms of IBP-bound tightness, the more heavily the weights are regularized towards monotonicity, which can substantially reduce standard accuracy (Mao et al., 2023). Empirical studies report that pure IBP pushes networks toward near-exactness in the propagated intervals (tightness $\tau \to 1$), often at a pronounced cost to clean accuracy. Hybrid approaches (e.g. partial-IBP, TAPS, IBP-R) attenuate this tension by reducing over-regularization, thus raising both clean and certified accuracy.

Table: Representative certified and standard accuracy results (natural = clean, cert = IBP-verified, all at $\ell_\infty$ radius shown; CIFAR-10, TinyImageNet, MNIST):

Method	CIFAR-10 ($\epsilon=2/255$)	TinyImageNet ($\epsilon=1/255$)	MNIST ($\epsilon=0.3$)
IBP	nat 66.84%, cert 52.85%	nat 25.92%, cert 17.87%	nat 97.67%, cert 93.10%
SABR	nat 79.24%, cert 62.84%	nat 28.85%, cert 20.46%	-
TAPS	nat 75.09%, cert 61.56%	nat 28.34%, cert 20.82%	nat 97.94%, cert 93.62%
STAPS	nat 79.76%, cert 62.98%	nat 28.98%, cert 22.16%	nat 98.53%, cert 93.51%

TAPS/STAPS achieve higher certified and standard accuracy versus pure IBP. Adopting regularization and improved initialization significantly reduces the training schedule from thousands to $\mathcal{O}(80{-}160)$ epochs without sacrificing guarantees (Shi et al., 2021).

6. Theoretical Perspectives and Limitations

IBP-based verifiable training has been rigorously analyzed in terms of convergence, tightness, and expressiveness:

• Convergence: For small $\epsilon$ and sufficiently overparameterized two-layer ReLU networks, gradient descent on the IBP robust loss converges linearly to zero robust training error with high probability (Wang et al., 2022).
• Tightness and Regularization: IBP’s propagation tightness decays polynomially with width and exponentially with depth at initialization. Training automatically increases tightness, but exactness in intervals imposes monotonic sign constraints and may curtail network expressiveness (Mao et al., 2023). Very high tightness ($\tau\approx1$) is neither strictly necessary nor sufficient for strong certified robustness; moderate hybrid levels often yield better accuracy–robustness trade-off.
• Extensions and Applicability: QA-IBP demonstrates that IBP can be adapted for robust training and complete verification of quantized neural networks, critical for deployment on low-precision hardware (Lechner et al., 2022). Similarly, blockwise (ETB) and linearly-relaxed IBP (LBP) provide tighter bounds at moderate computational cost (Alsubaihi et al., 2019, Lyu et al., 2021).

7. Future Directions and Open Problems

While IBP underlies the state of the art for scalable certified robust training, open avenues remain:

• Adaptive Bound Tightening: Developing mechanisms to propagate bounds only on critical neurons or layers, or to use tighter relaxations selectively, may maintain strong certifiability without incurring over-regularization (Mao et al., 2023).
• Dynamic Region and Weight Scheduling: Automatically adjusting the fraction of IBP-based training (hybridization), the robust loss weight, or the perturbation size during optimization could strike better accuracy–robustness trade-offs.
• Activation Diversity: Using novel activations such as ParamRamp allows escape from the dead ReLU pathology and improves certified performance on challenging datasets (Lyu et al., 2021).
• Applications beyond Classification: Embedding IBP into control barrier function synthesis, or into verification-in-the-loop retraining pipelines, supports certification in sequential, safety-critical tasks (Wang et al., 2023).
• Expressivity vs. Certifiability: Understanding the fundamental limits imposed by IBP-optimization-induced regularization, and seeking architectures (e.g., via width, customized non-linearities) or objectives that reduce this representational loss, remain significant research targets.

A plausible implication is that further improvements in certifiable robustness may come less from pursuing maximal tightness in IBP bounds, and more from hybrid, adaptive, or architecture-tailored propagation and training regimes that preserve both the expressive capacity and verifiability of the model.