Rolling-Horizon OBBT for Neural Network Verification

• Rolling-Horizon OBBT is a scalable framework for neural network verification that decomposes global MIP bound tightening into smaller, rolling-horizon subproblems.
• It processes consecutive network layer windows to update bounds and stabilize ReLU activations, balancing solution tightness with computational efficiency.
• Empirical results show that OBBT-RH reduces bound-tightening time by about 30% and outperforms LP-based methods by closing additional unsolved benchmark instances.

Rolling-Horizon Optimization-Based Bound Tightening (OBBT-RH) is a scalable algorithmic framework for neural network verification that decomposes the classical MIP-based OBBT process into tractable subproblems using a rolling-horizon scheme. This approach leverages the layered architecture of feed-forward ReLU neural networks to achieve near-global bound tightness while maintaining computational feasibility, particularly for deep or wide models. OBBT-RH supports extensive parallelization and demonstrates superior verification performance over both global OBBT and LP-based bound-propagation schemes on standard VNN-COMP benchmarks (Zhao et al., 2024).

1. Mathematical Formulation of the Bound-Tightening Problem

Consider a feed-forward ReLU-activated neural network with $L$ layers, weight matrices $W^{(i)}\in\mathbb{R}^{n_i \times n_{i-1}}$, biases $b^{(i)}\in\mathbb{R}^{n_i}$, and input set $$\mathbf{x}^{(0)}\in\mathcal{C}\subseteq\mathbb{R}^{n_0}$$. The pre-activations and post-activations at layer $i$ are denoted as $\mathbf{y}^{(i)}$ and $\mathbf{x}^{(i)}$ respectively, with the layer-wise propagation

$$\mathbf{y}^{(i)} = W^{(i)} \mathbf{x}^{(i-1)} + b^{(i)}, \quad \mathbf{x}^{(i)} = \mathrm{ReLU}(\mathbf{y}^{(i)}), \quad i = 1, \dots, L-1.$$

Neural network verification queries properties encoded by a final affine function $f(\mathbf{y}^{(L)})$ expressed as a MIP: $$\begin{aligned} \min_{\mathbf{x}, \mathbf{y}, z} \quad & f(\mathbf{y}^{(L)}) \ \text{s.t.} \quad & \mathbf{y}^{(i)} = W^{(i)} \mathbf{x}^{(i-1)} + b^{(i)} \quad (\forall i) \ & x^{(i)}_k \ge 0,\; x^{(i)}_k \ge y^{(i)}_k,\; x^{(i)}_k \le y^{(i)}_k - \ell^{(i)}_k (1-z^{(i)}_k), \ & x^{(i)}_k \le u^{(i)}_k z^{(i)}_k, \; z^{(i)}_k \in \{0,1\} \ & \mathbf{x}^{(0)} \in \mathcal{C} \end{aligned}$$ where $\ell^{(i)}_k, u^{(i)}_k$ are lower/upper bounds on $y^{(i)}_k$, and each binary variable $z^{(i)}_k$ encodes ReLU switches. Optimization-Based Bound Tightening (OBBT) refines the box bounds $\ell^{(i)} \leq \mathbf{y}^{(i)} \leq u^{(i)}$ and $\ell^{(i)}_x \leq \mathbf{x}^{(i)} \leq u^{(i)}_x$, strengthening the global MIP relaxation by solving, for pre-activation $y_k^{(t)}$,

$$\underline{y}^{(t)}_k = \min y^{(t)}_k \;\; \text{s.t. full MIP constraints} \quad \overline{y}^{(t)}_k = \max y^{(t)}_k \;\; \text{s.t. full MIP constraints}.$$

While exact, global OBBT incurs prohibitive cost as network depth increases.

2. Rolling-Horizon Decomposition: Rationale and Subgraph MIP Formulation

OBBT-RH introduces a rolling-horizon decomposition by restricting each bound-tightening MIP to small subgraphs consisting of $H$ consecutive layers (the “horizon”). Specifically, OBBT-RH proceeds over a sequence of windows $(s_j, t_j)$, each with $t_j-s_j=H$, spanning layers $s_j+1$ through $t_j$. For neuron $k$ at layer $t_j$, the sub-MIP for local bound refinement is given by

$\min/\max \; y^{(t)}_k$

subject to all layer-wise affine and big-M constraints within the window, using the most recent available bounds as big-M coefficients. The process yields $\underline{y}^{(t)}_k$, $\overline{y}^{(t)}_k$, and—if either $\overline{y}^{(t)}_k\le 0$ or $\underline{y}^{(t)}_k\ge 0$—permits ReLU stabilization and dropping of the corresponding binary variable for subsequent calls.

By incrementally rolling the window from input toward output layers and propagating updated bounds forward, OBBT-RH balances tightness against MIP tractability. Short horizons facilitate faster solves but looser bounds; longer horizons yield tighter bounds at higher computational expense.

3. Algorithmic Workflow and Termination

The OBBT-RH algorithm executes the following workflow for each window in a preselected sequence:

1. For each window $(s_j,t_j)$, iterate through all neurons $k=1,\dots,n_{t_j}$ at layer $t_j$.
2. For each neuron, solve the sub-MIP twice (minimization and maximization) over the window to refine bounds.
3. Update $\ell^{(t_j)}_k$ and $u^{(t_j)}_k$ based on the solve.
4. Stabilize ReLU activation status if applicable.
5. After all windows, repeat or terminate if no bounds are significantly updated.

Termination occurs upon completion of all windows with negligible bound changes or expiration of a global resource budget. Sub-MIPs are subject to an explicit time limit (30 seconds per instance), and early stopping is invoked upon ReLU stabilization.

4. Comparison with Global OBBT and Bound Propagation

Direct OBBT—solving the full-depth MIP for every neuron—produces tightest possible bounds and maximal ReLU stabilization, but scales poorly. LP-based bound-propagation algorithms such as Interval Bound Propagation (IBP), CROWN, and α-CROWN are more scalable but fundamentally limited to convex-relaxation tightness, inadequate for capturing all deep network dependencies.

OBBT-RH, for moderately sized window horizons ($H\ll L$), recovers approximately global-OBBT bounds with a small fraction of the computational burden, as it focuses optimization on rolling subgraphs, always using the best available bounds from prior windows. This approach yields bound tightness and ReLU stabilization comparable to global OBBT but with improved tractability.

5. Parallelization and Implementation Details

Each neuron’s bound in a window can be tightened independently, admitting straightforward parallelization. OBBT-RH exploits this by multi-threading at the node level (2 CPU threads per sub-MIP) and distributing windows across nodes via MPI. A global coordinator manages job dispatch ($(s_j, t_j, k)$), result collection, and ReLU stabilization state, enabling high throughput across large models. The open-source implementation is based on the Gravity modeling tool with support for generic neural network models.

6. Empirical Performance on VNN-COMP Benchmarks

On the VNN-COMP MCNIST benchmark suite (90 4- and 7-layer fully connected ReLU networks), OBBT-RH—when composed with end-to-end MIP verification—yields the tightest average pre-activation bounds among tested methods. Selected results:

Method	Stabilized ReLUs	Bounds Range	LP Bound	Verified Instances	BT (s)	MIP (s)	Total (s)
IBP	122.4	2028.35	-25232.25	37	3.5	708.6	712.1
CROWN	454.4	96.16	-1169.45	62	3.5	345.9	349.4
α-CROWN	486.6	53.52	-577.91	61	4.9	340.7	345.7
OBBT	653.9	12.87	-0.64	79	121.2	81.1	202.3
OBBT-RH	654.2	12.38	-24.77	80	90.2	64.7	154.9

OBBT-RH and global OBBT outperform IBP, CROWN, and α-CROWN by an order of magnitude in stabilization and bound tightness metrics, with OBBT-RH reducing bound-tightening time by approximately 30% compared to global OBBT. OBBT-RH closes four additional benchmark instances unsolved by other complete verifiers. Horizon lengths $H=3$–5 enable near-global-OBBT quality at 2× speedup. The early-stop criterion has negligible impact on bound quality but further accelerates bound-tightening.

7. Significance and Implications

OBBT-RH achieves an overview between the power of MIP-based bound tightening and practical scalability for large networks. By decomposing the process into parallelizable rolling windows and propagating improved bounds, OBBT-RH realizes near-global relaxation strength at a fraction of the computational resource requirement. This methodology enables complete verification or sharp under-approximations of adversarial robustness on network instances previously beyond reach by global methods or LP-based relaxations (Zhao et al., 2024). A plausible implication is that OBBT-RH may be adapted to other layered, piecewise linear architectures or verification properties, provided the windowing paradigm and parallel sub-MIP solves remain tractable.