Linear Bound Propagation (LBP)

• Linear Bound Propagation (LBP) is a method for computing rigorous upper and lower bounds by propagating affine relaxations through decomposed system components.
• It decomposes complex systems into substructures and applies local linear programming relaxations with message-passing to iteratively refine global bounds.
• LBP finds applications in probabilistic inference, neural network verification, and local Lipschitz estimation, balancing scalability with tight approximations.

Linear Bound Propagation (LBP) is a family of techniques for computing rigorous upper and lower bounds on relevant functions of interest—such as marginals in graphical models, output ranges of neural networks, or Lipschitz constants—by systematically propagating linear relaxations through a computational graph or probabilistic structure. By replacing intractable constraints with tractable (often affine) surrogates and exchanging side information between overlapping subproblems, LBP achieves scalability and correctness guarantees that encompass both classical message-passing and modern neural network verification frameworks.

1. Foundational Principles of Linear Bound Propagation

At its core, Linear Bound Propagation leverages the structural decomposition of a high-dimensional probabilistic or computational system into manageable components. For a graphical model, LBP constructs local tractable LPs for small clusters of variables, propagating upper and lower bounds on marginal probabilities between overlapping clusters and iteratively refining these until convergence (Kappen et al., 2011). In neural networks, LBP propagates affine over- and under-estimators through each layer (affine transformation, activation, etc.), resulting in global bounds for the network output given an input region (Vertovec et al., 9 Nov 2025).

The essential steps are:

• Identification of substructures (clusters in graphical models, layers in neural networks)
• Local relaxation: Replacing intractable functions (e.g., ReLU, nonlinear potentials) by tightest possible affine over/under-approximations
• Propagation: Combining (via LP operations or matrix algebra) local relaxations across the structure, producing increasingly precise global bounds
• Message-passing or bound-passing: Neighboring subproblems share constraint information, tightening bounds on their respective domains.

LP (linear programming) is the computational primitive. Constraints and objectives are constructed so that the resulting bound holds for all possible configurations within prescribed relaxation parameters, providing both scalability and rigorous guarantees.

2. Formulations: Graphical Models and Neural Networks

Graphical Models

For probabilistic graphical models, the LBP method (sometimes known as "bound propagation") introduces the following elements (Kappen et al., 2011):

• For a cluster $C$ with center nodes $C_\mathrm{mar}$, separator nodes $C_\mathrm{sep}$, and optional other nodes $C_\mathrm{oth}$, bounds $L_C(x_C) \leq p(x_C) \leq U_C(x_C)$ are maintained for all joint states $x_C$.
• Defining an auxiliary distribution $q(x_\mathrm{sep})$ over the separators, the marginal $p(x_\mathrm{mar})$ is linear in $q$:

$$p(x_\mathrm{mar}) = \sum_{x_\mathrm{sep}, x_\mathrm{oth}} P(x_\mathrm{mar}, x_\mathrm{oth}\mid x_\mathrm{sep}) q(x_\mathrm{sep})$$

• The optimal (tightest) bounds are obtained by solving two LPs (maximize or minimize the objective above) over feasible $q$, with constraints arising from previously computed bounds on adjacent clusters.
• Iterative message-passing refines all bounds until convergence, with theoretical guarantees of correctness and convergence for arbitrary decomposed graphs.

Neural Networks

In feedforward or control-affine neural networks, LBP proceeds via layer-wise relaxations (Vertovec et al., 9 Nov 2025, Shi et al., 2022):

• At each layer $i$, with pre-activations $y_i$ and activations $z_i = \sigma(y_i)$ over input region $\Delta$, one computes affine (piecewise-linear) upper and lower relaxations:

$$\underline{G}_m^{(i)} y_{i,m} + \underline{g}_m^{(i)} \leq \sigma(y_{i,m}) \leq \overline{G}_m^{(i)} y_{i,m} + \overline{g}_m^{(i)}$$

• Propagate these through subsequent affine and activation transformations, splitting weight matrices into positive and negative parts to ensure monotonicity of bounds.
• For gradient (Jacobian) bounds, propagate affine relaxations of each activation's derivative, employing McCormick relaxation for bilinear products encountered in the chain rule.
• The result is two affine functions $$\underline{\mathtt{NN}}(x), \overline{\mathtt{NN}}(x)$$ satisfying $$\underline{\mathtt{NN}}(x) \leq \mathtt{NN}(x) \leq \overline{\mathtt{NN}}(x)$$ for all $x\in\Delta$.

3. Algorithmic Procedures and Relaxations

A key attribute of LBP is its ability to handle nonlinearities and couplings through systematically derived affine envelopes.

Piecewise-Linear Relaxations

For activation functions $\sigma$, the tightest affine over- and under-approximations are constructed elementwise over the relevant intervals. For ReLU, leaky-ReLU, sigmoid, and tanh, such envelopes can be computed analytically.

McCormick Relaxations

Products of two bounded variables $a\cdot b$ are relaxed using McCormick inequalities: $$\begin{aligned} a \cdot b &\geq a_\mathrm{min}\, b + a\, b_\mathrm{min} - a_\mathrm{min}\, b_\mathrm{min} \ a \cdot b &\geq a_\mathrm{max}\, b + a\, b_\mathrm{max} - a_\mathrm{max}\, b_\mathrm{max} \ a \cdot b &\leq a_\mathrm{min}\, b + a\, b_\mathrm{max} - a_\mathrm{min}\, b_\mathrm{max} \ a \cdot b &\leq a_\mathrm{max}\, b + a\, b_\mathrm{min} - a_\mathrm{max}\, b_\mathrm{min} \end{aligned}$$ Tight bounds are formed by convex-combining the two valid lower and upper bounds, optimizing an internal parameter $\eta \in [0,1]$.

Adaptive Region Refinement

To reduce conservatism, the input region $\Delta$ (polytope, simplex, or box) is adaptively partitioned—along its largest edge in the case of simplices—focusing computational effort in regions where the bounds are weakest (e.g., near decision boundaries or invariance margins).

GPU-Batched Computation

Modern LBP implementations vectorize and batch the bound computations across many regions, leveraging GPU parallelism for scalability in high-dimensional or large-model scenarios (Vertovec et al., 9 Nov 2025).

4. Specializations and Tightening via Norm Coupling

Single-Parameter $\ell_2$-Coupling (SDP-CROWN)

SDP-CROWN demonstrates that, for neural networks with $\ell_2$-ball input constraints, the tight convex SDP relaxation's inter-neuron coupling can be reduced to a single scalar parameter $\lambda$ per layer (Chiu et al., 7 Jun 2025): $$c^T \mathrm{ReLU}(x) \geq g^T x + h(g, \lambda) \qquad \forall x \in B_2(\hat{x}, \rho)$$ with

$$h(g, \lambda) = -\frac{1}{2}\left[\lambda(\rho^2 - \|\hat{x}\|_2^2) + \frac{1}{\lambda} \|\phi(g, \lambda)\|_2^2 \right]$$

Here, $$\phi_i(g,\lambda) = \min\left(c_i - g_i - \lambda \hat{x}_i,\; g_i + \lambda \hat{x}_i,\; 0\right)$$. The optimal offset for output bounds, in the zero-centered case, shifts from the classical $\ell_1$-norm form to an $\ell_2$-norm, yielding up to a $\sqrt{n}$-factor improvement in tightness over per-neuron triangle relaxations. This reduction enables practical scaling to large neural networks with minor computational overhead.

Integration with Existing Verification Frameworks

In backward LiRPA-style verifiers, the only modification required to exploit SDP-CROWN is replacing the classical scalar offset $d^{(k)}_\mathrm{old}$ with the single-parameter offset $d^{(k)}$ involving a maximization over $\lambda$. This preserves the existing machinery while inheriting the improved tightness of the SDP-derived relaxation.

5. Applications Across Domains

Graphical Model Marginal Estimation

In large Bayesian networks and undirected graphical models (e.g., Ising models), LBP has provided tight probabilistic bounds where exact inference is infeasible. Empirically, bounds on marginals become increasingly sharp as separator cluster size increases, with convergence in a modest number of iterations (Kappen et al., 2011). In networks with moderate Markov blanket sizes, LBP yields rigorous confidence intervals at costs competitive with standard approximate inference.

Neural Network Verification

For neural control barrier function (CBF) verification, LBP enables scalable computation of piecewise-linear output and gradient bounds over regions of the input state-space (Vertovec et al., 9 Nov 2025):

• Verified CBF conditions are reduced to feasibility problems for systems of affine inequalities.
• McCormick relaxation enables tractable treatment of all bilinear (gradient-system term) constraints.
• Adaptive simplex refinement focuses computational resources around critical safety boundaries, supporting networks orders of magnitude larger than prior SMT/MIP-based approaches.

Local Lipschitz Estimation

LBP, when supplied with specifically derived tight relaxations for absolute value, activation, and their derivatives, provides efficient upper bounds on local Lipschitz constants of neural networks (Shi et al., 2022). By using high-order backward computational graphs and integrating Branch-and-Bound procedures, it achieves significant scalability and tightness over prior relaxation-based or exact mixed-integer programming approaches.

6. Complexity, Scalability, and Performance

Complexity and scalability characteristics depend on problem structure. In graphical models, cost per LP is exponential in separator size but polynomial in the number of clusters and converges rapidly in practice (Kappen et al., 2011). In neural network verification:

• Each LBP pass is $O(LN)$ for output bounds and up to $O(L^2 N^2)$ for gradient bounds per region, where $L$ is network depth and $N$ is width (Vertovec et al., 9 Nov 2025).
• Empirical runtimes: 64 × 64 tanh networks complete in ≈1.7 s, [128,256,128] CBF networks in 145 s, compared to SMT-based times of 635 s or timeouts.
• Integration with GPU-accelerated batched matrix operations yields near-linear per-region cost and facilitates region/branch batching in adaptive refinement or BaB frameworks.
• Classical full-SDP verifiers become intractable above ≈$10^4$ neurons, whereas SDP-CROWN and LBP variants scale to $10^5$ neurons and $10^6$ parameters.

A summary table of empirical performance for neural verification is as follows:

Method	Verified Accuracy (Large Model)	Runtime Scaling
$\alpha$-CROWN	<10%	Baseline
SDP-CROWN (LBP + SDP)	>60%	$1.5\times$–$3\times$ Baseline
Full SDP Solver	Timeout	Infeasible
Tightest LP Relaxation	20–30% lower than SDP-CROWN	Hours

7. Limitations, Interpretations, and Outlook

LBP-based frameworks balance scalability and tightness—a direct tradeoff determined by the granularity of relaxations and the structure of local regions (cluster size, input simplex size). As separator size grows (graphical models) or region refinement deepens (NN domains), bounds become tighter, but computational cost increases. In networks, the adaptation of single-parameter norm-couplings (as in SDP-CROWN) indicates that almost all of the tightness of rich SDP relaxations can be retained with negligible resource overhead, conditional on the expressivity of the single coupling parameter per layer.

LBP unifies several strands of research in probabilistic inference and neural verification, generalizing message-passing, triangle and polyhedral relaxations, and norm-based coupling. Its extensions to gradient bounds, nonstandard activations, and global Lipschitz estimation are well-documented and continue to be an area of active research, particularly for high-dimensional or safety-critical systems.

A plausible implication is that future research directions may further exploit the balance between complexity and tightness, possibly via hybrid relaxations, higher-order region partitioning, or learning-based adaptation of propagation parameters.