Failure-Boundary Alignment (BPPO)

• Failure-Boundary Alignment (BPPO) is a method that replaces KL-based trust regions with overlap geometry, using the Bhattacharyya coefficient and Hellinger distance to enforce deterministic clipping of policy updates.
• The technique utilizes square-root likelihood ratio clipping to ensure that updates remain within a strict failure boundary, thereby preventing destabilizing large excursions in policy likelihood ratios.
• Empirical results show that BPPO achieves higher performance metrics, such as a near-unity mean likelihood ratio and improved IQM scores, outperforming conventional KL-based approaches in continuous control tasks.

Failure-boundary alignment in the context of Bhattacharyya-PPO (BPPO) refers to the principled bounding of policy likelihood-ratio updates by enforcing overlap between the old and new policy through square-root ratio clipping. This approach replaces conventional Kullback-Leibler (KL) trust regions with overlap geometry, leveraging the Bhattacharyya coefficient and the related Hellinger distance. The method deterministically defines and enforces “failure boundaries” on policy update steps, yielding robust control over rare, large likelihood-ratio excursions that can destabilize training.

1. Overlap Geometry: Bhattacharyya Coefficient and Hellinger Distance

The policy overlap at a fixed state $s$ is quantified by the Bhattacharyya coefficient:

$$B(\pi_{\text{old}}, \pi_\theta; s) = \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}.$$

The squared Hellinger distance serves as a measure of separation:

$$H^2(\pi_{\text{old}}, \pi_\theta; s) = 1 - B(\pi_{\text{old}}, \pi_\theta; s).$$

Averaging over the discounted state occupancy $d^{\pi_{\text{old}}}(s)$ leads to

$$\bar{B}(\pi_{\text{old}}, \pi_\theta) = \mathbb{E}_{s \sim d^{\pi_{\text{old}}}}\left[\sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}\right],$$

with average squared Hellinger distance $\bar H^2 = 1 - \bar B$. These quantities regulate the overlap between policies, directly impacting the magnitude of allowable update steps.

2. Trust-Region Formulation with Overlap Constraints

The canonical trust-region objective is replaced by a constraint on average overlap. The optimization problem is:

$$\begin{align*} \text{maximize}_\theta &\quad \mathbb{E}_{\text{old}}\left[ r_\theta(s,a)A_{\text{old}}(s,a) \right] \ \text{subject to} &\quad \mathbb{E}_{s \sim d^{\pi_{\text{old}}}}\left[ \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)} \right] \ge 1 - \delta, \end{align*}$$

where $$r_\theta(s,a) = \pi_\theta(a|s) / \pi_{\text{old}}(a|s)$$. The corresponding Lagrangian introduces a penalty for violating the overlap constraint. Algebraic manipulation enables a practical penalty term that relies on the squared deviation of the square-root ratio, producing a quadratic Hellinger/Bhattacharyya penalty in Bhattacharyya-TRPO (BTRPO). This form fundamentally differs from KL-based regularization, ensuring direct control over the worst-case deviations.

3. Square-root Likelihood Ratio and the BPPO Objective

The central construct is the square-root likelihood ratio:

$$q_\theta(s,a) = \sqrt{\frac{\pi_\theta(a|s)}{\pi_{\text{old}}(a|s)}} = \exp\left(\frac{1}{2} [\log \pi_\theta(a|s) - \log \pi_{\text{old}}(a|s)]\right).$$

The standard likelihood ratio is then $r_\theta = q_\theta^2$. For $$B(\pi_{\text{old}}, \pi_\theta; s) = \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}.$$0 near unity, a first-order Taylor expansion yields $$B(\pi_{\text{old}}, \pi_\theta; s) = \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}.$$1. This relationship provides the analytic basis for a Hellinger-weighted surrogate, crucial for stable policy updates.

Within BPPO, the clipped surrogate objective becomes:

$$B(\pi_{\text{old}}, \pi_\theta; s) = \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}.$$2

where $$B(\pi_{\text{old}}, \pi_\theta; s) = \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}.$$3 is the failure-boundary parameter in $$B(\pi_{\text{old}}, \pi_\theta; s) = \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}.$$4-space. This constrains $$B(\pi_{\text{old}}, \pi_\theta; s) = \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}.$$5 deterministically to $$B(\pi_{\text{old}}, \pi_\theta; s) = \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}.$$6, meaning $$B(\pi_{\text{old}}, \pi_\theta; s) = \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}.$$7 is bounded by $$B(\pi_{\text{old}}, \pi_\theta; s) = \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}.$$8.

4. Failure-boundary Alignment and Its Distinction from KL-based Methods

Traditional KL-based trust regions, as in TRPO or the KL-penalized PPO, enforce an average constraint:

$$B(\pi_{\text{old}}, \pi_\theta; s) = \sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}.$$9

which does not preclude rare, large excursions in $$H^2(\pi_{\text{old}}, \pi_\theta; s) = 1 - B(\pi_{\text{old}}, \pi_\theta; s).$$0. BPPO’s overlap geometry, by contrast, directly bounds $$H^2(\pi_{\text{old}}, \pi_\theta; s) = 1 - B(\pi_{\text{old}}, \pi_\theta; s).$$1 for all outcomes if $$H^2(\pi_{\text{old}}, \pi_\theta; s) = 1 - B(\pi_{\text{old}}, \pi_\theta; s).$$2, providing robust, pointwise guarantees. The choice $$H^2(\pi_{\text{old}}, \pi_\theta; s) = 1 - B(\pi_{\text{old}}, \pi_\theta; s).$$3 calibrates the failure boundary, with the resulting constraint $$H^2(\pi_{\text{old}}, \pi_\theta; s) = 1 - B(\pi_{\text{old}}, \pi_\theta; s).$$4 applying deterministically rather than in expectation (Trivedi et al., 6 Feb 2026).

Empirical characterization reveals BPPO maintains a near-unity mean $$H^2(\pi_{\text{old}}, \pi_\theta; s) = 1 - B(\pi_{\text{old}}, \pi_\theta; s).$$5 across samples while retaining a nontrivial upper tail, opposing the steady collapse to unity seen in KL-based PPO. This suggests that overlap-constrained methods avert the starvation of productive policy updates that can afflict mean-focused policy trust regions.

5. Theoretical Guarantees and Worst-case Boundaries

The Hellinger distance guarantees that, if $$H^2(\pi_{\text{old}}, \pi_\theta; s) = 1 - B(\pi_{\text{old}}, \pi_\theta; s).$$6,

$$H^2(\pi_{\text{old}}, \pi_\theta; s) = 1 - B(\pi_{\text{old}}, \pi_\theta; s).$$7

yielding

$$H^2(\pi_{\text{old}}, \pi_\theta; s) = 1 - B(\pi_{\text{old}}, \pi_\theta; s).$$8

In BPPO, setting the clipping width $$H^2(\pi_{\text{old}}, \pi_\theta; s) = 1 - B(\pi_{\text{old}}, \pi_\theta; s).$$9 enforces this as a hard failure boundary. These pointwise bounds on likelihood ratios ensure that no individual update step’s stochastic deviation can exceed those limits, sharply contrasting with the looser average bounds of KL-based constraints.

6. Algorithmic Procedure for BPPO

The BPPO algorithm proceeds as follows:

1. Initialize policy parameters $d^{\pi_{\text{old}}}(s)$0.
2. Iterate:
   • Collect $d^{\pi_{\text{old}}}(s)$1 transitions $d^{\pi_{\text{old}}}(s)$2 using the current policy, storing $d^{\pi_{\text{old}}}(s)$3.
   • Compute advantages $d^{\pi_{\text{old}}}(s)$4.
   • For each sample, compute $d^{\pi_{\text{old}}}(s)$5 and $d^{\pi_{\text{old}}}(s)$6.
   • Update parameters $d^{\pi_{\text{old}}}(s)$7 to maximize

   $d^{\pi_{\text{old}}}(s)$8

• Optionally update value-function parameters.

This procedure ensures that each gradient step enforces the overlap constraint by deterministically clipping the square-root likelihood ratio (Trivedi et al., 6 Feb 2026).

7. Empirical Characterization

In continuous control benchmarks (MuJoCo suite), BPPO attains the highest Interquartile Mean (IQM) in 5 out of 6 tasks, with an overall Mean IQM of 1877.5, outperforming PPO’s 1405.0 under identical training conditions. Tail diagnostic analysis on Humanoid-v5 demonstrates that BPPO maintains a stable upper percentile of $d^{\pi_{\text{old}}}(s)$9, whereas PPO’s $$\bar{B}(\pi_{\text{old}}, \pi_\theta) = \mathbb{E}_{s \sim d^{\pi_{\text{old}}}}\left[\sum_a \sqrt{\pi_{\text{old}}(a|s)} \sqrt{\pi_\theta(a|s)}\right],$$0 percentile collapses to unity, indicating loss of meaningful update magnitude. Across RLiable reporting metrics (IQM, Optimality Gap), both BTRPO and BPPO outperform their KL-based analogues on continuous control and DM Control tasks. On Procgen, BPPO remains competitive, especially in exploration-constrained games, despite mixed results overall (Trivedi et al., 6 Feb 2026).

These results support the interpretation that deterministic, overlap-based failure-boundary constraints provide a practical, stable, and update-efficient alternative to KL-based trust regions, without sacrificing policy improvement capacity.