Boundary-Guided Policy Optimization (BoPO)

• BoPO is a family of reinforcement-learning approaches that optimize policy improvement using explicit boundary objects rather than unconstrained reward objectives.
• In diffusion LLMs, BGPO replaces an ELBO-based objective with a memory-efficient lower bound, while in budget-aware routing, BoPO employs dual boundary policies and reference-guided advantages.
• Empirical results indicate improved sample efficiency, better likelihood approximation in BGPO, and effective cost-success trade-offs in BAAR’s budget-aware agentic routing.

Boundary-Guided Policy Optimization (BoPO) denotes, in recent arXiv usage, a family of reinforcement-learning procedures in which policy improvement is organized around an explicit boundary object rather than around an unconstrained reward objective alone. Two direct uses of the name are prominent in the current literature. In diffusion LLMs, “Boundary-Guided Policy Optimization” appears as BGPO, a memory-efficient on-policy surrogate for ELBO-based reinforcement learning that exploits a specially constructed lower bound with linearity and on-policy equivalence properties (Lin et al., 13 Oct 2025). In budget-aware agentic routing, BoPO names the reinforcement-learning stage of Budget-Aware Agentic Routing, where routing policies are optimized relative to two boundary policies—always-small and always-large—and are shaped by boundary-relative rewards plus a reference-guided advantage (Zhang et al., 4 Feb 2026). More broadly, adjacent work suggests a wider boundary-centered design space, including bounded Hilbert projections, adaptive clipping boundaries, Lyapunov barriers, and explicit bounded-ratio optima (Zixian, 24 Feb 2026, Ao et al., 20 Apr 2026).

1. Terminological scope and main usages

The term is not currently attached to a single universally standardized algorithm. In the papers considered here, it is used in two distinct technical settings, with several nearby formulations using different names but closely related intuitions.

Usage	Domain	Defining mechanism
BGPO (Lin et al., 13 Oct 2025)	RL for diffusion LLMs	Maximizes a lower bound of the ELBO-based objective that is linear over Monte Carlo samples and equivalent in value and gradient during on-policy training
BoPO in BAAR (Zhang et al., 4 Feb 2026)	Budget-aware agentic routing	Uses boundary policies, boundary-relative rewards, and reference-guided advantages to optimize sequential model routing

This multiplicity matters because the “boundary” is not the same object across usages. In BGPO, the operative boundary is a lower-bound surrogate over Monte Carlo likelihood estimates for diffusion models. In BAAR, the operative boundaries are behavioral anchors defined by two extreme routing policies, together with task-relative cost boundaries derived from those policies. Nearby literature extends the same general pattern to other objects: nonnegativity boundaries in Hilbert-space policy updates, reasoning-boundary rewards for “I DON’T KNOW” behavior, Lyapunov safe sets, or explicit lower and upper likelihood-ratio bounds (Zixian, 24 Feb 2026, Liu et al., 16 Jan 2026, Sikchi et al., 2021, Ao et al., 20 Apr 2026).

2. BGPO for diffusion LLMs

BGPO was proposed for reinforcement learning of diffusion LLMs, where exact sequence likelihoods are intractable because generation is iterative rather than left-to-right factorized. The paper writes the policy-ratio objective as

$$\mathcal{J}(\theta)=\mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)} A(x,y)=\mathbb{E}\,\mathcal{R}(x,y),$$

with

$$\mathcal{R}(x,y)=e^{\log\pi_\theta(y\mid x)-\log\pi_{\theta_{\text{old}}}(y\mid x)}A(x,y).$$

Because $\log \pi_\theta(y\mid x)$ is intractable for a dLLM, prior methods replace it with an ELBO

$$B_{\pi_\theta}(y \mid x) \triangleq \mathbb{E}_{t \sim \mathcal{U}[0,1],\, y_t \sim q(\cdot \mid t, y, x)} \ell_{\pi_\theta}(y_t, t, y \mid x),$$

and then estimate that ELBO by Monte Carlo. The central problem identified by BGPO is that the nonlinear dependence of the RL objective on the Monte Carlo average forces retention of all forward graphs across $n_t$ samples, producing memory growth with $n_t$ and thereby restricting feasible sample sizes (Lin et al., 13 Oct 2025).

BGPO replaces the ELBO-based objective with a specially constructed lower bound. For each Monte Carlo sample,

$$d_j = \ell_{\pi_\theta}(y_{t^{(j)}}, t^{(j)}, y \mid x) - \ell_{\pi_{\theta_{\text{old}}}}(y_{t^{(j)}}, t^{(j)}, y \mid x),$$

and

$g_j = \begin{cases} \frac{(1+d_j)A(x,y)}{n_t}, & A(x,y)\ge 0,\[4pt] \frac{e^{d_j}A(x,y)}{n_t}, & A(x,y)<0. \end{cases}$

The lower-bound objective is then

$$\hat{\mathcal{R}}_{\text{lb}}(x,y)=\sum_{j=1}^{n_t} g_j, \qquad \hat{\mathcal{J}}_{\text{lb}}(\theta)=\mathbb{E}\,\hat{\mathcal{R}}_{\text{lb}}(x,y).$$

The two key properties are the ones emphasized by the paper: linearity, because each $g_j$ depends only on a single Monte Carlo sample and can therefore be backpropagated independently with gradient accumulation; and equivalence, because under on-policy training, where $$\mathcal{R}(x,y)=e^{\log\pi_\theta(y\mid x)-\log\pi_{\theta_{\text{old}}}(y\mid x)}A(x,y).$$0 and hence $$\mathcal{R}(x,y)=e^{\log\pi_\theta(y\mid x)-\log\pi_{\theta_{\text{old}}}(y\mid x)}A(x,y).$$1 for all $$\mathcal{R}(x,y)=e^{\log\pi_\theta(y\mid x)-\log\pi_{\theta_{\text{old}}}(y\mid x)}A(x,y).$$2, the lower bound matches the ELBO-based objective in both value and gradient.

This construction turns the memory bottleneck into a sample-wise accumulable loss. The practical consequence is effectively constant memory with respect to $$\mathcal{R}(x,y)=e^{\log\pi_\theta(y\mid x)-\log\pi_{\theta_{\text{old}}}(y\mid x)}A(x,y).$$3, while compute still scales with the number of Monte Carlo samples. The paper presents this as the mechanism that allows larger $$\mathcal{R}(x,y)=e^{\log\pi_\theta(y\mid x)-\log\pi_{\theta_{\text{old}}}(y\mid x)}A(x,y).$$4, improved likelihood approximation, and better estimation of the RL objective. In practice it uses group-normalized advantages,

$$\mathcal{R}(x,y)=e^{\log\pi_\theta(y\mid x)-\log\pi_{\theta_{\text{old}}}(y\mid x)}A(x,y).$$5

and the final training loss

$$\mathcal{R}(x,y)=e^{\log\pi_\theta(y\mid x)-\log\pi_{\theta_{\text{old}}}(y\mid x)}A(x,y).$$6

Conceptually, the “boundary-guided” aspect of BGPO is therefore not a geometric trust-region boundary in ratio space; it is the use of a lower-bound surrogate that preserves the local on-policy optimization behavior of the ELBO objective while removing its multi-sample memory coupling. The paper’s own emphasis is memory efficiency, equivalence on-policy, and larger feasible Monte Carlo sample sizes rather than a projection-based boundary geometry.

3. BoPO in budget-aware agentic routing

In “Budget-Aware Agentic Routing via Boundary-Guided Training,” BoPO is the reinforcement-learning component of a broader framework for choosing at each step of a long-horizon agent trajectory whether to invoke a cheap/small model or an expensive/large one. The problem is formulated over trajectories

$$\mathcal{R}(x,y)=e^{\log\pi_\theta(y\mid x)-\log\pi_{\theta_{\text{old}}}(y\mid x)}A(x,y).$$7

with binary routing actions

$$\mathcal{R}(x,y)=e^{\log\pi_\theta(y\mid x)-\log\pi_{\theta_{\text{old}}}(y\mid x)}A(x,y).$$8

step costs $$\mathcal{R}(x,y)=e^{\log\pi_\theta(y\mid x)-\log\pi_{\theta_{\text{old}}}(y\mid x)}A(x,y).$$9, path dependence, sparse rewards, and task-level budget concerns (Zhang et al., 4 Feb 2026).

The distinctive element is the use of two boundary policies: $\log \pi_\theta(y\mid x)$0, which routes every step to the small model, and $\log \pi_\theta(y\mid x)$1, which routes every step to the large model. Each task is profiled by running both policies $\log \pi_\theta(y\mid x)$2 times, and the resulting success rates induce a three-way taxonomy: easy tasks, on which $\log \pi_\theta(y\mid x)$3 succeeds in at least $\log \pi_\theta(y\mid x)$4 out of $\log \pi_\theta(y\mid x)$5 runs; hard tasks, on which $\log \pi_\theta(y\mid x)$6 is unreliable but $\log \pi_\theta(y\mid x)$7 can succeed; and intractable tasks, on which even $\log \pi_\theta(y\mid x)$8 fails across all trials. These boundary policies are then used in three roles: supervision anchors for Boundary-Guided Supervised Fine-Tuning (BoSFT), normalization anchors for cost-aware reward shaping, and reference points for online policy optimization.

The training pipeline has two stages. BoSFT synthesizes routing trajectories. For easy and intractable tasks it uses the always-small trajectory. For hard tasks it samples $\log \pi_\theta(y\mid x)$9 mixed-routing trajectories with swept large-model probabilities

$$B_{\pi_\theta}(y \mid x) \triangleq \mathbb{E}_{t \sim \mathcal{U}[0,1],\, y_t \sim q(\cdot \mid t, y, x)} \ell_{\pi_\theta}(y_t, t, y \mid x),$$0

and selects the minimum-cost successful trajectory $$B_{\pi_\theta}(y \mid x) \triangleq \mathbb{E}_{t \sim \mathcal{U}[0,1],\, y_t \sim q(\cdot \mid t, y, x)} \ell_{\pi_\theta}(y_t, t, y \mid x),$$1 among the sampled successful set. This warm start is intended to avoid the degenerate cheap-failure mode that would otherwise dominate early RL.

BoPO then refines the router online with a GRPO-style objective, but the reward is boundary-relative. The normalized trajectory cost is defined by the lower and upper boundary-policy costs:

$$B_{\pi_\theta}(y \mid x) \triangleq \mathbb{E}_{t \sim \mathcal{U}[0,1],\, y_t \sim q(\cdot \mid t, y, x)} \ell_{\pi_\theta}(y_t, t, y \mid x),$$2

where

$$B_{\pi_\theta}(y \mid x) \triangleq \mathbb{E}_{t \sim \mathcal{U}[0,1],\, y_t \sim q(\cdot \mid t, y, x)} \ell_{\pi_\theta}(y_t, t, y \mid x),$$3

The appendix gives the reward parameters

$$B_{\pi_\theta}(y \mid x) \triangleq \mathbb{E}_{t \sim \mathcal{U}[0,1],\, y_t \sim q(\cdot \mid t, y, x)} \ell_{\pi_\theta}(y_t, t, y \mid x),$$4

Easy tasks are pushed toward frugality, hard tasks receive an additional subsidy for successful expensive reasoning, and intractable tasks are encouraged to fail cheaply. The paper also defines a reference-guided advantage that compares each sampled trajectory not only with the group mean but also with the reward of the BoSFT reference trajectory; the accompanying description states that a trajectory receives positive advantage only if it beats both the group baseline and the known reference level.

The formal distinction between soft and hard budgets is also central. Training is primarily soft-budget, with cost embedded in the reward:

$$B_{\pi_\theta}(y \mid x) \triangleq \mathbb{E}_{t \sim \mathcal{U}[0,1],\, y_t \sim q(\cdot \mid t, y, x)} \ell_{\pi_\theta}(y_t, t, y \mid x),$$5

Hard-budget deployment is handled separately by Budget-Constrained Decoding (BCD). The remaining budget is

$$B_{\pi_\theta}(y \mid x) \triangleq \mathbb{E}_{t \sim \mathcal{U}[0,1],\, y_t \sim q(\cdot \mid t, y, x)} \ell_{\pi_\theta}(y_t, t, y \mid x),$$6

and the decoding rule is

$$B_{\pi_\theta}(y \mid x) \triangleq \mathbb{E}_{t \sim \mathcal{U}[0,1],\, y_t \sim q(\cdot \mid t, y, x)} \ell_{\pi_\theta}(y_t, t, y \mid x),$$7

The paper explicitly presents this as a compromise: BoPO itself is trained with a soft-budget objective and a fixed risk profile controlled by $$B_{\pi_\theta}(y \mid x) \triangleq \mathbb{E}_{t \sim \mathcal{U}[0,1],\, y_t \sim q(\cdot \mid t, y, x)} \ell_{\pi_\theta}(y_t, t, y \mid x),$$8, while exact feasibility under strict budgets is enforced at inference time by BCD.

4. Boundary mechanisms in adjacent formulations

Several closely related papers do not use the name BoPO, but they sharpen what “boundary-guided” can mean technically. In GOPO, policy updates are lifted to the Hilbert space $$B_{\pi_\theta}(y \mid x) \triangleq \mathbb{E}_{t \sim \mathcal{U}[0,1],\, y_t \sim q(\cdot \mid t, y, x)} \ell_{\pi_\theta}(y_t, t, y \mid x),$$9 using the fluctuation variable

$n_t$0

so probability nonnegativity becomes the hard lower boundary

$n_t$1

The bounded Hilbert projection then yields the closed-form thresholding rule

$n_t$2

with exact sparsity: if $n_t$3, then $n_t$4 (Zixian, 24 Feb 2026). This is a mathematically explicit instance of a feasibility boundary governing the policy update.

In ETR, the boundary is the clipping interval itself. The static GRPO/PPO clipping radius is replaced by a dynamic trust-region width

$n_t$5

so positive-advantage samples receive looser boundaries and high-variance groups receive larger update budgets (Zhang et al., 7 Jan 2026). In Bounded Policy Optimization, the boundary is an explicit lower and upper likelihood-ratio interval,

$n_t$6

with analytic optimum

$n_t$7

and a monotonic-improvement theorem for the ideal bounded-ratio step (Ao et al., 20 Apr 2026). In Lyapunov Barrier Policy Optimization, the boundary is a certified safe set of policies and the update is repelled from its edge by a logarithmic barrier

$n_t$8

(Sikchi et al., 2021).

A different but still adjacent interpretation appears in Boundary-Aware Policy Optimization (BAPO) for agentic search, where the “boundary” is a reasoning limit. If all $n_t$9 sampled trajectories for a query have non-positive correctness reward, then an IDK rollout receives

$n_t$0

subject to a stage-level and sample-level adaptive modulator (Liu et al., 16 Jan 2026). This is not BoPO by name, but it extends the same pattern of policy optimization around an explicitly modeled boundary condition.

5. Empirical profile

The two explicit BoPO usages are evaluated in markedly different regimes.

System	Setting	Selected reported findings
BGPO (Lin et al., 13 Oct 2025)	RL for diffusion LLMs	Beats diffu-GRPO and VRPO-OL on all reported tasks; e.g. MATH500 $n_t$1 vs $n_t$2 for VRPO-OL, GSM8K $n_t$3 vs $n_t$4; uses $n_t$5 for most tasks and $n_t$6 for Sudoku
BoPO in BAAR (Zhang et al., 4 Feb 2026)	Long-horizon agentic routing	Improves the cost–success frontier; on AppWorld at $n_t$7, reaches $n_t$8, effectively matching always-large’s $n_t$9 while respecting budget; on SciWorld at $$d_j = \ell_{\pi_\theta}(y_{t^{(j)}}, t^{(j)}, y \mid x) - \ell_{\pi_{\theta_{\text{old}}}}(y_{t^{(j)}}, t^{(j)}, y \mid x),$$0, reaches $$d_j = \ell_{\pi_\theta}(y_{t^{(j)}}, t^{(j)}, y \mid x) - \ell_{\pi_{\theta_{\text{old}}}}(y_{t^{(j)}}, t^{(j)}, y \mid x),$$1 success rate using $$d_j = \ell_{\pi_\theta}(y_{t^{(j)}}, t^{(j)}, y \mid x) - \ell_{\pi_{\theta_{\text{old}}}}(y_{t^{(j)}}, t^{(j)}, y \mid x),$$2 of the budget

For BGPO, the central empirical claim is that larger Monte Carlo sample sizes improve approximation quality and performance, and that the lower-bound reformulation makes those larger sample sizes feasible. The ablation over $$d_j = \ell_{\pi_\theta}(y_{t^{(j)}}, t^{(j)}, y \mid x) - \ell_{\pi_{\theta_{\text{old}}}}(y_{t^{(j)}}, t^{(j)}, y \mid x),$$3 on math tasks reports MATH500 scores of $$d_j = \ell_{\pi_\theta}(y_{t^{(j)}}, t^{(j)}, y \mid x) - \ell_{\pi_{\theta_{\text{old}}}}(y_{t^{(j)}}, t^{(j)}, y \mid x),$$4 and $$d_j = \ell_{\pi_\theta}(y_{t^{(j)}}, t^{(j)}, y \mid x) - \ell_{\pi_{\theta_{\text{old}}}}(y_{t^{(j)}}, t^{(j)}, y \mid x),$$5 for $$d_j = \ell_{\pi_\theta}(y_{t^{(j)}}, t^{(j)}, y \mid x) - \ell_{\pi_{\theta_{\text{old}}}}(y_{t^{(j)}}, t^{(j)}, y \mid x),$$6, with a corresponding GSM8K trend of $$d_j = \ell_{\pi_\theta}(y_{t^{(j)}}, t^{(j)}, y \mid x) - \ell_{\pi_{\theta_{\text{old}}}}(y_{t^{(j)}}, t^{(j)}, y \mid x),$$7 and $$d_j = \ell_{\pi_\theta}(y_{t^{(j)}}, t^{(j)}, y \mid x) - \ell_{\pi_{\theta_{\text{old}}}}(y_{t^{(j)}}, t^{(j)}, y \mid x),$$8. The paper also reports a direct study of gradient standard deviation and bias using BGPO with $$d_j = \ell_{\pi_\theta}(y_{t^{(j)}}, t^{(j)}, y \mid x) - \ell_{\pi_{\theta_{\text{old}}}}(y_{t^{(j)}}, t^{(j)}, y \mid x),$$9 as a “golden gradient” approximation, and argues that its memory behavior is what allows these gains to be realized in practice (Lin et al., 13 Oct 2025).

For BAAR’s BoPO, the central empirical claims concern sequential budget allocation rather than policy-ratio estimation. The paper states that BoPO “consistently yields higher success rates at lower costs compared to all baselines,” and describes Figure 1 as showing the method pushing the frontier toward the top-left. The hard-budget results are explicitly framed as out-of-distribution generalization, because the policy is trained under a soft-budget objective and then constrained at inference. The allocation analysis is especially diagnostic: at similar total costs, vanilla RL spends $g_j = \begin{cases} \frac{(1+d_j)A(x,y)}{n_t}, & A(x,y)\ge 0,\[4pt] \frac{e^{d_j}A(x,y)}{n_t}, & A(x,y)<0. \end{cases}$0 of its cost on intractable cases, whereas BoPO allocates only $g_j = \begin{cases} \frac{(1+d_j)A(x,y)}{n_t}, & A(x,y)\ge 0,\[4pt] \frac{e^{d_j}A(x,y)}{n_t}, & A(x,y)<0. \end{cases}$1 there and spends $g_j = \begin{cases} \frac{(1+d_j)A(x,y)}{n_t}, & A(x,y)\ge 0,\[4pt] \frac{e^{d_j}A(x,y)}{n_t}, & A(x,y)<0. \end{cases}$2 on hard tasks, which the paper treats as the clearest validation of the boundary taxonomy and reward shaping (Zhang et al., 4 Feb 2026).

6. Misconceptions, limitations, and conceptual status

A common misunderstanding is to treat BoPO as a single algorithmic recipe. The available literature does not support that reading. BGPO and BAAR’s BoPO differ in state representation, action space, boundary object, and optimization mechanism. BGPO is a lower-bound construction for dLLM likelihood surrogates; BAAR’s BoPO is a GRPO-style routing optimizer grounded in two boundary policies and cost-normalized rewards. Closely related methods such as GOPO, BAPO, ETR, LBPO, and BPO are not introduced under the BoPO name, even though they are often conceptually adjacent (Lin et al., 13 Oct 2025, Zhang et al., 4 Feb 2026, Zixian, 24 Feb 2026, Liu et al., 16 Jan 2026).

Another misconception is to overgeneralize the strongest formal claims. In BGPO, equivalence is stated for on-policy training, where $g_j = \begin{cases} \frac{(1+d_j)A(x,y)}{n_t}, & A(x,y)\ge 0,\[4pt] \frac{e^{d_j}A(x,y)}{n_t}, & A(x,y)<0. \end{cases}$3 and therefore $g_j = \begin{cases} \frac{(1+d_j)A(x,y)}{n_t}, & A(x,y)\ge 0,\[4pt] \frac{e^{d_j}A(x,y)}{n_t}, & A(x,y)<0. \end{cases}$4 for all Monte Carlo samples. The paper does not present this as a global equality for arbitrary off-policy parameter settings. In BAAR, similarly, the training objective is primarily soft-budget; strict feasibility is imposed by BCD at inference time, and the authors explicitly note that directly training under hard constraints is unstable because the feasible set is combinatorial and informative feasible trajectories are rare (Lin et al., 13 Oct 2025, Zhang et al., 4 Feb 2026).

The reported limitations are also domain-specific. BGPO remains part of an approximation chain: the original RL objective is replaced by an ELBO estimator, and BGPO then optimizes a lower bound of that ELBO-based objective; the paper also notes that experiments are limited to 8B-scale models (Lin et al., 13 Oct 2025). BAAR requires substantial upfront profiling and data synthesis, is restricted to binary routing between one small and one large model, and learns a static risk profile because the training objective uses a fixed $g_j = \begin{cases} \frac{(1+d_j)A(x,y)}{n_t}, & A(x,y)\ge 0,\[4pt] \frac{e^{d_j}A(x,y)}{n_t}, & A(x,y)<0. \end{cases}$5 rather than directly optimizing a policy conditioned on the remaining budget state (Zhang et al., 4 Feb 2026). More broadly, the adjacent literature suggests that “boundary-guided policy optimization” is best read, at present, as a design principle rather than a single settled formalism: boundaries may appear as lower-bound surrogates, anchor policies, feasibility sets, clipping intervals, or explicit ratio caps, and the technical consequences depend on which of those objects is chosen.