PBSO in Reflective Autoformalization

• PBSO is a reinforcement learning objective that assigns distinct rewards to final formal statements and intermediate critiques in reflective autoformalization.
• It employs a clipped, discounted prospective return to achieve stable, position-specific credit assignment throughout the generation-and-correction process.
• PBSO enhances semantic consistency in autoformalization benchmarks, significantly outperforming baseline methods in evaluating formal statement fidelity.

Prospective Bounded Sequence Optimization (PBSO) is a reinforcement-learning objective introduced as the training mechanism for ReForm, a reflective autoformalization framework that alternates between generating a Lean formal statement and self-validating its semantic fidelity to a natural-language mathematical problem. In this setting, the model must learn not only to output a final formalization that is syntactically valid and semantically consistent, but also to produce intermediate critiques that are themselves faithful and diagnostically useful. PBSO addresses this by assigning different rewards to different positions in the reflective sequence and propagating them through a clipped discounted return, yielding position-specific credit assignment for both generation and critique tokens (Chen et al., 28 Oct 2025).

1. Problem setting and motivation

PBSO is designed for a training regime in which a model produces a trajectory rather than a single answer. The core difficulty is that ReForm requires two distinct behaviors within one sequence: a primary task and an auxiliary task. The primary task is to produce a final formal statement that is both syntactically correct, as measured by PassesLean, and semantically consistent with the original natural-language problem, as measured by IsConsistent. The auxiliary task is to produce faithful semantic critiques at intermediate reflection steps, denoted IsFaithfulCritique (Chen et al., 28 Oct 2025).

The paper frames this as a failure mode of treating autoformalization as a simplistic translation problem. LLMs can often generate syntactically well-formed formal statements, but they frequently fail to preserve semantic intent. In the ReForm setting, this problem is compounded by the need to interleave statement generation with self-evaluation. A single terminal reward is therefore insufficient, because the model can obtain credit for the final answer without learning how to diagnose semantic mismatches in the intermediate steps.

The stated concern is degeneration of the critique channel into empty critiques, generic comments, hallucinated errors, or premature claims that the formalization is already correct. PBSO is introduced specifically to prevent this collapse by supplying direct supervision to the intermediate validation process rather than only to the terminal answer (Chen et al., 28 Oct 2025).

2. Reflective trajectory in ReForm

For a question $Q$, ReForm maintains a history

$$\mathcal{H}_t = \{(S_1, C_1), \ldots, (S_{t-1}, C_{t-1})\}.$$

At iteration $t$, the model first generates an autoformalization

$S_t = \pi(Q, \mathcal{H}_t),$

with the first step reducing to standard one-pass autoformalization,

$S_1 = \pi(Q).$

It then produces a semantic self-validation

$C_t = \pi(Q, \mathcal{H}_t, S_t),$

where $C_t$ evaluates whether $S_t$ faithfully captures the problem statement (Chen et al., 28 Oct 2025).

This loop repeats until the statement is semantically faithful or a stopping limit is reached. The trajectory therefore contains multiple generation steps $S_t$, multiple critique steps $C_t$, and a final answer. PBSO is the RL mechanism used to train this full generation-and-correction process rather than only the last output.

A plausible implication is that PBSO treats reflection not as free-form chain-of-thought but as a supervised latent subtask whose correctness matters independently of terminal success. This distinguishes the framework from sequence-level objectives that optimize only final outputs.

3. Heterogeneous reward design

PBSO uses heterogeneous rewards at different positions in the sequence. The final formal statement, denoted Ans, receives a task reward only if it satisfies both Lean validity and semantic correctness: $$\mathcal{H}_t = \{(S_1, C_1), \ldots, (S_{t-1}, C_{t-1})\}.$$0 Here, PassesLean checks syntactic correctness via the Lean compiler, and IsConsistent checks semantic fidelity using an LLM-based judge (Chen et al., 28 Oct 2025).

Each intermediate critique $$\mathcal{H}_t = \{(S_1, C_1), \ldots, (S_{t-1}, C_{t-1})\}.$$1 receives an auxiliary reward: $$\mathcal{H}_t = \{(S_1, C_1), \ldots, (S_{t-1}, C_{t-1})\}.$$2 This reward is defined to test whether the critique correctly identifies semantic mismatches, including false positives, false negatives, and premature termination.

The significance of this design is explicit in the paper: PBSO does not reward merely sounding reflective. It rewards being correct about the reflection. This means that the critique stream is optimized as an operational component of the algorithm, not as stylistic scaffolding around the final answer.

4. Prospective bounded return and position-specific credit assignment

The central mechanism of PBSO is a future-looking return computed at every step rather than only at sequence termination. For a trajectory with rewards

$$\mathcal{H}_t = \{(S_1, C_1), \ldots, (S_{t-1}, C_{t-1})\}.$$3

the method defines a bounded backward return

$$\mathcal{H}_t = \{(S_1, C_1), \ldots, (S_{t-1}, C_{t-1})\}.$$4

where $$\mathcal{H}_t = \{(S_1, C_1), \ldots, (S_{t-1}, C_{t-1})\}.$$5 is the discount factor, $$\mathcal{H}_t = \{(S_1, C_1), \ldots, (S_{t-1}, C_{t-1})\}.$$6, and $$\mathcal{H}_t = \{(S_1, C_1), \ldots, (S_{t-1}, C_{t-1})\}.$$7 bounds the return to the reward range (Chen et al., 28 Oct 2025).

The clipping operation is described as important for stability because it prevents reward accumulation from exploding over long reflective trajectories. The return at step $$\mathcal{H}_t = \{(S_1, C_1), \ldots, (S_{t-1}, C_{t-1})\}.$$8 is interpreted as the composite reward for the $$\mathcal{H}_t = \{(S_1, C_1), \ldots, (S_{t-1}, C_{t-1})\}.$$9-th reflective iteration, evaluating the generated statement $t$0, the critique $t$1, and that iteration’s contribution to eventual success. This is the basis for the term “prospective”: each step is scored by its future impact.

After bounded returns are computed, PBSO normalizes them into standardized advantages over a batch of sampled trajectories. For $t$2 sampled trajectories, where trajectory $t$3 has length $t$4,

$t$5

with

$t$6

All returns from all trajectories are normalized together, and every token in iteration $t$7 receives the same advantage $t$8.

This yields position-specific credit assignment. Early iterations can be strongly rewarded if they identify major issues; later iterations can receive higher or lower credit depending on their actual contribution; critique tokens are not treated the same as final-answer tokens. In the paper’s terminology, this is the “bounded sequence optimization” component of PBSO (Chen et al., 28 Oct 2025).

5. Training pipeline and relation to GRPO

The paper describes a two-stage pipeline. The first stage is supervised fine-tuning (SFT), for which reflective trajectories are constructed using a multi-agent system. The reported procedure is: generate an initial Lean formalization, check syntactic validity, check semantic consistency, generate a correction using feedback if inconsistent, and repeat up to 3 rounds. These trajectories provide the data used for SFT (Chen et al., 28 Oct 2025).

The second stage is reinforcement learning with PBSO. The described RL procedure is to start with a diverse batch of math problems, sample 8 candidate formalizations per problem from the SFT model, evaluate each sample with compiler verification and semantic consistency, build a 2048-item RL dataset stratified by difficulty / pass rate, and optimize with PBSO + GRPO. The implementation details reported are an RL learning rate of $t$9, batch size 32, group size per question 16, temperature 1.0, top-p 0.95, KL coefficient 0.0, and entropy coefficient 0.0.

PBSO is not presented as a replacement for policy-gradient training. Rather, the paper states that PBSO updates the policy using the position-specific advantages with standard GRPO. PBSO therefore defines the reward and advantage structure, while GRPO performs the policy update. This suggests a modular division between reward shaping and optimization algorithm: the novelty lies in how the reflective sequence is scored and credited, not in replacing the underlying policy-gradient machinery.

6. Empirical behavior, ablations, and scope

The paper reports that ReForm achieves an average improvement of 17.2 percentage points over the strongest baselines across four autoformalization benchmarks, and PBSO is identified as the RL mechanism that trains the reflective loop underlying that result (Chen et al., 28 Oct 2025). The contribution of PBSO is further isolated through ablations.

Removing clipping (w/o clip) yields semantic consistency drops from full ReForm performance on all four listed benchmarks: miniF2F $S_t = \pi(Q, \mathcal{H}_t),$0, ProofNet $S_t = \pi(Q, \mathcal{H}_t),$1, Putnam $S_t = \pi(Q, \mathcal{H}_t),$2, and AIME2025 $S_t = \pi(Q, \mathcal{H}_t),$3. Removing the auxiliary reward (w/o r_aux) especially affects harder tasks, with Putnam $S_t = \pi(Q, \mathcal{H}_t),$4 and AIME2025 $S_t = \pi(Q, \mathcal{H}_t),$5. Removing the RL phase entirely (w/o RL) also degrades performance: miniF2F $S_t = \pi(Q, \mathcal{H}_t),$6, ProofNet $S_t = \pi(Q, \mathcal{H}_t),$7, Putnam $S_t = \pi(Q, \mathcal{H}_t),$8, and AIME2025 $S_t = \pi(Q, \mathcal{H}_t),$9.

The reported training dynamics are also consistent with the intended behavior of PBSO. Training reward increases steadily from 0.30 to 0.47, response length grows from 2,300 to 4,800 tokens, and held-out Putnam semantic consistency improves from 47% to 57%. The paper further states that after RL, the model explores deeper iteration depths, including 4- and 5-step reflective trajectories not seen in SFT. This suggests that PBSO does not merely optimize short-horizon corrections but can induce longer reflective behaviors when they are beneficial (Chen et al., 28 Oct 2025).

A broader contextual point in the paper is the introduction of ConsistencyCheck, a benchmark of 859 expert-annotated items used to validate LLMs as judges and to show that autoformalization is intrinsically difficult: even human experts produce semantic errors in up to 38.5% of cases. Within that context, PBSO can be understood as a response to a hard evaluation problem as much as to a hard generation problem. It trains a model not only to formalize mathematics, but also to assess whether its own formalization preserves meaning.

7. Conceptual significance and limitations

In one-sentence form, PBSO is defined as ReForm’s RL objective that assigns terminal reward for final semantic correctness and intermediate rewards for faithful critiques, then propagates them through a clipped discounted prospective return to produce position-specific advantages, thereby enabling stable training of the reflective generation-and-correction loop without collapsing into superficial self-validation (Chen et al., 28 Oct 2025).

The main misconception addressed by the formulation is that reflection can be optimized indirectly through final-answer reward alone. The paper argues the opposite: if intermediate validation steps receive no direct signal, the model can satisfy the outer objective while generating critiques that are empty, generic, hallucinated, or prematurely terminating. PBSO is therefore designed to prevent “appearing to validate” from substituting for actual semantic diagnosis.

At the same time, PBSO remains tightly scoped. Its claims are tied to the ReForm setting, where the sequence alternates between candidate formalizations and semantic critiques and where correctness is defined through PassesLean, IsConsistent, and IsFaithfulCritique. Any broader transfer to other reflective RL settings would be an inference rather than a stated result. What the paper establishes directly is that, in reflective autoformalization, reward heterogeneity across sequence positions is necessary for aligning intermediate critique behavior with final semantic correctness.