Post-Completion Learning: Reflective Training

• Post-Completion Learning is a training paradigm that extends traditional supervision by using a post-answer segment for self-assessment.
• It divides the sequence into reasoning and reflection regions, allowing the model to evaluate its own answers and predict rewards.
• Empirical evidence shows that PCL enhances performance on reasoning datasets by integrating dual-track supervision and white-box RL without extra inference overhead.

Post-Completion Learning (PCL) is a training paradigm for LLMs that extends supervision beyond the model’s normal answer boundary. Instead of terminating learning at <eos>, it replaces the usual end token with <post-completion>, allows the model to continue into a structured self-evaluation and reward-prediction segment, and supervises that post-completion region during training while still stopping generation at the completion boundary during inference (Fei et al., 27 Jul 2025). In this formulation, the previously ignored sequence space after answer completion becomes a source of learning signal for reasoning quality, self-evaluation, reward prediction, and quality awareness or calibration.

1. Conceptual definition and motivation

PCL is built around the claim that standard language-model training wastes information at the moment generation reaches <eos>. In conventional setups, the model is trained on the reasoning trace and answer, but not on whether the answer is correct, whether it satisfies the required format, whether the model can explain its own score, or whether it can compute the reward function that would otherwise remain external. PCL treats this omitted region as the post-completion space and uses it for explicit self-assessment (Fei et al., 27 Jul 2025).

The framework divides the target sequence into two functional regions. The reasoning region contains think + answer, and the reflection region contains evaluation + reward. This distinction is central. The method does not merely append extra text; it assigns different supervisory roles to the two regions. The paper characterizes the intended shift as moving from “passive reward acceptance” to “active self-evaluation.”

The motivation is analogous to post-task reflection. After producing an answer, the model is trained to inspect that answer, analyze whether it is correct, and predict reward-relevant scores. The post-completion segment is therefore not user-facing output in deployment, but a training scaffold for self-assessment. A plausible implication is that PCL attempts to internalize part of the reward-evaluation process into the base policy itself, rather than leaving reward interpretation entirely to external supervision.

2. Sequence formulation and training pipeline

The canonical PCL target sequence is

$$\text{question} \rightarrow \text{think} \rightarrow \text{answer} \rightarrow \langle \text{post-completion} \rangle \rightarrow \text{evaluation} \rightarrow \text{reward}.$$

This sequence design introduces a hard boundary between visible completion and reflective continuation. During training, the model continues after <post-completion>; during inference, decoding stops when <post-completion> is emitted, so the reflection region is not generated for the user (Fei et al., 27 Jul 2025).

The training pipeline has three coordinated stages. In reasoning SFT, the model is trained on ordinary supervised reasoning data of the form (question, think, answer), except that the original <eos> is replaced by <post-completion>. Loss is applied only on the think + answer region. In evaluation SFT, a teacher model generates full PCL-format sequences, and the student is trained to continue from think + answer + <post-completion> into evaluation + reward; here the loss is applied only on the reflection region. In RL over complete PCL sequences, the model samples full sequences including the post-completion segment and is optimized with reward functions defined over answer correctness, format correctness, and self-evaluation consistency.

Teacher data construction is itself structured. The paper states that teacher generation uses GPT-4.1 with in-context demonstrations of the PCL format. Generated samples are then automatically validated by recomputing reward scores, and examples are kept when the teacher correctly evaluates its own answer, even if the answer itself is wrong. This matters because the evaluation track is meant to teach evaluation skill, not merely to duplicate answer supervision.

The inference-time consequence is one of the method’s defining properties. The post-completion region is learned but not deployed. The model stops at <post-completion>, which replaces <eos> as the stop token. The paper explicitly claims zero additional inference overhead in this setting, because the reflective tokens are not decoded at test time.

3. White-box reinforcement learning and reward alignment

The paper describes its RL component as white-box reinforcement learning. The distinction is that the model is not only rewarded by externally computed scores; it is also required to explicitly compute and verbalize the reward logic in the post-completion region, then align its predicted reward with the true reward (Fei et al., 27 Jul 2025).

Three reward functions are defined. The accuracy reward is

$$R_a = \mathbf{1}[\text{answer} = \text{ground\_truth}],$$

the format reward is

$$R_f = \mathbf{1}[\text{contains\_all\_sections}(\text{output})],$$

and the consistency reward measures agreement between self-predicted and externally computed reward:

$$R_c = 1 - \left| R_{\text{pred}} - R_{\text{true}} \right|_1.$$

The format reward is explicitly modified to require all four sections—think, answer, evaluation, and reward—so the RL stage supervises the full structured output, not only the visible answer. The consistency reward is the core of the white-box design: the model is rewarded when its predicted score matches the actual reward-function output.

The RL objective is given as

$$\mathcal{L}_{\text{RL}} = -\mathbb{E}_{\pi_\theta}\big[(R_a + R_f + R_c)\log \pi_\theta(a \mid s)\big].$$

The implementation uses GRPO, samples 8 responses per prompt as a group, and retains the default KL constraint with coefficient $\beta = 0.04$. The central alignment rule is straightforward: compute external reward $R_{\text{true}}$, ask the model to produce $R_{\text{pred}}$ in the post-completion reward section, and reward the model for minimizing the discrepancy between them.

This design makes the post-completion segment more than an auxiliary explanation. It becomes a structured interface through which the policy represents the reward rules that are simultaneously used to optimize it.

4. Dual-track supervision and hybrid optimization

PCL organizes supervised learning into two distinct tracks. The first is reasoning-track SFT, which trains the model to solve the task; the second is evaluation-track SFT, which trains it to assess the completed solution. The losses are region-specific (Fei et al., 27 Jul 2025):

$$\mathcal{L}_{\text{SFT1}} = -\sum_{t \in \{\text{think}, \text{answer}\}} \log P(x_t \mid x_{<t}, \text{question}),$$

$$\mathcal{L}_{\text{SFT2}} = -\sum_{t \in \{\text{evaluation}, \text{reward}\}} \log P(x_t \mid x_{<t}, \text{context}),$$

where the context for the second track includes the completed solution through <post-completion>.

The combined objective is additive:

$$R_a = \mathbf{1}[\text{answer} = \text{ground\_truth}],$$0

The paper emphasizes concurrent rather than sequential optimization. Reasoning SFT, evaluation SFT, and RL are mixed within the same batch. The reported training recipe keeps the reasoning SFT : evaluation SFT ratio at 1:1, the RL : SFT ratio at 1:1, and trains for 2 epochs, with other hyperparameters following OpenR1 defaults.

A theoretical intuition is also provided. The paper defines an extended target sequence

$$R_a = \mathbf{1}[\text{answer} = \text{ground\_truth}],$$1

with $$R_a = \mathbf{1}[\text{answer} = \text{ground\_truth}],$$2 the reasoning sequence, $$R_a = \mathbf{1}[\text{answer} = \text{ground\_truth}],$$3 self-evaluation, and $$R_a = \mathbf{1}[\text{answer} = \text{ground\_truth}],$$4 reward prediction, and then writes

$$R_a = \mathbf{1}[\text{answer} = \text{ground\_truth}],$$5

Because the argument assumes

$$R_a = \mathbf{1}[\text{answer} = \text{ground\_truth}],$$6

it follows that

$$R_a = \mathbf{1}[\text{answer} = \text{ground\_truth}],$$7

The paper presents this as an information-theoretic intuition for why post-completion supervision can enrich learning. It also gives convergence and sample-complexity-style expressions, but explicitly notes that these should be read as high-level justification rather than as a rigorous modern theorem-proof development.

5. Empirical evidence

The experimental evaluation covers three reasoning datasets—GSM8K, StrategyQA, and MathQA—using Qwen2.5-7B, Qwen2.5-1.5B, LLaMA3.2-3B on GSM8K, and LLaMA3.1-8B on StrategyQA and MathQA. The training framework is based on Hugging Face OpenR1 with an extended GRPOTrainer (Fei et al., 27 Jul 2025).

The reported average results are as follows:

Dataset	Baseline averages	PCL average
GSM8K	SFT 60.12; SFT + RL 64.48; Joint SFT + RL 64.87; Joint SFT + RL w/ eval output 65.93	67.93
StrategyQA	SFT 69.00; SFT + RL 70.55	75.11
MathQA	SFT 60.69; SFT + RL 65.82	66.96

On GSM8K, PCL improves the average score from 60.12 under SFT to 67.93, and from 64.48 under SFT + RL to 67.93. The per-model gains are +5.76 for Qwen2.5-7B, +7.20 for Qwen2.5-1.5B, and +10.47 for LLaMA3.2-3B relative to SFT. On StrategyQA, the average rises to 75.11, exceeding 69.00 for SFT and 70.55 for SFT + RL. On MathQA, the average reaches 66.96, above 60.69 for SFT and 65.82 for SFT + RL, although the gains are more mixed.

The ablation results indicate that the full formulation matters. On GSM8K average, teacher distillation only reaches 63.37; PCL without evaluation SFT reaches 67.02; PCL without consistency reward reaches 66.08; and Joint SFT + RL w/ eval output reaches 65.93. These figures support the paper’s claim that the improvement is not merely a consequence of making the sequence longer.

Training dynamics also separate the reward channels. The paper reports that format rewards quickly reach about 1.0, accuracy reward improves more gradually from about 0.6 to 0.9, and consistency reward fluctuates early but stabilizes around 0.9. This suggests that structured formatting is the easiest part of the objective, while correct answering and calibrated self-evaluation remain harder.

The paper also records negative or mixed findings. The clearest case is MathQA for Qwen2.5-1.5B, where PCL reaches 60.40, slightly below the 60.44 obtained by SFT + RL. More broadly, the authors note that gains are weaker or mixed for smaller models on more complex tasks.

6. Scope, limitations, and terminological ambiguity

PCL occupies a specific position relative to adjacent methods. The paper describes it as extending outcome supervision beyond completion, remaining compatible with process supervision, partially internalizing aspects of reward modeling / RLHF, and differing from inference-time reflection methods such as Self-Refine or Reflexion by emphasizing training-time reflection, inference-time efficiency (Fei et al., 27 Jul 2025). It also overlaps conceptually with verifier training, but instead of delegating scoring to a separate model, it makes the base policy itself generate evaluation and reward traces.

The framework’s limitations are equally explicit. Training is substantially more elaborate than ordinary SFT: it requires additional teacher-generated data, longer training sequences, extra reward computation, and mixed SFT+RL optimization. The method depends on reward functions that are computable and structurally expressible in the post-completion segment. The paper also states that broader generalization beyond reasoning tasks is not yet established.

A separate source of confusion is terminological. In arXiv literature, PCL is an overloaded acronym. It refers to Peer-Contrastive Learning in unsupervised sentence embedding learning (Wu et al., 2022), Progressive Continual Learning in spoken keyword spotting (Huang et al., 2022), Population-based Continual Learning in rehearsal-free continual learning (Lu et al., 10 Feb 2025), and Prompt Curriculum Learning in reasoning-focused RL post-training (Gao et al., 1 Oct 2025). In the language-model training context discussed here, however, PCL specifically denotes Post-Completion Learning (Fei et al., 27 Jul 2025).

In summary, Post-Completion Learning is a training-time mechanism that uses the sequence region after answer completion to supervise self-evaluation and reward prediction without increasing inference-time generation. Its defining elements are the <post-completion> boundary, dual-track supervision over reasoning and reflection regions, and white-box RL based on agreement between self-predicted and externally computed rewards. The empirical record in the paper shows consistent gains on GSM8K and StrategyQA, more mixed behavior on MathQA, and a clear distinction between PCL and merely appending evaluation text.