--- title: 'Post-Completion Learning: Reflective Training' url: https://www.emergentmind.com/topics/post-completion-learning-pcl type: topic --- # Post-Completion Learning: Reflective Training Post-Completion Learning (PCL) is a training paradigm for language models that extends supervision beyond the model’s normal answer boundary. Instead of terminating learning at ``, it replaces the usual end token with ``, allows the model to continue into a structured self-evaluation and reward-prediction segment, and supervises that post-completion region during training while still stopping generation at the completion boundary during inference [2507.20252]. In this formulation, the previously ignored sequence space after answer completion becomes a source of learning signal for reasoning quality, self-evaluation, reward prediction, and quality awareness or calibration. ## 1. Conceptual definition and motivation PCL is built around the claim that standard language-model training wastes information at the moment generation reaches ``. In conventional setups, the model is trained on the reasoning trace and answer, but not on whether the answer is correct, whether it satisfies the required format, whether the model can explain its own score, or whether it can compute the reward function that would otherwise remain external. PCL treats this omitted region as the **post-completion space** and uses it for explicit self-assessment [2507.20252]. The framework divides the target sequence into two functional regions. The **reasoning region** contains `think + answer`, and the **reflection region** contains `evaluation + reward`. This distinction is central. The method does not merely append extra text; it assigns different supervisory roles to the two regions. The paper characterizes the intended shift as moving from **“passive reward acceptance”** to **“active self-evaluation.”** The motivation is analogous to post-task reflection. After producing an answer, the model is trained to inspect that answer, analyze whether it is correct, and predict reward-relevant scores. The post-completion segment is therefore not user-facing output in deployment, but a training scaffold for self-assessment. A plausible implication is that PCL attempts to internalize part of the reward-evaluation process into the base policy itself, rather than leaving reward interpretation entirely to external supervision. ## 2. Sequence formulation and training pipeline The canonical PCL target sequence is \[ \text{question} \rightarrow \text{think} \rightarrow \text{answer} \rightarrow \langle \text{post-completion} \rangle \rightarrow \text{evaluation} \rightarrow \text{reward}. \] This sequence design introduces a hard boundary between visible completion and reflective continuation. During training, the model continues after ``; during inference, decoding stops when `` is emitted, so the reflection region is not generated for the user [2507.20252]. The training pipeline has three coordinated stages. In **reasoning SFT**, the model is trained on ordinary supervised reasoning data of the form `(question, think, answer)`, except that the original `` is replaced by ``. Loss is applied only on the `think + answer` region. In **evaluation SFT**, a teacher model generates full PCL-format sequences, and the student is trained to continue from `think + answer + ` into `evaluation + reward`; here the loss is applied only on the reflection region. In **RL over complete PCL sequences**, the model samples full sequences including the post-completion segment and is optimized with reward functions defined over answer correctness, format correctness, and self-evaluation consistency. Teacher data construction is itself structured. The paper states that teacher generation uses **GPT-4.1** with in-context demonstrations of the PCL format. Generated samples are then automatically validated by recomputing reward scores, and examples are kept when the teacher correctly evaluates its own answer, even if the answer itself is wrong. This matters because the evaluation track is meant to teach evaluation skill, not merely to duplicate answer supervision. The inference-time consequence is one of the method’s defining properties. The post-completion region is learned but not deployed. The model stops at ``, which replaces `` as the stop token. The paper explicitly claims **zero additional inference overhead** in this setting, because the reflective tokens are not decoded at test time. ## 3. White-box reinforcement learning and reward alignment The paper describes its RL component as **white-box reinforcement learning**. The distinction is that the model is not only rewarded by externally computed scores; it is also required to explicitly compute and verbalize the reward logic in the post-completion region, then align its predicted reward with the true reward [2507.20252]. Three reward functions are defined. The **accuracy reward** is \[ R_a = \mathbf{1}[\text{answer} = \text{ground\_truth}], \] the **format reward** is \[ R_f = \mathbf{1}[\text{contains\_all\_sections}(\text{output})], \] and the **consistency reward** measures agreement between self-predicted and externally computed reward: \[ R_c = 1 - \left| R_{\text{pred}} - R_{\text{true}} \right|_1. \] The format reward is explicitly modified to require all four sections—think, answer, evaluation, and reward—so the RL stage supervises the full structured output, not only the visible answer. The consistency reward is the core of the white-box design: the model is rewarded when its predicted score matches the actual reward-function output. The RL objective is given as \[ \mathcal{L}_{\text{RL}} = -\mathbb{E}_{\pi_\theta}\big[(R_a + R_f + R_c)\log \pi_\theta(a \mid s)\big]. \] The implementation uses **GRPO**, samples **8 responses per prompt** as a group, and retains the default KL constraint with coefficient \(\beta = 0.04\). The central alignment rule is straightforward: compute external reward \(R_{\text{true}}\), ask the model to produce \(R_{\text{pred}}\) in the post-completion reward section, and reward the model for minimizing the discrepancy between them. This design makes the post-completion segment more than an auxiliary explanation. It becomes a structured interface through which the policy represents the reward rules that are simultaneously used to optimize it. ## 4. Dual-track supervision and hybrid optimization PCL organizes supervised learning into two distinct tracks. The first is **reasoning-track SFT**, which trains the model to solve the task; the second is **evaluation-track SFT**, which trains it to assess the completed solution. The losses are region-specific [2507.20252]: \[ \mathcal{L}_{\text{SFT1}} = -\sum_{t \in \{\text{think}, \text{answer}\}} \log P(x_t \mid x_{`. The combined objective is additive: \[ \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{SFT1}} + \mathcal{L}_{\text{SFT2}} + \mathcal{L}_{\text{RL}}. \] The paper emphasizes concurrent rather than sequential optimization. Reasoning SFT, evaluation SFT, and RL are mixed within the same batch. The reported training recipe keeps the **reasoning SFT : evaluation SFT ratio at 1:1**, the **RL : SFT ratio at 1:1**, and trains for **2 epochs**, with other hyperparameters following **OpenR1** defaults. A theoretical intuition is also provided. The paper defines an extended target sequence \[ y^+ = y \oplus e \oplus r, \] with \(y\) the reasoning sequence, \(e\) self-evaluation, and \(r\) reward prediction, and then writes \[ I(\theta; y^+ \mid x) = I(\theta; y \mid x) + I(\theta; e,r \mid x,y). \] Because the argument assumes \[ I(\theta; e,r \mid x,y) > 0, \] it follows that \[ I(\theta; y^+ \mid x) > I(\theta; y \mid x). \] The paper presents this as an information-theoretic intuition for why post-completion supervision can enrich learning. It also gives convergence and sample-complexity-style expressions, but explicitly notes that these should be read as high-level justification rather than as a rigorous modern theorem-proof development. ## 5. Empirical evidence The experimental evaluation covers three reasoning datasets—**GSM8K**, **StrategyQA**, and **MathQA**—using **Qwen2.5-7B**, **Qwen2.5-1.5B**, **LLaMA3.2-3B** on GSM8K, and **LLaMA3.1-8B** on StrategyQA and MathQA. The training framework is based on Hugging Face **OpenR1** with an extended **GRPOTrainer** [2507.20252]. The reported average results are as follows: | Dataset | Baseline averages | PCL average | |---|---|---| | GSM8K | SFT 60.12; SFT + RL 64.48; Joint SFT + RL 64.87; Joint SFT + RL w/ eval output 65.93 | 67.93 | | StrategyQA | SFT 69.00; SFT + RL 70.55 | 75.11 | | MathQA | SFT 60.69; SFT + RL 65.82 | 66.96 | On **GSM8K**, PCL improves the average score from **60.12** under SFT to **67.93**, and from **64.48** under SFT + RL to **67.93**. The per-model gains are **+5.76** for Qwen2.5-7B, **+7.20** for Qwen2.5-1.5B, and **+10.47** for LLaMA3.2-3B relative to SFT. On **StrategyQA**, the average rises to **75.11**, exceeding **69.00** for SFT and **70.55** for SFT + RL. On **MathQA**, the average reaches **66.96**, above **60.69** for SFT and **65.82** for SFT + RL, although the gains are more mixed. The ablation results indicate that the full formulation matters. On GSM8K average, **teacher distillation only** reaches **63.37**; **PCL without evaluation SFT** reaches **67.02**; **PCL without consistency reward** reaches **66.08**; and **Joint SFT + RL w/ eval output** reaches **65.93**. These figures support the paper’s claim that the improvement is not merely a consequence of making the sequence longer. Training dynamics also separate the reward channels. The paper reports that **format rewards quickly reach about 1.0**, **accuracy reward improves more gradually from about 0.6 to 0.9**, and **consistency reward fluctuates early but stabilizes around 0.9**. This suggests that structured formatting is the easiest part of the objective, while correct answering and calibrated self-evaluation remain harder. The paper also records negative or mixed findings. The clearest case is **MathQA** for **Qwen2.5-1.5B**, where PCL reaches **60.40**, slightly below the **60.44** obtained by SFT + RL. More broadly, the authors note that gains are weaker or mixed for smaller models on more complex tasks. ## 6. Scope, limitations, and terminological ambiguity PCL occupies a specific position relative to adjacent methods. The paper describes it as extending **outcome supervision** beyond completion, remaining compatible with **process supervision**, partially internalizing aspects of **reward modeling / RLHF**, and differing from inference-time reflection methods such as **Self-Refine** or **Reflexion** by emphasizing **training-time reflection, inference-time efficiency** [2507.20252]. It also overlaps conceptually with **verifier training**, but instead of delegating scoring to a separate model, it makes the base policy itself generate evaluation and reward traces. The framework’s limitations are equally explicit. Training is substantially more elaborate than ordinary SFT: it requires additional teacher-generated data, longer training sequences, extra reward computation, and mixed SFT+RL optimization. The method depends on reward functions that are computable and structurally expressible in the post-completion segment. The paper also states that broader generalization beyond reasoning tasks is not yet established. A separate source of confusion is terminological. In arXiv literature, **PCL** is an overloaded acronym. It refers to **Peer-Contrastive Learning** in unsupervised sentence embedding learning [2201.12093], **Progressive Continual Learning** in spoken keyword spotting [2201.12546], **Population-based Continual Learning** in rehearsal-free continual learning [2502.06210], and **Prompt Curriculum Learning** in reasoning-focused RL post-training [2510.01135]. In the language-model training context discussed here, however, PCL specifically denotes **Post-Completion Learning** [2507.20252]. In summary, Post-Completion Learning is a training-time mechanism that uses the sequence region after answer completion to supervise self-evaluation and reward prediction without increasing inference-time generation. Its defining elements are the `` boundary, dual-track supervision over reasoning and reflection regions, and white-box RL based on agreement between self-predicted and externally computed rewards. The empirical record in the paper shows consistent gains on GSM8K and StrategyQA, more mixed behavior on MathQA, and a clear distinction between PCL and merely appending evaluation text. Source: https://www.emergentmind.com/topics/post-completion-learning-pcl