Zone of Proximal Policy Optimization (ZPPO)

• ZPPO is a reinforcement learning method that uses teacher prompt scaffolding to guide small student models without altering the on-policy gradient.
• It employs BCQ and NCQ prompt reformulations to provide contextual scaffolding, enabling meaningful gradient signals even on hard questions.
• The approach uses a FIFO replay buffer and adaptive advantage estimation to overcome mode-seeking bias, improving both in-domain and out-of-domain performance.

Searching arXiv for the specified paper to ground the article and confirm bibliographic details. Zone of Proximal Policy Optimization (ZPPO) is a reinforcement-learning-based post-training method for small student models in which a stronger teacher is placed inside the prompt rather than inside the policy gradient. It is introduced in "Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients" (Lee et al., 16 Jun 2026). The method is motivated by two failures identified for small-student post-training: standard knowledge distillation is brittle when a limited-capacity student is forced to match the sharp modes of a much larger teacher, and on-policy RL yields zero gradient on hard questions when every student rollout is incorrect. ZPPO addresses this regime by reformulating hard prompts so that teacher information appears as contextual scaffolding while the optimization remains on-policy at the response level, and by replaying unresolved prompts until they either "graduate" or are evicted from a finite FIFO buffer (Lee et al., 16 Jun 2026).

1. Motivation and conceptual basis

ZPPO is formulated for the small-student regime, where a student policy $\pi_\theta$ is trained in the presence of a frozen teacher policy $\pi_T$ on prompts $x$, described in the source as image–question pairs (Lee et al., 16 Jun 2026). The central problem statement is that classic distillation, in which a student minimizes a divergence such as cross-entropy or Jensen–Shannon divergence to match the teacher’s soft logits $p_T(y \mid x)$, becomes brittle when the capacity gap is large. In that setting, the student overfits the teacher’s sharp peaks, an effect characterized as "mode-seeking bias," and loses generalization on out-of-distribution or held-out benchmark families (Lee et al., 16 Jun 2026).

The paper contrasts this with on-policy RL methods such as PPO and GRPO, where the student samples its own rollouts and receives binary correctness rewards $r(x,y)\in\{0,1\}$ (Lee et al., 16 Jun 2026). On hard questions where the sampled group is all-wrong or all-right, the reward standard deviation is zero, the advantage vanishes, and those questions provide zero gradient. A natural repair—injecting a correct teacher response as if it had been sampled by the student—is rejected because it breaks the on-policy assumption and induces policy drift by pushing the policy toward actions it has never sampled (Lee et al., 16 Jun 2026).

The method’s name is derived from Vygotsky’s "zone of proximal development" (ZPD), defined in the source as the set of tasks a learner cannot yet do alone but can solve with minimal guidance from a more capable peer (Lee et al., 16 Jun 2026). ZPPO adopts this as an organizing principle: the teacher does not enter the gradient; instead, hard questions in the student’s current zone are scaffolded through prompt reformulations that provide guidance without altering the on-policy character of the response sampling (Lee et al., 16 Jun 2026).

2. Objective and on-policy formulation

The paper positions ZPPO against both PPO-style policy optimization and distillation losses. The PPO surrogate is given tokenwise as

$$L_{\rm PPO}(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigl[\min\bigl(\rho_t(\theta)A_t,\;\mathrm{clip}(\rho_t(\theta),1-\epsilon,\,1+\epsilon)A_t\bigr)\Bigr],$$

where

$$\rho_t(\theta)=\pi_\theta(y_t\mid x,y_{<t})/\pi_{\theta_{\rm old}}(y_t\mid x,y_{<t}).$$

In the source, distillation is characterized as injecting teacher distributions into the gradient, whereas ZPPO keeps teacher knowledge in the prompt; the gradient remains $\nabla_\theta \log \pi_\theta(y \mid x')$ for prompts $x'$ that may be plain, BCQ, or NCQ, so the update is always on-policy at the response level (Lee et al., 16 Jun 2026).

For advantage estimation, the method adopts the two-step estimator of REINFORCE++ with zero-advantage groups excluded from batch statistics (Lee et al., 16 Jun 2026). The within-group centering step is

$A'^{(g,i)}=r(x,y^{(g,i)})-\bar r_x.$

The second step performs batch normalization over non-trivial groups $\pi_T$0:

$\pi_T$1

The PPO surrogate is then built on these normalized advantages (Lee et al., 16 Jun 2026). This design directly targets the failure mode in which hard questions disappear from learning because all sampled responses are uniformly incorrect.

A plausible implication is that the method redefines the unit of intervention: rather than changing the target distribution in gradient space, it changes the observation context in prompt space. In the terminology of the paper, this preserves on-policy training while still exposing the student to teacher-derived structure (Lee et al., 16 Jun 2026).

3. Prompt reformulations: BCQ and NCQ

ZPPO introduces two prompt constructions for hard questions, defined by the condition $\pi_T$2 (Lee et al., 16 Jun 2026). Teacher and student rollouts are first collected, with the teacher’s correct rollouts retained and student rollouts partitioned according to correctness. Each response is then compressed by a shared teacher prompt into a short trace $\pi_T$3, a design used so that surface length does not betray correctness (Lee et al., 16 Jun 2026).

The first reformulation is the Binary Candidate-included Question (BCQ). In BCQ, a hard prompt $\pi_T$4 is augmented with the instruction that two candidate responses are provided and that one is correct and one is wrong. One candidate is a correct teacher response and the other is an incorrect student response; the pair is shuffled and anonymized (Lee et al., 16 Jun 2026). The student then samples new rollouts from the reformulated prompt $\pi_T$5 under its own policy.

The second reformulation is the Negative Candidate-included Question (NCQ). Here the prompt explicitly states that a set of candidate answers are all wrong, and those candidates are drawn from multiple incorrect student rollouts (Lee et al., 16 Jun 2026). The aim is to surface shared failure modes across the student’s wrong answers, after which the student again samples its own responses from the reformulated prompt $\pi_T$6.

These two constructions play distinct roles. BCQ juxtaposes a correct teacher candidate with an incorrect student candidate, while NCQ aggregates only student failures (Lee et al., 16 Jun 2026). The paper’s later analysis states that BCQ’s relative contribution shrinks with model scale, whereas NCQ’s grows, and that the same pattern holds on the LLM+Video generalization family: BCQ lifts out-of-domain performance most at small scales, while NCQ matters more at mid-scales (Lee et al., 16 Jun 2026). This suggests that BCQ primarily supplies missing positive contrast when a small student cannot generate any useful successes on its own, whereas NCQ becomes more informative once the student can already exploit negative evidence at somewhat larger scales.

4. Training loop and replay buffer

The ZPPO training loop combines new prompts from the dataset with replayed prompts sampled uniformly from a prompt replay buffer $\pi_T$7 (Lee et al., 16 Jun 2026). The method is parameterized by student and teacher group sizes $\pi_T$8 and $\pi_T$9, replay fraction $x$0, augmentation fraction $x$1, buffer capacity $x$2, and an admission/graduation threshold $x$3 (Lee et al., 16 Jun 2026).

For each rollout step, a batch of new prompts is sampled from the dataset and a replay batch is sampled from the buffer, with replay size set to $x$4 (Lee et al., 16 Jun 2026). For each prompt $x$5 in the combined batch, the student draws $x$6 rollouts, rewards are computed, and the mean reward $x$7 is used to identify hard questions. If $x$8, teacher rollouts are drawn, correct teacher responses are retained, and the prompt becomes a candidate for reformulation if there is at least one correct teacher response or any incorrect student response (Lee et al., 16 Jun 2026).

Among hard questions, the method selects the top $x$9 by ascending $p_T(y \mid x)$0 and builds up to two reformulations, BCQ and NCQ, while capping total reformulations at $p_T(y \mid x)$1 (Lee et al., 16 Jun 2026). Student rollouts on the reformulated prompts are then scored, all groups of size $p_T(y \mid x)$2 across plain and reformulated prompts are collected, advantages are computed via the two-step REINFORCE++ procedure, and $p_T(y \mid x)$3 PPO updates are performed on $p_T(y \mid x)$4 (Lee et al., 16 Jun 2026).

The replay buffer stores raw prompts only, not responses, and is a FIFO buffer with capacity $p_T(y \mid x)$5; the source gives the practical setting $p_T(y \mid x)$6 (Lee et al., 16 Jun 2026). After each rollout step, prompts are admitted if they are not already in the buffer and satisfy $p_T(y \mid x)$7, graduated immediately if they are in the buffer and satisfy $p_T(y \mid x)$8, and evicted from the head if capacity is exceeded (Lee et al., 16 Jun 2026). The buffer update is summarized as

$p_T(y \mid x)$9

This mechanism is explicitly described as amplifying BCQ and NCQ inside the student’s current zone of proximal development by repeatedly revisiting unresolved hard questions until they either become solvable at mean rollout accuracy one-half or leave the finite buffer (Lee et al., 16 Jun 2026).

5. Experimental configuration and benchmark suite

The reported experiments use the Qwen3.5 family at four student scales: Qwen3.5-0.8B, 2B, 4B, and 9B (Lee et al., 16 Jun 2026). Each student is further post-trained as a vision-LLM on a 77 K-sample multimodal RL corpus of image–question pairs with short gold answers (Lee et al., 16 Jun 2026). The teacher is a frozen Qwen3.5-27B trained on the same RL corpus (Lee et al., 16 Jun 2026). During prompt reformulation, including BCQ, NCQ, and "hint," every candidate is compressed by the teacher under a shared "summarize to $r(x,y)\in\{0,1\}$0 tokens" prompt so that surface length does not betray correctness (Lee et al., 16 Jun 2026).

Evaluation is conducted on a 31-benchmark suite comprising 16 VLM benchmarks, 10 LLM benchmarks, and 5 Video benchmarks (Lee et al., 16 Jun 2026). The 16 VLM benchmarks are AI2D, BabyVision, CharXiv, DynaMath, EmbSpatial, InfoVQA, MathVerse, MathVision, MathVista, MMMU-Pro, MM-Vet, OCR-EN, OCR-ZH, VisualPuzzles, VLM-are-Blind, and WeMath (Lee et al., 16 Jun 2026). The 10 LLM benchmarks are AIME2025, AIME2026, C-Eval, GPQA-Diamond, HLE, IMO-AnswerBench, MMLU, MMLU-Pro, MMLU-Redux, and MultiChallenge (Lee et al., 16 Jun 2026). The 5 Video benchmarks are MMVU, MVBench, Video-MME (±subtitles), and Video-MMMU (Lee et al., 16 Jun 2026).

The benchmark partition is important to the paper’s argument. The 16 VLM tasks are designated in-domain, whereas the 10 LLM and 5 Video tasks are treated as out-of-domain generalization families (Lee et al., 16 Jun 2026). This framing is central to the paper’s comparison between prompt-based on-policy scaffolding and forms of distillation that are said to narrow the student onto the teacher’s distribution.

6. Results, ablations, and interpretation

The main reported result is that across all scales and benchmark families, ZPPO outperforms both off-/on-policy distillation and pure GRPO (Lee et al., 16 Jun 2026). On the 16 VLM benchmarks, the gains over the best non-ZPPO method are +9.3 percentage points at 0.8B, +5.2 at 2B, +4.0 at 4B, and +2.8 at 9B; the 0.8B result is specified as $r(x,y)\in\{0,1\}$1 (Lee et al., 16 Jun 2026). On the 10 LLM + 5 Video benchmarks, the gains over the best non-ZPPO baseline are +6.8 percentage points at 0.8B, +4.3 at 2B, +2.7 at 4B, and +2.7 at 9B; the 0.8B result is specified as $r(x,y)\in\{0,1\}$2 (Lee et al., 16 Jun 2026).

The source also reports macro-averaged scores across families at each scale:

Model scale	VLM avg / $r(x,y)\in\{0,1\}$3	LLM+Video avg / $r(x,y)\in\{0,1\}$4
Qwen3.5-0.8B	41.0 / +9.3	25.2 / +6.8
Qwen3.5-2B	56.8 / +5.2	45.3 / +4.3
Qwen3.5-4B	66.9 / +4.0	61.6 / +2.7
Qwen3.5-9B	70.2 / +2.8	64.9 / +2.7

Ablation studies proceed along the sequence "GRPO $r(x,y)\in\{0,1\}$5 + buffer $r(x,y)\in\{0,1\}$6 + BCQ $r(x,y)\in\{0,1\}$7 + NCQ $r(x,y)\in\{0,1\}$8 ZPPO" and are reported to show that each sub-component is helpful and that buffer$r(x,y)\in\{0,1\}$9BCQ or buffer$$L_{\rm PPO}(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigl[\min\bigl(\rho_t(\theta)A_t,\;\mathrm{clip}(\rho_t(\theta),1-\epsilon,\,1+\epsilon)A_t\bigr)\Bigr],$$0NCQ is super-additive (Lee et al., 16 Jun 2026). The paper further states that BCQ’s relative contribution shrinks with scale, while NCQ’s grows; on LLM+Video, BCQ contributes most to out-of-domain lift at small scales, whereas NCQ matters more at mid-scales (Lee et al., 16 Jun 2026).

The discussion section attributes the largest gains at the smallest scales to the fact that small students generate almost all-wrong plain rollouts, so GRPO sees zero advantage on the hardest questions (Lee et al., 16 Jun 2026). BCQ "instantly recovers a mixed-advantage signal" by juxtaposing one correct teacher candidate with one wrong student candidate, while NCQ surfaces collective negative patterns from multiple student failures (Lee et al., 16 Jun 2026). The replay buffer then repeatedly practices these reformulations inside the student’s ZPD, with the strongest compounding effect when the zone is widest, identified in the source as the 0.8B $$L_{\rm PPO}(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigl[\min\bigl(\rho_t(\theta)A_t,\;\mathrm{clip}(\rho_t(\theta),1-\epsilon,\,1+\epsilon)A_t\bigr)\Bigr],$$1 2B regime (Lee et al., 16 Jun 2026).

7. Limitations, misconceptions, and open directions

A common misunderstanding would be to treat ZPPO as another form of distillation. The paper distinguishes it from distillation by insisting that the teacher never enters the gradient; teacher knowledge is placed in the prompt, and the student still samples and optimizes its own responses under an on-policy objective (Lee et al., 16 Jun 2026). Another potential misconception is that the method only improves in-domain performance. The reported evaluation includes both in-domain VLM benchmarks and out-of-domain LLM and Video benchmarks, and the paper states that ZPPO preserves and improves the student’s ability to tackle unseen LLM and Video tasks, whereas distillation degrades out-of-domain generalization by locking the student onto the teacher’s narrow distribution (Lee et al., 16 Jun 2026).

The paper identifies concrete limitations. BCQ requires at least one correct teacher rollout on a prompt; when the teacher fails, only NCQ remains available, and the source explicitly states that NCQ is less powerful in that case (Lee et al., 16 Jun 2026). The zone of proximal development is therefore bounded by the teacher’s capability (Lee et al., 16 Jun 2026). This suggests that ZPPO’s ceiling is linked to the coverage of the frozen teacher and to the availability of correct teacher responses under the rollout budget.

The open directions listed in the source are likewise bounded and specific: expanding the zone beyond the current teacher’s coverage through synthetic candidate generation, multi-teacher ensembles, or curriculum-aware question selection; combining ZPPO with dynamic sampling that discards all-wrong prompts through a hybrid architecture that first applies BCQ/NCQ and then possibly samples anew; extending the framework to multi-step or agentic reasoning tasks; and investigating theoretical guarantees for on-policy reformulations and buffer dynamics (Lee et al., 16 Jun 2026). These are framed not as established properties but as future questions arising from the method’s prompt-centric strategy for preserving on-policy training while using teacher-derived scaffolding.