DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant advances in the reasoning capabilities of LLMs. However, effectively managing the exploration and exploitation trade-off remains a critical challenge. In this paper, we fully analyze the exploration and exploitation dilemma of extremely hard and easy samples during the training and propose a new fine-grained trade-off mechanism. Concretely, we introduce a perplexity space disentangling strategy that divides the sample space into distinct exploration (high perplexity) and exploitation (low perplexity) subspaces, thereby mining fine-grained samples requiring exploration-exploitation trade-off. Subsequently, we propose a bidirectional reward allocation mechanism with a minimum impact on verification rewards to implement perplexity-guided exploration and exploitation, enabling more stable policy optimization. Finally, we have evaluated our method on two mainstream tasks: mathematical reasoning and function calling, and experimental results demonstrate the superiority of the proposed method, confirming its effectiveness in enhancing LLM performance by fine-grained exploration-exploitation trade-off.

• The paper presents DiPO, a novel method that mitigates gradient collapse and overfitting by disentangling perplexity spaces via PSD and BRR.
• It partitions the perplexity space into distinct quadrants to selectively allocate rewards, thereby fine-tuning the exploration-exploitation trade-off.
• Empirical evaluations on mathematical reasoning and function calling benchmarks show consistent performance improvements across multiple LLM scales.

Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off

Introduction

Disentangled Perplexity Policy Optimization (DiPO) (2604.13902) addresses the persistent challenge of balancing exploration and exploitation in RL-based post-training of LLMs with verifiable rewards. Traditional RLVR algorithms, particularly GRPO and DAPO, suffer from two critical deficiencies: i) the gradient collapse for extreme sample groups (easy/hard) due to reward saturation, and ii) suboptimal exploration-exploitation signaling where hard samples stagnate and easy samples overfit. DiPO introduces two synergistic mechanisms—Perplexity Space Disentangling (PSD) and Bidirectional Reward Reallocation (BRR)—to precisely partition exploratory and exploitative sample subspaces and stabilize policy updates. Empirical validation on mathematical reasoning (five benchmarks) and function calling (BFCLv3) demonstrates robust improvement over prior RLVR paradigms.

Figure 1: (a) The temporal proportion of Easy/Normal/Hard groups in DAPO training. (b) The PPL distribution in validation at the 300th step, showing overlap between correct/error samples. (c) Fine-grained partitioning of four quadrants after PSD.

Fine-grained Exploration-Exploitation via Perplexity Space Disentangling

The dominant RLVR approaches, rooted in group-based advantage assignment (GRPO, DAPO), rely on reward binarization rendering extreme groups inert (zero gradient). DiPO advances sample discernment via PSD, which partitions the perplexity (PPL) space conditioned on correctness, yielding optimal threshold $\tau^*$ separating exploitation (low PPL) and exploration (high PPL) regions. The core insight leverages statistical conditional probability estimation, dynamically updated with online batches, and robust threshold selection (minimizing classification errors subject to confidence intervals). Four quadrants emerge: CL (correct/low-PPL), CH (correct/high-PPL), EL (error/low-PPL), EH (error/high-PPL). This mechanism allows targeted encouragement—exploitation for CH, exploration for EL—rather than indiscriminate reward amplification, mitigating ineffective improvement in RLVR.

Figure 2: DiPO architecture comprising PPL Queue, PSD module, and BRR; PPL Queue facilitates conditional estimation, PSD isolates quadrants, BRR performs reward reallocation.

Stable Policy Optimization via Bidirectional Reward Reallocation

Directly using PPL as reward bias, as in DACE or CDE, destabilizes updates by introducing variance orthogonal to verifiable rewards. DiPO formulates BRR, which only intervenes in gradient-zero easy/hard groups. For hard samples (in exploitation region), BRR allocates maximum-PPL reward, driving entropy increase and exploratory update. For easy samples (in exploration region), maximum-PPL is penalized, lowering entropy and strengthening exploitation. Mathematical proof and experimental verification confirm entropy modulation: reward increases entropy, penalty decreases it. Crucially, BRR is minimally disruptive to the original reward distribution and remains orthogonal to correctness-based reward, enabling independent tuning traceable to the hyperparameter $\alpha$.

Figure 3: Entropy curves under maximum-PPL reward (increasing) and penalty (decreasing), as theoretically and empirically predicted.

Comparative Evaluation: Mathematical Reasoning and Function Calling

Across three LLM scales (Qwen3-4B/8B, Qwen2.5-7B), DiPO consistently delivers best-in-class mean@8 accuracy on 6 mathematical reasoning benchmarks. For Qwen3-8B-Base, DiPO achieves 54.79% in AVG compared to 53.23% for DAPO. Performance advantage is pronounced in harder datasets (AIME24/25), where exploration is essential. DiPO’s efficacy extends to function calling on BFCLv3: overall accuracy on Qwen2.5-7B-Instruct is 62.51%, compared to 61.06% for ToolRL+DAPO, with substantial gains in Multi-Turn Acc reflecting improved long-horizon reasoning.

Figure 4: PPL distribution post-training with DAPO vs DiPO; DiPO achieves better separation between correct/error samples.

Figure 5: PPL distribution visualization (top: DiPO; bottom: DAPO); DiPO maintains high-PPL for error samples and low-PPL for correct samples.

Hyperparameter Robustness and Ablation Analysis

BRR’s orthogonality to validation rewards allows precise scaling via $\alpha$; performance is optimal at $\alpha=0.1$ and stable across a wide range (see Table: hyperparameter analysis). Ablation studies demonstrate that PSD and BRR yield substantial gains when combined; PSD alone delivers improvements by selectively optimizing sample quadrants, while BRR outperforms direct PPL reward shaping by ensuring stable reward distributions and avoiding collapse.

Quantitative Properties and Distributional Insights

DiPO rectifies the RLVR tendency for excessive overlap in PPL distribution between correct and error samples, especially in hard groups. Visualization shows DiPO maintains discriminative PPL clusters, facilitating fine-grained exploration-exploitation. Training curves on benchmarks such as AIME reveal that DiPO sustains exploratory growth even as DAPO plateaus in later iterations.

Figure 6: Risk prediction prompt, exemplifying the application of DiPO reward shaping in high-stakes public opinion analysis.

Case Visualization: Entropy and Correct/Error Dynamics

Comparative case visualization for DAPO and DiPO evidences that DiPO delivers concentrated low-entropy trajectories for correct answers and denser high-entropy regions for incorrect answers—coherently reflecting the targeted reward allocation and improved sample discrimination.

Figure 7: Correct case visualization for DAPO, showing entropy map.

Figure 8: Error case for DAPO, illustrating higher entropy concentration.

Figure 9: Correct case for DiPO, demonstrating lower entropy relative to DAPO.

Figure 10: Error case for DiPO, showing focused high-entropy token distribution.

Implications, Theoretical and Practical

The introduction of PSD and BRR sets a new direction in RLVR: adaptive, disentangled reward shaping that maintains orthogonality to verification signals and leverages entropy/PPL only in contexts with zero gradients. Theoretically, this enables granular policy updates and circumvents the reward collapse endemic to GRPO/DAPO. Practically, DiPO is robust (hyperparameter insensitivity), generalizable (mathematical reasoning, function calling, risk prediction), and scalable to large models. Future research efforts can exploit PSD/BRR-style disentanglement for additional RLVR tasks, integrate non-binary verification rewards, and broaden the conditional estimation horizon.

Conclusion

DiPO (2604.13902) advances RLVR for LLMs via Perplexity Space Disentangling and Bidirectional Reward Reallocation. The method achieves precise, stable exploration-exploitation trade-off, markedly enhancing the performance of mathematical reasoning and function calling models. DiPO’s mechanisms address reward saturation and ineffective policy gradients in extreme groups, yielding practical gains, hyperparameter stability, and distributional clarity. Its architectural contribution is foundational for next-generation LLM-RL frameworks centered on robust reward disentanglement and orthogonality.