Perplexity-aware RL Algorithm

• The paper demonstrates that integrating perplexity as a modulation signal in RL reduces evaluation bias and boosts accuracy in both mathematical critique and dialogue event detection tasks.
• It employs tailored reinforcement learning frameworks, using perplexity-based sample weighting and curriculum strategies to balance model prior preferences.
• Empirical results show marked improvements, with bias indicators nearly halved and accuracy significantly increased, highlighting the approach’s utility in fine-tuning large language models.

Perplexity-aware reinforcement learning (RL) algorithms utilize the concept of language-model perplexity to inform sample selection, trajectory evaluation, and policy updates during RL fine-tuning of LLMs. These algorithms systematically address biases or plateaus induced by model prior preferences—typically towards lower-perplexity, more fluently generated outputs—by explicitly integrating perplexity as either a modulation variable in the policy objective, as a sample weighting factor, or as a curriculum-learning signal. Two lines of work provide paradigmatic instances: Perplexity-aware Group Relative Policy Optimization (GRPO) for LLM mathematical critique (Tian et al., 13 Nov 2025), and Adaptive Perplexity-Aware RL (APARL) for event detection in dialogue systems (Zhang et al., 2 Jul 2025). Both demonstrate significant improvements in in-domain and out-of-domain performance via perplexity-driven mechanisms.

1. Imbalanced Evaluation Preference and the Role of Perplexity

Empirical analysis reveals that LLMs critiquing mathematical solution chains display a systematic imbalanced evaluation preference: solutions with lower perplexity under the model's own distribution are disproportionately favored as “correct,” whereas those with higher perplexity are judged “wrong” at elevated rates, independently of actual correctness. This occurs because perplexity—measuring the negative log-likelihood of a sequence given the model's parameters—reflects compatibility with the model's own generative style. As a result, stylistic fluency is conflated with solution validity, leading to over-exploitation of self-consistent but potentially spurious patterns and under-exploration of valid, divergent outputs (Tian et al., 13 Nov 2025). This phenomenon is also linked to poor utilization of hard or atypical examples in industrial dialogue event detection tasks (Zhang et al., 2 Jul 2025).

2. Formal Reinforcement Learning Problem Statements

Perplexity-aware RL algorithms instantiate the Markov Decision Process (MDP) framework with tailored state, action, and reward definitions:

• Mathematical Critique: Each state $s=(x,y)$ encapsulates a math problem $x$ and a candidate solution $y$, with the action space comprising sequential token generation for stepwise critiques, verdicts (${correct, wrong}$), and error-span identification. The reward combines a format term ($r_f = 0.1$ if output adheres to the required template) and an answer term ($r_a$, rewarding correct verdicts and error localization), with detailed rules for both correct and incorrect solution identification (Tian et al., 13 Nov 2025).
• Dialogue Event Detection: States are concatenations of event-type sets and $k$-turn utterance histories, actions are selections among $M$ pre-defined event classes, and the reward mixes correctness and output formatting, with penalties for both error types (Zhang et al., 2 Jul 2025).

In all cases, perplexity is computed for each solution or input as a sequence-level negative log-likelihood, serving as either a sample attribute or a trajectory weighting factor.

3. Perplexity-Aware Policy Optimization and Sample Weighting

Perplexity-aware GRPO and APARL adapt established RL optimization frameworks (group-relative PPO, standard PPO with KL regularization) by introducing perplexity into sample weighting and/or batch aggregation:

• Perplexity-aware GRPO (Tian et al., 13 Nov 2025) partitions samples by ground-truth label and model prediction, then applies group-specific, linear perplexity-based weights $w_i$ to modulate the policy-gradient advantage estimator $A_i^p = w_i A_i$. Over-preferred (low-perplexity, predicted correct) samples receive reduced weight; counter-preference (high-perplexity, judged correct or low-perplexity, judged wrong) samples are upweighted. Batch aggregation is performed separately and symmetrically for correct and incorrect solution groups, ensuring balanced coverage of both preference-consistent and preference-countering cases. The overall surrogate loss includes sequencewise PPO-style ratios with clipping and a KL penalty to prevent drift from the reference model.
• APARL (Zhang et al., 2 Jul 2025) maintains a dynamic estimate of per-example proficiency (empirical success probability $p_i$), proxies task difficulty using solution perplexity, and dynamically allocates sampling probability via a power-law schedule centered at the batch mean proficiency. This forces the curriculum to shift focus from initially easy (low perplexity, high $p_i$) to later harder (high perplexity, low $p_i$) samples. Policy updates within each batch are performed using clipped PPO loss with explicit KL regularization.

4. Dual-Loop and Modulation Strategies

Central to APARL is a dual-loop architecture comprising an outer loop for adaptive curriculum construction and an inner loop for policy updates:

Component	Role	Mechanism
Outer loop	Adjusts curriculum over training	Estimates $p_i$, $\mu_p$; samples batch by perplexity-adaptive weights
Inner loop	RL policy update on current batch	PPO with KL regularization; per-token reward, advantage

Similarly, perplexity-aware GRPO employs a two-stage modulation: first, groupwise partition and weighting by perplexity; then, class-level aggregation of losses before standard optimizer updates.

5. Benchmark Construction and Evaluation Metrics

The One-to-many Problem-Solution (OPS) benchmark (Tian et al., 13 Nov 2025) is designed to quantify imbalanced evaluation preference and the efficacy of perplexity-aware reinforcement signals. OPS comprises 1,890 items from the MATH test set, each paired with three LLM-generated solutions (Qwen2-7B, LLaMA3.1-8B, Mistral-7B), constructed to balance label distributions (correct vs. wrong) and maintain answer diversity. Evaluation metrics include accuracy and the Balance Indicator (BI), defined as $BI = FPR - FNR$, where $FPR$ and $FNR$ are the false positive and false negative rates for solution correctness judgments. An ideal model achieves $BI \approx 0$ and high accuracy.

In the dialogue domain, APARL is evaluated on a food-delivery dialogue dataset (55k train, 9k in-domain test) and on three anonymized out-of-domain sets (8.8k, 8.5k, 5k); metrics are precision, recall, and F1 score (Zhang et al., 2 Jul 2025).

6. Empirical Outcomes and Comparative Results

Substantial improvements in both bias rectification and generalization are observed:

• Mathematical Critique (Tian et al., 13 Nov 2025): On OPS, the perplexity-aware GRPO method yields a reduction of $|BI|$ from 34.39 (Base Qwen2-7B) and 35.24 (vanilla GRPO) to 17.14, with accuracy increasing from 64.71% (Base) to 79.79% (Perplexity-aware GRPO). Cross-model consistency is improved, as measured by reduced score variance across subsets. On ProcessBench, perplexity-aware GRPO achieves the highest step-localization F1 scores across GSM8K, MATH, Olympiad, and Omni-MATH.

| Model | |BI| ↓ | Acc% ↑ | |-------------------------|--------|---------| | Base Qwen2-7B | 34.39 | 64.71 | | + Vanilla GRPO | 35.24 | 74.76 | | + DrGRPO | 29.63 | 77.14 | | + Perplexity-aware GRPO | 17.14 | 79.79 |

• Dialogue Event Detection (Zhang et al., 2 Jul 2025): APARL demonstrates an average absolute F1 improvement of 17.19 percentage points in-domain (e.g., 83.38% for Qwen-14B vs. 66.19% for baseline), and 9.59 percentage points out-of-domain (e.g., 79.01% vs. 69.42%). Ablation analyses confirm the necessity of both adaptive sampling and KL regularization: removing either reduces F1 by 4–7 points. Learning curves reveal that perplexity-weighted curricula avoid early-plateau effects, steadily shifting training focus from easy to hard examples as proficiency increases.

7. Applications, Significance, and Related Directions

Perplexity-aware reinforcement learning constitutes a scalable solution for bias mitigation and curriculum management in the fine-tuning of LLM critics, multi-step reasoners, and dialogue event detectors. By using perplexity as a proxy of difficulty or stylistic preference, such algorithms prevent overfitting to model-consistent, low-diversity samples and enhance robustness to counter-preference or rare-case scenarios. This approach enables significant gains in accuracy, error-localization, and out-of-domain transferability, with particular utility for mathematical reasoning and industrial event detection. These methods provide a principled framework for integrating model-internal measures of sample uncertainty into RL-based optimization—addressing longstanding challenges in LLM self-critique and OOD robustness (Tian et al., 13 Nov 2025, Zhang et al., 2 Jul 2025). A plausible implication is widespread applicability in domains where stylistic biases and rare-sample underutilization limit classical RL and SFT approaches.