ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression

Abstract: Large reasoning models (LRMs) achieve strong performance via extended chain-of-thought (CoT) reasoning, yet suffer from excessive token consumption and high inference latency. Existing reinforcement learning (RL) approaches for CoT compression rely on uniform, static length penalties that neglect model capability dynamics and problem-level difficulty variation. We propose \textbf{ExpThink}\xspace, an RL framework that addresses both dimensions through two complementary mechanisms. First, \emph{experience-guided reward shaping} tracks the shortest correct solution found so far for each problem and applies a three-tier reward: full credit for concise correct responses, discounted credit for verbose correct ones, and zero for incorrect ones. The threshold tightens automatically with model improvement, forming a self-evolving curriculum that requires no manual scheduling. Second, \emph{difficulty-adaptive advantage} replaces standard deviation normalization with correct-count normalization, yielding monotonically difficulty-scaled gradients that amplify learning on hard problems to preserve accuracy while suppressing gradients on easy ones to encourage brevity. Together, these mechanisms enforce an accuracy-first, compression-second training objective. Experiments on multiple mathematical reasoning benchmarks demonstrate that \textbf{ExpThink}\xspace reduces average response length by up to 77\% while simultaneously improving accuracy, achieving up to $3\times$ higher accuracy-efficiency ratio (accuracy divided by average token count) than the vanilla baseline and outperforming existing RL-based compression methods on both metrics.

• The paper introduces an adaptive reward shaping mechanism via a global experience buffer that dynamically sets compression thresholds based on task difficulty.
• It demonstrates difficulty-adaptive advantage estimation by normalizing policy gradients using correct rollout counts to improve training efficiency.
• ExpThink achieves significant gains with a 77.2% token reduction and a 5.33% accuracy improvement across diverse mathematical reasoning benchmarks.

Experience-Guided Reinforcement Learning for Chain-of-Thought Compression: An Analysis of ExpThink

Introduction

The proliferation of Large Reasoning Models (LRMs) has yielded considerable gains in complex reasoning tasks, particularly via chain-of-thought (CoT) prompting. While extended CoT traces enhance reliability on high-difficulty benchmarks, this comes at the expense of excessive token generation, leading to inflated computational costs, latency, and adverse sample efficiency. Prevailing reinforcement learning (RL) approaches for CoT compression suffer from static, formulaic length regularization that disregards the evolving capabilities of the underlying LRM and the distribution of task difficulties encountered during curriculum learning. The ExpThink framework addresses this structural deficiency through adaptive reward shaping and difficulty-aware advantage scaling, leveraging cross-epoch trajectory information to establish a self-evolving compression curriculum. This essay systematically examines the core mechanisms, empirical contributions, and implications for both practical deployment and theoretical reinforcement learning for LRMs.

Structural Limitations in Standard RL-based CoT Compression

Most established RL methods for CoT compression (e.g., L1, ThinkPrune, LC-R1, Laser) impose static or globally synchronized penalties for verbosity, but treat every training epoch as an independent optimization phase. This leads to persistent inefficiency: as the LRM progressively discovers shorter valid solutions, these are discarded after each batch, leaving the reward schedule agnostic to the model's demonstrated compression capabilities. Empirical analysis reveals that even after a model learns to solve a subset of tasks concisely, its average response length remains significantly above the established lower bounds, confirming that valuable trajectory information is not exploited.

The ExpThink Framework

Experience-Guided Reward Shaping

ExpThink introduces a global experience buffer that records, for each training instance, the shortest correct solution observed across all epochs. At every update, the buffer is updated via a non-increasing running minimum over trajectory lengths that yield correct answers. This information induces a per-problem, adaptively tightening compression threshold, naturally implementing an implicit curriculum without manual schedule tuning.

A three-tier reward mechanism is used to ensure positive reinforcement for correct-but-verbose solutions without conflating correctness and brevity signals. The highest reward is granted to concise and correct rollouts, a discounted reward to verbose but correct ones, and zero for incorrect outputs. The discounted reward parameter $\tau_{pen}$ trades off aggression in the enforcement of brevity, preventing collapse into trivial, short, incorrect sequences.

Difficulty-Adaptive Advantage Estimation

Standard group-relative policy optimization approaches (e.g., GRPO, DAPO) normalize advantage using minibatch-standard-deviation, which is uninformative regarding actual difficulty: it generates similar normalization for trivially easy and nearly-unsolved instances. ExpThink replaces the denominator with a count of correct rollouts per task, making the magnitude of the policy gradient explicitly difficulty-aware. On instances where the policy already solves most or all sampled rollouts, the normalization suppresses gradients, increasing compression pressure; on hard tasks, gradients amplify, prioritizing correctness first.

Together, these mechanisms induce a learning dynamic optimally biased: the model is only challenged to compress responses once accuracy for a task is reliably attained, sidestepping the prevailing accuracy–efficiency trade-off intrinsic to fixed-penalty methods.

Training Objective

The training loop adopts a DAPO-style RL underpinning, with the shaped reward and difficulty-adaptive advantage applied to the policy update. Importantly, the method incorporates importance weight clipping to ensure stable off-policy updates when utilizing samples that diverge from the current policy.

Empirical Results

Compression and Accuracy Trade-off

ExpThink consistently demonstrates simultaneous improvement in accuracy and response length across distinct LRM scales. For example, with the Qwen3-8B backbone, ExpThink achieves a 77.2% reduction in token usage alongside a 5.33% improvement in accuracy, with an IPT up to 51.04—more than 3× that of the vanilla baseline. This dominance holds across all examined mathematical reasoning benchmarks, with compression and accuracy improving in distinctive, non-competing phases. The model is never penalized for verbosity before demonstrating reliability on correctness, as ensured by the curriculum encoded in the experience buffer.

Ablation studies emphasize the necessity of the three-tier reward and correct-count-based normalization: overly aggressive penalization ($\tau_{pen}=0$) or reverting to standard deviation normalization both yield substantial collapses in accuracy or undermine compression, respectively.

Overthinking Suppression and Difficulty Adaptivity

Lexical analysis of generated traces verifies that ExpThink suppresses overthinking markers (e.g., but, wait, alternatively) by over 78%, primarily eliminating hedging or cyclic reasoning while retaining depth where necessary for genuinely difficult tasks. The compression is difficulty-adaptive: easy instances are heavily compressed without loss of accuracy, while the model allocates token budget as needed for challenging inputs, often improving on both response length and solution rate for the hardest segments of each dataset.

Trajectory Diversity and Inference Parallelizability

ExpThink-trained models exhibit higher solution trajectory diversity at any fixed compute budget compared to both vanilla baselines and prior RL approaches. This manifests as higher coefficient of variation in output length, thus reducing compute waste during repeated sampling and increasing the effectiveness of majority-voting or sampling-based inference strategies.

Out-of-Distribution Generalization

The learned efficiency generalizes without further fine-tuning from mathematical reasoning benchmarks to code-oriented and scientific QA tasks (LiveCodeBench, GPQA-Diamond, MMLU), yielding both accuracy and IPT improvements over all considered baselines. This supports the claim that ExpThink does not rely on math-specific patterns but instead engrains a general capacity for concise, non-redundant reasoning.

Limitations and Future Directions

The framework presently relies on the availability of verifiably correct answers and is evaluated primarily in the setting of step-by-step mathematical reasoning. Transfer to multimodal, non-verifiable, or interactive settings (e.g., code execution, tool use) remains empirical open ground. Both the margin parameter for the reward buffer and the discounted reward level are somewhat empirical and may require domain-sensitive tuning.

Potential extensions include storing and replaying entire successful trajectories in the buffer for off-policy exploration and replay (as in RLHF-style experience replay), and extension to multimodal reasoning tasks where compression and accuracy must be balanced with visual grounding or tool-calling constraints.

Conclusion

ExpThink introduces cross-epoch experience-guided reward shaping and count-based advantage normalization into RL for CoT compression, overcoming fundamental inefficiencies in prior epoch-independent methods. It autonomously defines an implicit curriculum, allocating optimization pressure toward correctness on hard problems and toward brevity once reliability is achieved. This produces models that not only generate correct solutions with fewer tokens, but do so with greater diversity and sample efficiency, generalizing beyond in-domain tasks. The framework establishes new standards for intelligence-per-token in LRM reasoning, and highlights trajectory information reuse as a critical direction for future research in efficient generative modeling.