ExpThink: Adaptive CoT Compression

• ExpThink is an off-policy reinforcement learning framework that adapts chain-of-thought compression in large reasoning models to balance token efficiency and accuracy.
• It leverages an experience buffer and a difficulty-adaptive advantage estimator to dynamically set problem-specific compression targets based on historical correct responses.
• Empirical results demonstrate up to a 77% reduction in response length, enhanced accuracy, and a 5× increase in intelligence per token compared to baseline methods.

ExpThink is an off-policy reinforcement-learning framework for adaptive chain-of-thought (CoT) compression in large reasoning models (LRMs). It is designed for the setting in which extended CoT improves reasoning performance but incurs excessive token consumption and high inference latency. The framework addresses two limitations attributed to prior RL-based CoT compression methods—uniform, static length penalties and the absence of problem-level difficulty adaptation—through an experience buffer that records the shortest correct solution observed for each problem and a difficulty-adaptive advantage estimator based on the count of correct rollouts. In the reported experiments, ExpThink reduces average response length by up to 77\% while also improving accuracy, and achieves substantially higher accuracy-efficiency ratios than a vanilla baseline (Bian et al., 8 May 2026).

1. Problem formulation and design objective

ExpThink is introduced for training distributions of prompt–answer pairs $\langle q, y^* \rangle$, where, at each training step, a group of $G$ rollouts $\{o_i\}_{i=1}^G$ is sampled under the old policy $\pi_{\theta_{\mathrm{old}}}$. Each rollout $o_i$ has a length $|o_i|$ and produces a candidate answer $y_i$. The number of correct rollouts in the group is defined as

$$|\mathcal{C}_q| = \sum_{i=1}^{G}\mathbf{1}[\,y_i=y^*\,].$$

Within this setting, the framework targets a joint objective that can be summarized as “accuracy-first, compression-second.” The stated motivation is that standard RL approaches to CoT compression rely on static length penalties that do not reflect either model capability dynamics over training or variation in problem difficulty across prompts (Bian et al., 8 May 2026).

The conceptual distinction is important. ExpThink does not treat brevity as an unconditional optimization target. Instead, it conditions compression pressure on demonstrated correctness, both historically for each problem and locally within the current rollout group. This suggests that the method is intended to avoid the common failure mode in which aggressive compression degrades answer accuracy before the policy has become reliably competent.

2. Experience-guided reward shaping

The first core mechanism is experience-guided reward shaping. ExpThink maintains a global dictionary $\mathcal{E}$ mapping each prompt $q$ to the shortest correct rollout length ever observed. The running minimum at step $G$0 is

$G$1

with initialization $G$2, the context-window limit. Because this quantity is a running minimum, $G$3 can only decrease over training. The paper characterizes this as an automatic, per-problem tightening of the compression target (Bian et al., 8 May 2026).

Reward shaping uses a three-tier structure with tolerance parameter $G$4 and discount $G$5:

$G$6

This scheme distinguishes concise-correct, verbose-correct, and incorrect rollouts. Concise-correct rollouts receive full reward; verbose but correct rollouts receive discounted positive reward; incorrect rollouts receive zero reward. The practical consequence is that correct reasoning retains a training signal even when it is longer than the historical target, while the boundary between “concise” and “verbose” tightens automatically as shorter correct trajectories are discovered (Bian et al., 8 May 2026).

This mechanism differs from static length penalties in two ways. First, the compression threshold is problem-specific rather than global. Second, the threshold evolves as a consequence of search history rather than through an externally scheduled curriculum. A plausible implication is that prompts with intrinsically longer valid derivations are not immediately forced toward the same absolute length regime as simpler prompts.

3. Difficulty-adaptive advantage and optimization

The second core mechanism is difficulty-adaptive advantage. The paper contrasts ExpThink with standard GRPO or PPO formulations in which advantages are normalized by the group-wise standard deviation $G$7 of rewards. According to the paper, this normalization peaks when half the rollouts are correct and vanishes at both extremes, thereby cancelling the intended difficulty signal. ExpThink replaces standard-deviation normalization with correct-count normalization (Bian et al., 8 May 2026).

The estimator is

$G$8

with a small $G$9 to avoid division by zero, and with the denominator clamped so that it is at least $\{o_i\}_{i=1}^G$0 when no rollouts are correct. Because $\{o_i\}_{i=1}^G$1 directly measures how many samples in the group were solved correctly, the gradient magnitude becomes monotonically scaled by empirical difficulty (Bian et al., 8 May 2026).

The paper identifies two resulting regimes. On hard problems, where $\{o_i\}_{i=1}^G$2 is small, the denominator is small and gradients remain large, preserving pressure to improve accuracy. On easy problems, where $\{o_i\}_{i=1}^G$3 is large, the denominator is larger, which suppresses the overall gradient and makes the intra-group reward gap $\{o_i\}_{i=1}^G$4 more prominent. In the stated interpretation, this causes the policy to compress primarily when correctness is already reliable.

ExpThink then broadcasts $\{o_i\}_{i=1}^G$5 uniformly across the token sequence of rollout $\{o_i\}_{i=1}^G$6, giving a per-token advantage $\{o_i\}_{i=1}^G$7, and optimizes a DAPO-style clipped objective:

$\{o_i\}_{i=1}^G$8

The clipping thresholds satisfy $\{o_i\}_{i=1}^G$9 and are described as asymmetric thresholds inherited from DAPO. Inserting the shaped reward into the advantage estimator and then into this objective is presented as the mechanism by which the framework jointly enforces correctness preservation and response shortening (Bian et al., 8 May 2026).

4. Update procedure and algorithmic structure

The high-level ExpThink update step is given explicitly. For each minibatch of prompts $\pi_{\theta_{\mathrm{old}}}$0, the method first samples $\pi_{\theta_{\mathrm{old}}}$1 rollouts for each prompt under $\pi_{\theta_{\mathrm{old}}}$2, records rollout lengths and answers, computes the per-rollout shaped rewards, and updates the stored shortest-correct length $\pi_{\theta_{\mathrm{old}}}$3 using any newly observed correct rollout lengths (Bian et al., 8 May 2026).

In the second stage, the method computes the reward mean $\pi_{\theta_{\mathrm{old}}}$4 for each prompt, computes the correct-count $\pi_{\theta_{\mathrm{old}}}$5, and forms the rollout-level advantage $\pi_{\theta_{\mathrm{old}}}$6. In the third stage, the rollout-level advantage is broadcast to all tokens in the rollout, and tokenwise importance weights are computed. In the fourth stage, the clipped objective $\pi_{\theta_{\mathrm{old}}}$7 is formed and a gradient step $\pi_{\theta_{\mathrm{old}}}$8 is taken. The update concludes by setting $\pi_{\theta_{\mathrm{old}}}$9 (Bian et al., 8 May 2026).

This pipeline makes the experience buffer a persistent cross-epoch state, whereas the advantage normalization is recomputed from the current rollout group. The former supplies a historical compression target; the latter supplies an online estimate of problem difficulty. Their combination is the basis for the paper’s characterization of ExpThink as a self-evolving curriculum without manual scheduling.

5. Experimental configuration

The reported training corpus is DeepScaleR, described as approximately 40 K mathematically diverse problems. The backbones are DeepSeek-R1-Distill-Qwen-1.5B, Qwen-7B, and Qwen3-8B. Evaluation is conducted on in-domain mathematics benchmarks and on out-of-domain benchmarks covering code, graduate science, and broad subject-matter knowledge (Bian et al., 8 May 2026).

Category	Reported items
Training corpus	DeepScaleR (≈ 40 K mathematically diverse problems)
Backbones	DeepSeek-R1-Distill-Qwen-1.5B; Qwen-7B; Qwen3-8B
In-domain evaluation	AIME24; AMC23; MATH-500; Minerva Math; OlympiadBench
Out-of-domain evaluation	LiveCodeBench; GPQA-Diamond; MMLU
Metrics	Pass@1 accuracy (Acc); Average response length in tokens (Token); Intelligence per Token (IPT) = Acc / (Token/1000); Average relative change $o_i$0Acc, $o_i$1Token vs. “Vanilla”

The training details are also specified. The implementation uses 8 × A800 80 GB GPUs, global batch size 512, maximum context length 16,384, and FSDP with CPU offload. PPO-style clipping uses $o_i$2, the learning rate is $o_i$3, the group size is $o_i$4, the reward discount is $o_i$5, and the tolerance is $o_i$6. Training proceeds for 300 steps, and validation uses AIME24 repeated 16 times and AMC23 repeated 8 times for stable estimates (Bian et al., 8 May 2026).

These details delimit the empirical scope of the reported results. In particular, the main evidence is centered on mathematical reasoning while also probing transfer to non-math domains through the out-of-domain benchmarks.

6. Reported empirical outcomes and interpretation

The principal quantitative claim is that ExpThink reduces average response length by up to 77.2\% while simultaneously improving accuracy. On Qwen3-8B, the paper reports a 77.2\% reduction in response length and a 5.3 percentage-point accuracy increase, with Intelligence per Token rising from 8.64 to 51.04, reported as a +5.9× change. On Qwen-7B and 1.5B, average token reduction exceeds 62\%, with accuracy gains of 1.8 percentage points and 4.5 percentage points respectively, yielding up to 3× higher IPT (Bian et al., 8 May 2026).

The comparative claim against prior RL-based compression baselines is also explicit: among LC-R1, Laser, AdaptThink, and JET, ExpThink is reported as the only method that simultaneously improves accuracy and achieves the shortest traces across all scales. The concluding remarks summarize the broader result as up to 77\% shorter reasoning traces with no accuracy loss and up to 5× higher intelligence per token across three model scales and multiple benchmarks (Bian et al., 8 May 2026).

Several interpretive points follow directly from the reported design. First, ExpThink should not be understood as a simple length-penalty method; verbose-but-correct trajectories still receive positive reward, and gradient scaling explicitly preserves learning pressure on hard problems. Second, the method should not be understood as relying on a hand-designed curriculum schedule; the shortening target emerges from the running minimum of correct trajectory lengths. Third, the framework’s compression behavior is conditioned on demonstrated competence, rather than imposed uniformly across prompts. This suggests that its central contribution lies in coupling historical trajectory reuse with count-based difficulty weighting rather than in introducing a fixed regularizer for short outputs.