InftyThink+: Effective and Efficient Infinite-Horizon Reasoning via Reinforcement Learning

Abstract: Large reasoning models achieve strong performance by scaling inference-time chain-of-thought, but this paradigm suffers from quadratic cost, context length limits, and degraded reasoning due to lost-in-the-middle effects. Iterative reasoning mitigates these issues by periodically summarizing intermediate thoughts, yet existing methods rely on supervised learning or fixed heuristics and fail to optimize when to summarize, what to preserve, and how to resume reasoning. We propose InftyThink+, an end-to-end reinforcement learning framework that optimizes the entire iterative reasoning trajectory, building on model-controlled iteration boundaries and explicit summarization. InftyThink+ adopts a two-stage training scheme with supervised cold-start followed by trajectory-level reinforcement learning, enabling the model to learn strategic summarization and continuation decisions. Experiments on DeepSeek-R1-Distill-Qwen-1.5B show that InftyThink+ improves accuracy by 21% on AIME24 and outperforms conventional long chain-of-thought reinforcement learning by a clear margin, while also generalizing better to out-of-distribution benchmarks. Moreover, InftyThink+ significantly reduces inference latency and accelerates reinforcement learning training, demonstrating improved reasoning efficiency alongside stronger performance.

• The paper presents InftyThink+, a reinforcement learning framework that improves accuracy and reduces computational costs in infinite-horizon reasoning by employing iterative summarization.
• InftyThink+ autonomously decides on summarization timing and its content, aligning decision-making across trajectory lengths and dramatically reducing induction-time costs.
• The framework enhances inference through structured, iterative summaries, yielding significant accuracy and speed-ups

Motivation and problem setting

Large reasoning models scale inference-time computation by producing long chains-of-thought (CoT), but this paradigm faces three structural barriers: quadratic self-attention cost in generation length, a hard ceiling imposed by the maximum context window, and degraded reasoning from lost-in-the-middle effects as traces grow. Iterative reasoning mitigates these issues by periodically compressing intermediate context and resuming reasoning, yet prior approaches either rely on supervised fine-tuning (SFT) that only imitates a format, or on fixed heuristics such as fixed-size chunking. The paper identifies three decisions that determine the quality of iterative reasoning — when to summarize, what to preserve, and how to resume — and argues these are sequential decisions with long-horizon consequences that SFT fundamentally cannot optimize.

The proposed method, InftyThink+, is an end-to-end reinforcement learning framework built on the InftyThink paradigm: the model decomposes reasoning into multiple iterations, each conditioned only on the original query and the previous self-generated summary, and autonomously decides when to terminate by emitting a conclusion instead of a summary. The central claim is that trajectory-level RL over this structure yields both higher accuracy and lower inference and training cost than conventional long-CoT RL.

Method

Training proceeds in two stages. Cold start: vanilla reasoning triples $(q, r, c)$ are transformed into InftyThink-style instances by segmenting $r$ under a length bound $\eta$ (6k tokens), generating summaries with an external LLM (Qwen3-4B-Instruct-2507) constrained to $\gamma = 1$k tokens, and performing SFT with special tokens for summary and history; loss is masked over query and history tokens. Notably, the authors deviate from the original InftyThink data construction by generating each summary conditioned only on $(r_i, s_{i-1})$ rather than the full reasoning prefix, aligning training-time information flow with inference-time constraints and reducing hallucination risk.

Reinforcement learning: rollouts are trajectory-level — a single query induces up to $\varphi$ sequential generations connected by extracted summaries, terminated on conclusion, format failure, or reaching $\varphi$. Rewards are assigned at the trajectory level and broadcast to all rounds: a binary task reward plus an optional efficiency reward $1 - ((n_i - 1)/\varphi)^2$, combined multiplicatively so efficiency shapes only correct trajectories. Optimization uses GRPO with token-level loss averaging (DAPO-style), asymmetric clipping, and shared advantages across all tokens of a trajectory — a design choice that credits early-round summaries for downstream success. IcePop-style gradient masking stabilizes training against train–inference probability mismatch. Format compliance requires no explicit format reward because malformed outputs cannot reach a verifiable answer and thus receive near-zero task reward.

The appendix provides theoretical support: an information-bottleneck argument showing that SFT objectives are independent of answer correctness and misaligned with the policy's own continuation capabilities, hence cannot optimize summary informativeness; and a complexity analysis showing InftyThink reduces attention cost from $O(L^2)$ to approximately $O(n\ell^2)$, i.e., roughly a factor of $r$0 when summaries are short relative to segments.

Main results

Experiments use DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-4B-Base, trained on DeepScaleR-Preview with GRPO, evaluated with 32 samples per query at temperature 0.7 using CompassVerifier-7B. Key findings:

Setting	Avg ACC (%)	Avg LAT (s)
Vanilla, no RL	41.69	110.96
Vanilla + task RL	47.31	149.44
InftyThink+ cold start only	44.06	77.57
InftyThink+ + task RL	53.96	100.21
InftyThink+ + task + efficiency RL	50.58	48.37

On AIME24, InftyThink+ with task-only RL reaches 50.94% versus 38.75% for vanilla RL — a gain of +21.46 points over cold start, compared to +12.08 for vanilla RL — indicating that structured iterative summaries provide a more exploitable substrate for RL optimization. On out-of-distribution GPQA-Diamond, gains of +5.19 points exceed vanilla's +0.41. Latency results are equally notable: even without RL, InftyThink+ cuts average latency from 110.96s to 77.57s despite slightly more tokens, and after task-only RL latency remains nearly flat on MATH500 and AIME24 while accuracy rises sharply — evidence that iterative summarization decouples reasoning depth from wall-clock time. Adding the efficiency reward trades modest accuracy (53.96 → 50.58 average) for a 52% latency reduction (100.21s → 48.37s). Gains generalize to Qwen3-4B-Base and to OOD code (HumanEval/MBPP) and scientific benchmarks (MMLU-redux, PHYBench).

RL training itself accelerates: per-step time drops from ~300–325s (vanilla) to ~225s, and further to ~175s with the efficiency reward, yielding roughly 25–40% speedup.

Analyses of when, how, and how-to-continue

Three controlled ablations isolate the learned strategic behaviors. When to compress: replacing adaptive interruption timing with fixed (5k-token) or random (3k–6k) budgets consistently degrades performance, and the penalty grows after RL (e.g., −3.02 on AIME24 for random timing), indicating RL sharpens timing policy. How to compress: a replacement experiment swaps internal summaries for external LLM summaries at inference. Before RL, external summaries improve accuracy (AIME24: 29.48 → 32.40), confirming SFT teaches format rather than summary quality; after RL, the substitution degrades performance (50.94 → 48.42), demonstrating that RL couples summary generation end-to-end with the policy's own continuation ability. How to continue: feeding InftyThink+ summaries into a vanilla-paradigm model yields worse continuation than the trained model's own continuation, with gains saturating after early iterations — late-stage summaries are intrinsically harder to exploit, and only end-to-end training learns to do so.

Additional analyses show performance scales monotonically with iteration budget, with front-loaded gains; RL-trained models extract more value from additional rounds than pre-RL models. Ablations over the iteration cap $r$1 and segment window $r$2 characterize them as complementary compute knobs: larger $r$3 improves task reward but increases variance and slows credit assignment, while larger $r$4 raises per-iteration quality and reduces turn count at higher latency. An ablation removing cold start shows systematic RL collapse (reward drop, pg-loss spike, entropy explosion) reproduced across two runs, establishing cold start as a prerequisite rather than a convenience. A checkpoint-level comparison against Delethink under matched base model, data, and configuration shows InftyThink+ consistently above Delethink on AIME24/25 throughout training.

Limitations

The paper concedes several constraints. The method assumes reasoning decomposes into relatively independent stages whose essential content fits a summary; tasks with entangled reasoning or strong reliance on continuous semantic flow may not benefit. Summaries are unstructured natural-language tokens, offering expressive flexibility but no explicit control over information priority or constraint strength, limiting fine-grained compression fidelity. The pipeline depends on task-specific cold-start data construction, adding engineering overhead for new domains. Evaluation is limited to 1.5B and 4B models, and the efficiency reward trades measurable accuracy for speed, so the appropriate operating point depends on deployment priorities. Whether the information-bottleneck framing translates into practical summary-quality metrics remains unaddressed.

Conclusion

InftyThink+ demonstrates that the strategic decisions underlying iterative reasoning — interruption timing, summary content, and continuation — can be optimized end-to-end via trajectory-level RL with shared advantages and multiplicative task-efficiency rewards. The resulting policies outperform both SFT-based iterative reasoning and standard long-context RL on mathematical, scientific, and code benchmarks while substantially reducing inference latency and RL training cost. The open questions left by the work concern scaling beyond small models, structured or latent summary representations, and applicability to agentic settings where stage boundaries are less well defined.