Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models (2509.26626v1)

Abstract: Test-time scaling methods improve the capabilities of LLMs by increasing the amount of compute used during inference to make a prediction. Inference-time compute can be scaled in parallel by choosing among multiple independent solutions or sequentially through self-refinement. We propose Recursive Self-Aggregation (RSA), a test-time scaling method inspired by evolutionary methods that combines the benefits of both parallel and sequential scaling. Each step of RSA refines a population of candidate reasoning chains through aggregation of subsets to yield a population of improved solutions, which are then used as the candidate pool for the next iteration. RSA exploits the rich information embedded in the reasoning chains -- not just the final answers -- and enables bootstrapping from partially correct intermediate steps within different chains of thought. Empirically, RSA delivers substantial performance gains with increasing compute budgets across diverse tasks, model families and sizes. Notably, RSA enables Qwen3-4B-Instruct-2507 to achieve competitive performance with larger reasoning models, including DeepSeek-R1 and o3-mini (high), while outperforming purely parallel and sequential scaling strategies across AIME-25, HMMT-25, Reasoning Gym, LiveCodeBench-v6, and SuperGPQA. We further demonstrate that training the model to combine solutions via a novel aggregation-aware reinforcement learning approach yields significant performance gains. Code available at https://github.com/HyperPotatoNeo/RSA.

• The paper demonstrates that RSA recursively aggregates candidate solutions to enhance deep, multi-step reasoning in LLMs.
• It combines parallel and sequential reasoning strategies to refine outputs and consistently outperforms existing test-time scaling methods.
• Aggregation-aware RL further improves generalization and performance across diverse benchmarks, bridging the gap between lightweight and heavyweight models.

Recursive Self-Aggregation Unlocks Deep Thinking in LLMs

Introduction

The paper introduces Recursive Self-Aggregation (RSA), a hybrid test-time scaling framework for LLMs that leverages both parallel and sequential reasoning strategies. RSA is motivated by evolutionary algorithms and is designed to improve LLM reasoning capabilities by recursively aggregating candidate solutions, enabling the model to recombine and refine reasoning chains without external verifiers or model parameter updates. The approach is evaluated across diverse tasks and model architectures, demonstrating substantial improvements over existing test-time scaling methods.

Taxonomy of Test-Time Scaling Methods

Test-time scaling methods for LLMs are categorized by their verification strategy and reasoning control flow:

• Verification Strategies:
  • External Verification: Uses external tools or learned reward models to score candidate solutions.
  • Self-Verification: Employs the LLM itself to judge correctness, exploiting the generation-verification gap.
  • Implicit Verification: Relies on the LLM to implicitly verify and improve solutions without explicit scoring.
• Reasoning Control Flows:
  • Parallel Scaling: Generates multiple independent reasoning chains and combines them (e.g., majority voting, Best-of-N).
  • Sequential Scaling: Iteratively refines a single reasoning chain, suitable for deep, multi-step reasoning.
  • Hybrid Scaling: Combines parallel and sequential strategies, often using evolutionary or ensemble methods.

RSA is positioned as a hybrid scaling method, integrating recursive aggregation steps into a self-improvement loop, maintaining a population of candidate solutions and iteratively recombining subsets to produce improved solutions.

Recursive Self-Aggregation (RSA) Algorithm

RSA operates as follows:

1. Initialization:
   • Generate an initial population $P_1$ of $N$ candidate solutions for a given query $x$ using the base LLM.
2. Subsampling and Aggregation:
   • For each of $N$ aggregation sets, sample $K$ distinct candidates from the current population.
   • Prompt the LLM with the query and the sampled set to produce an improved solution.
   • Form the next population $P_{t+1}$ from these aggregated solutions.
3. Recursion:
   • Repeat the subsampling and aggregation for $T$ steps, recursively refining the population.
4. Termination:
   • The final solution is selected from the last population, either by random sampling or majority voting.

Key implementation details:

• The aggregation prompt is designed to encourage the model to combine useful ideas and correct errors from multiple candidates.
• The choice of $K$ (aggregation set size), $N$ (population size), and $T$ (number of steps) are critical hyperparameters, with trade-offs between diversity, convergence speed, and compute requirements.
• RSA can be implemented with any LLM inference pipeline and does not require external verification or model retraining.

Pseudocode

def rsa(LLM, query, N, K, T): # Initialize population population = [LLM.generate(query) for _ in range(N)] for t in range(T): new_population = [] for _ in range(N): subset = random.sample(population, K) prompt = build_aggregation_prompt(query, subset) new_solution = LLM.generate(prompt) new_population.append(new_solution) population = new_population return select_final_solution(population)

Aggregation-Aware Reinforcement Learning

Standard RL post-training for LLMs does not align with the aggregation task at inference, often degrading RSA performance due to distribution shift. The paper proposes aggregation-aware RL, which augments the training dataset with aggregation prompts containing multiple candidate solutions. The RL objective is modified to optimize both standard and aggregation prompts, enabling the model to learn aggregation skills directly.

• Objective for Standard Prompts:

$$\max \mathbb{E}_{(x, y) \sim D} \left[ \mathbb{E}_{T \sim p_\theta(\cdot|x)} [r(T, y)] - \beta \mathrm{KL}(p_\theta(\cdot|x) \| p_{\text{ref}}(\cdot|x)) \right]$$

• Objective for Aggregation Prompts:

$$\max \mathbb{E}_{(x, y) \sim D, S_0 \sim p_{\text{ref}}(\cdot|x)} \left[ \mathbb{E}_{T \sim p_\theta(\cdot|x, S_0)} [r(T, y)] - \beta \mathrm{KL}(p_\theta(\cdot|x, S_0) \| p_{\text{ref}}(\cdot|x, S_0)) \right]$$

This approach is compatible with standard policy gradient algorithms (e.g., PPO, RLOO) and can be implemented with parameter-efficient fine-tuning.

Empirical Results

RSA is evaluated on math (AIME-25, HMMT-25), code generation (LiveCodeBench-v6), general reasoning (Reasoning Gym), and knowledge recall (SuperGPQA) benchmarks, using Qwen3-4B-Instruct-2507 and other models. Key findings:

• Performance Gains:
  • RSA consistently outperforms sequential (self-refinement) and parallel (majority voting, rejection sampling) baselines.
  • RSA enables smaller models (e.g., Qwen3-4B-Instruct-2507) to match or exceed the performance of larger models (e.g., DeepSeek-R1, o3-mini (high)).
  • Aggregation-aware RL further amplifies RSA's benefits, yielding superior results compared to standard RL fine-tuning.
• Scaling Behavior:
  • Increasing $K$ (aggregation set size) improves performance, with diminishing returns beyond $K=3$ due to context length constraints.
  • Larger $N$ (population size) enhances diversity and asymptotic performance but requires more steps $T$ for convergence.
  • Performance improves monotonically with $T$ (number of recursive steps), except in cases where diversity is lost too quickly.
• Generalization:
  • Aggregation-aware RL exhibits strong out-of-domain transferability, improving performance on tasks not present in the training set.

Implementation Considerations

• Computational Requirements:
  • RSA increases inference-time compute linearly with $N \times T$, but parallelization is straightforward.
  • Context length constraints limit the effective aggregation set size $K$; prompt engineering can mitigate this.
• Deployment:
  • RSA can be integrated into existing LLM serving pipelines with minimal changes.
  • Aggregation-aware RL fine-tuning requires additional data generation and training but is compatible with standard RLHF frameworks.
• Limitations:
  • Excessive population size $N$ without sufficient aggregation steps $T$ can slow convergence.
  • Loss of diversity due to repeated aggregation may hinder exploration of alternative reasoning paths.

Theoretical and Practical Implications

RSA demonstrates that test-time scaling via recursive aggregation can unlock deeper reasoning in LLMs, bridging the gap between lightweight and heavyweight models. The evolutionary perspective enables the reuse of partially correct intermediate steps, improving robustness and solution quality. Aggregation-aware RL aligns training objectives with inference strategies, mitigating distribution shift and enhancing generalization.

Future directions include integrating explicit fitness functions (e.g., self-verification), composing RSA with other test-time scaling methods, and developing multi-step RL policies for end-to-end RSA optimization.

Conclusion

Recursive Self-Aggregation provides a principled and effective framework for enhancing LLM reasoning at inference time, combining the strengths of parallel and sequential scaling. The method is simple to implement, generalizes across tasks and models, and benefits from aggregation-aware RL fine-tuning. RSA sets a new standard for test-time scaling, with clear implications for the development of more capable and efficient LLMs.