Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning

Abstract: Retrieval-augmented generation (RAG) has become a standard mechanism for grounding LLMs in external knowledge, yet conventional retrieval based on lexical or semantic similarity is poorly suited for complex reasoning tasks: a semantically similar problem may demand an entirely different solution strategy, while a superficially different problem may share the same underlying reasoning pattern. We propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that teaches LLMs to reason by analogy. RA-RFT uses gold-relevance distillation to train a retriever that ranks contexts by expected reasoning benefit rather than semantic overlap, and then fine-tunes the policy model via reinforcement fine-tuning methods with retrieved analogous demonstrations, so the model learns to leverage reasoning traces under verifiable outcome rewards. We further analyze the diversity of retrieved contexts and find that reasoning-aware retrieval surfaces complementary solution strategies that provide distinct reasoning scaffolds for individual problems. Across challenging mathematical reasoning benchmarks, RA-RFT consistently outperforms standard reinforcement fine-tuning methods. For example, it improves AIME 2025 average@32 accuracy by 7.1 and 2.8 points over GRPO for Qwen3-1.7B and Qwen3-4B respectively -- suggesting that reasoning-aware retrieval is a complementary axis of improvement and orthogonal to advances in reward design or training curricula.

• The paper introduces RA-RFT, a novel framework integrating reasoning-aware retrieval into RL fine-tuning for improved analogical reasoning.
• It leverages gold-relevance distillation and dense retriever training to prioritize structural similarity, achieving up to +7.1 points gain on math benchmarks.
• The approach underscores the importance of structural over semantic similarity, enabling effective transfer of problem-solving strategies across diverse tasks.

Retrieval-Augmented Reinforcement Fine-Tuning for Analogical Reasoning in LLMs

Introduction and Motivation

The paper "Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning" (2606.13680) identifies a critical gap in LLM-based mathematical reasoning: existing RL-from-verifiable-rewards (RLVR) protocols, such as GRPO, operate solely over model-parametric knowledge and are unable to effectively transfer solution strategies across superficially distinct but structurally analogous problems. While RAG pipelines have become standard for knowledge grounding, retrieval based on lexical or semantic similarity is misaligned with the requirement for reasoning transfer in open-domain math and science domains, often surfacing irrelevant exemplars or misleading the policy.

The motivation for RA-RFT is grounded in empirical observations and cognitive science research: expert problem-solvers utilize analogical reasoning, retrieving prior solution strategies whose transferability lies in similarities of reasoning structure rather than shared surface content. Consequently, optimizing retrieval pipelines to align with reasoning utility, not just semantic overlap, can bridge sparsity in parametric knowledge and enhance the effectiveness of RLVR.

Figure 1: The quality of retrieved context is critical for RLVR training: surface similarity can be misleading, while structurally analogous traces improve policy learning.

RA-RFT Framework

RA-RFT is a three-stage framework that operationalizes reasoning-aware retrieval within RL fine-tuning. The pipeline comprises:

1. Gold-Relevance Distillation: A strong judge model (GPT-4o) is used to annotate reasoning-utility relevance between queries and candidate corpus traces. The judge is provided with a pair of problems and their traces and returns a binary label indicating whether the candidate trace’s solution methods offer structurally transferable strategies for the query.
2. Reasoning-Aware Retriever Training: With these annotations, a dense retriever (Reason-ModernColBERT) is fine-tuned using contrastive InfoNCE loss to maximize alignment between queries and reasoning-relevant traces. This explicit supervision ensures retrieval prioritizes structural reasoning similarity over plain semantics.
3. RL Fine-Tuning with Retrieved Demonstrations: During RL rollouts, the policy model receives, alongside each query, a top-$k$ retrieved reasoning trace as demonstration. Policy optimization (e.g., GRPO, RLOO, DAPO) is then executed in this retrieval-augmented space, with the model rewarded only for verifiable, correct final answers. This directly incentivizes the model to synthesize and adapt solution strategies from retrieved analogues, learning when and how to utilize demonstrations rather than merely imitating them.

   Figure 2: The RA-RFT pipeline: gold-relevance distillation with a judge model, contrastive retriever training, and RL with reasoning-analogous context augmentation.

Experimental Results and Analysis

The empirical evaluation on competition-grade benchmarks (AIME 2024/2025, HMMT, BrUMO) across Qwen3-1.7B and Qwen3-4B yields several key findings:

• Consistent Outperformance vs. RLVR+SFT Baselines: RA-RFT outperforms strong RLVR baselines. On Qwen3-1.7B, it achieves +7.1 points over GRPO on AIME 2025 and +4.1 average gain across all benchmarks. On Qwen3-4B, gains persist despite a stronger base, confirming orthogonality to reward shaping, curriculum tuning, or increased scale.
• Retrieval Quality as a Primary Bottleneck: Ablation studies highlight that the quality of retrieved traces is critical; structurally irrelevant traces (e.g., those surfaced by semantic-only retrievers) can degrade performance or provide no benefit. Fine-tuning a multi-vector retriever with reasoning relevance supervision yields recall@1 over 43%, and the downstream accuracy directly tracks improvements in retriever fidelity.
• Supervised Fine-Tuning Is Insufficient: Simply augmenting SFT with retrieved traces (RA-SFT) provides negligible benefit—the efficacy arises only when the objective allows for exploration and selective integration under verifiable reward.
• Contextual Diversity Matters: For challenging problems, different retrieved traces can lead to widely varying sample-level accuracy, exposing the non-trivial effect of reasoning scaffold selection. Some problems are only solvable with structurally analogous context not discoverable through typical RL exploration.

  Figure 3: RA-RFT achieves superior learning curves and final accuracy compared to GRPO, with especially strong gains as the policy learns to utilize retrieved traces.

  

  Figure 4: Per-sample accuracy distribution under different retrieved contexts reveals that retrieval surfaces complementary solution strategies, enabling gains for previously unsolved cases.

  

  Figure 5: Case study: RA-RFT, with a reasoning-analogous trace about coloring a convex n-gon, enables correct block-counting on an adjacency-based configuration problem, while standard GRPO applies a non-transferable DP recurrence and fails.

Architectural Components, Prompting, and Corpus Construction

• Prompt Engineering: The prompt format for both RL training and inference explicitly prepends the retrieved demonstration (problem and compact step-trace) to the user query, maximizing the opportunity for analogical transfer.

  Figure 6: Prompt structure for RA-RFT: a retrieved reference question and compact reasoning trace conditions the model before the main query.

• Corpus Curation: The retrieval corpus is generated by high-quality teacher traces (Qwen3-235B-A22B), which are then further condensed into stepwise traces using prompt-based summarization to extract reusable structural strategies.

  Figure 7: Raw solutions are condensed into step-by-step traces to highlight transferable reasoning strategies.

• Gold-Relevance Judging: The judge prompt is crafted to elicit structural reasoning similarity, not simply surface overlap.

  Figure 8: GPT-4o judge prompt template for gold-relevance distillation explicitly asks for structural reasoning similarity.

Implications and Future Directions

RA-RFT demonstrates that retrieval-augmented reinforcement fine-tuning, when the retriever is explicitly supervised for reasoning utility, enables parameter-efficient transfer of problem-solving strategies and robust performance improvements on complex mathematical reasoning. This finding underscores several broader theoretical and practical implications:

• Separation of Structural vs. Surface Similarity: For any reasoning-intensive domain (math, code, theorem proving), corpus structure and retriever supervision must align with structural rather than semantic similarity. This decoupling introduces a new axis for retrieval model optimization, orthogonal to model or optimizer scaling.
• Analogical Reasoning as a Precursor to More Generalizable LLM Policies: As AI systems are increasingly deployed in open-world scientific and technical domains, retrieval-augmented frameworks modularizing structural analogy may prove essential.
• Reusability Across Domains and Architectures: The retriever is trained once and can be reused across models, and the approach is compatible with different policy optimization backends, making it broadly extensible.
• Limitations: The framework introduces an upfront cost (gold-relevance annotation), but this is amortized and modest relative to RLVR compute budgets. The primary bottleneck becomes corpus and annotation quality, indicating fruitful directions for improving silver-relevance distillation, corpus expansion, or semi-supervised trace summarization.

Conclusion

RA-RFT advances the state of the art in reasoning-intensive RL for LLMs, offering a principled and empirically validated approach for analogical transfer by conditioning policy optimization on structurally relevant demonstrations. The critical insight—grounding retrieval in reasoning utility and integrating context during exploration, not only at test time—outperforms approaches based on imitation or semantic context injection and opens new directions for AI research in modularizing, scaling, and generalizing machine reasoning beyond surface correspondences.