RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems (2510.02263v1)

Abstract: Reasoning requires going beyond pattern matching or memorization of solutions to identify and implement "algorithmic procedures" that can be used to deduce answers to hard problems. Doing so requires realizing the most relevant primitives, intermediate results, or shared procedures, and building upon them. While RL post-training on long chains of thought ultimately aims to uncover this kind of algorithmic behavior, most reasoning traces learned by large models fail to consistently capture or reuse procedures, instead drifting into verbose and degenerate exploration. To address more effective reasoning, we introduce reasoning abstractions: concise natural language descriptions of procedural and factual knowledge that guide the model toward learning successful reasoning. We train models to be capable of proposing multiple abstractions given a problem, followed by RL that incentivizes building a solution while using the information provided by these abstractions. This results in a two-player RL training paradigm, abbreviated as RLAD, that jointly trains an abstraction generator and a solution generator. This setup effectively enables structured exploration, decouples learning signals of abstraction proposal and solution generation, and improves generalization to harder problems. We also show that allocating more test-time compute to generating abstractions is more beneficial for performance than generating more solutions at large test budgets, illustrating the role of abstractions in guiding meaningful exploration.

• The paper introduces RLAD, a framework that trains LLMs to discover high-level reasoning abstractions to guide solution generation.
• It employs a two-agent reinforcement learning setup with reward masking and curriculum training, achieving notable pass@k and coverage improvements.
• Results demonstrate that abstraction-driven reasoning enhances semantic diversity and robustness across diverse domains like math and program synthesis.

RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems

Introduction and Motivation

The RLAD framework addresses a central challenge in LLM-based reasoning: the tendency of RL-finetuned models to generate verbose, locally coherent but globally suboptimal chains of thought, often failing to reuse or generalize procedural knowledge. RLAD introduces the concept of reasoning abstractions—concise, natural language representations of procedural or factual knowledge that serve as high-level guides for solution generation. The method operationalizes abstraction discovery and utilization via a two-player RL paradigm, jointly training an abstraction generator and an abstraction-conditioned solution generator. This approach aims to decouple the exploration of high-level strategies from low-level solution search, thereby improving both the diversity and quality of reasoning traces.

Reasoning Abstractions: Definition and Empirical Utility

Reasoning abstractions are defined as compressed, input-dependent textual artifacts that encode reusable procedures, lemmas, or heuristics relevant to a given problem. Unlike static retrieval-augmented generation (RAG) or prompt engineering, RLAD's abstractions are dynamically generated by the model itself, conditioned on the problem instance.

Empirical evaluation demonstrates that conditioning solution generation on high-quality abstractions yields consistent improvements in pass@k and coverage metrics across math and program synthesis benchmarks. For example, on the ARC-AGI benchmark, abstraction conditioning improves pass@16 from 24.7% to 33.2% and coverage from 35.0% to 42.9%. These gains are robust to the number of samples and abstraction diversity, and are observed across domains including math, law, and healthcare.

Figure 1: Illustration of abstraction categories, showing the diversity of procedural and factual knowledge types captured by RLAD's abstraction generator.

Figure 2: Grid plot showing that abstraction conditioning improves accuracy, especially when abstractions are sufficiently informative and the solution generator is capable of leveraging them.

RLAD Training Paradigm

RLAD employs a cooperative two-agent RL setup:

• Abstraction Generator ($abs$): Proposes one or more abstractions $z$ given a problem $x$.
• Solution Generator ($sol$): Generates a solution $\tilde{y}$ conditioned on both $x$ and $z$.

The reward for $abs$ is the expected accuracy of $sol$ when conditioned on the proposed abstraction, while $sol$ is directly rewarded for correctness. To prevent degenerate solutions (e.g., abstractions that leak answers or are ignored), RLAD introduces several critical design choices:

• Reward Masking: Rewards for solution traces without abstractions are zeroed out, ensuring $sol$ must utilize abstractions to optimize reward.
• Curriculum Training: Training proceeds from easy to hard problems, improving stability and generalization.
• Warmstarting: $abs$ is initialized via SFT on synthetic problem-abstraction pairs, generated by summarizing diverse solution traces using a strong LLM and filtered to avoid answer leakage.

  Figure 3: RLAD training workflow. The abstraction generator proposes abstractions, which are then used by the solution generator to produce answers. Rewards are computed based on the solution's correctness, and both agents are updated via RL.

Experimental Results

Math Reasoning Benchmarks

RLAD is evaluated on AIME 2025, DeepScaleR Hard, and AMC 2023, using Qwen3-1.7B as the base model. RLAD consistently outperforms both the base model and DAPO RL-finetuning:

Approach	AIME 2025 (w/ abs, best)	DeepScaleR Hard (w/ abs, best)	AMC 2023 (w/ abs, best)
Qwen-3-1.7B	40.00	32.50	84.53
+ DAPO	39.79	33.54	88.44
+ RLAD	48.33	35.54	91.72

Notably, RLAD-trained models also generalize better even when no abstraction is provided at inference, indicating that exposure to diverse abstractions during training enhances the model's intrinsic reasoning capabilities.

Compute Allocation: Abstractions vs. Solution Sampling

A key finding is that, under fixed inference compute budgets, allocating more compute to generating diverse abstractions (rather than simply sampling more solutions per abstraction) yields superior performance, especially as the total compute budget increases. This suggests that abstraction diversity is a more effective axis for scaling test-time compute than solution sampling alone.

Figure 4: Tradeoff between abstraction and solution generation on AIME 2025. Allocating more compute to abstraction generation yields better performance efficiency across compute budgets and normalization offsets.

Semantic Diversity and Adherence

RLAD-trained solution generators produce solution traces that are both more semantically diverse (when conditioned on different abstractions) and more adherent to the provided abstraction, as measured by LLM-based classifiers and embedding similarity. This indicates that abstractions are not only utilized but also drive exploration of distinct solution strategies.

Figure 5: Example of a reasoning abstraction and its influence on the solution trace. The solution generator references and meaningfully incorporates keywords from the abstraction.

Ablation and Generalization

Ablation studies confirm the necessity of curriculum training, inclusion of no-abstraction prompts, and reward masking for optimal performance. RLAD's abstraction-based improvements generalize to non-mathematical domains, with average and best abstraction conditioning outperforming standard prompting by 18% and 30% respectively across 37 tasks.

Theoretical and Practical Implications

RLAD's abstraction mechanism can be viewed as a form of learned, input-dependent option discovery in the RL literature, where abstractions serve as high-level subgoals or skills. This approach enables structured exploration and mitigates the "depth-only" bias of standard RL-finetuned LLMs. The empirical results suggest that abstraction-guided reasoning is a scalable and generalizable paradigm, orthogonal to model size and chain-of-thought length scaling.

From a practical perspective, RLAD provides a blueprint for training LLMs that can both propose and leverage high-level strategies, improving sample efficiency and robustness on hard reasoning tasks. The method is compatible with both online and offline RL, and can be integrated with existing LLM architectures and RLHF pipelines.

Limitations and Future Directions

While RLAD demonstrates strong results on mathematical and program synthesis tasks, its efficacy on open-ended or less-structured reasoning remains to be established. Training a single model to both propose and utilize abstractions proved challenging, with RL quickly degrading abstraction generation capabilities. Addressing this may require large-scale mid-training or active RL interventions. Additionally, understanding the mechanisms by which abstraction exposure improves generalization—even in the absence of abstractions at inference—remains an open research question.

Conclusion

RLAD introduces a principled framework for abstraction discovery and utilization in LLM-based reasoning, yielding substantial improvements over prior RL-finetuning approaches. By decoupling high-level strategy exploration from low-level solution search, RLAD enables more diverse, robust, and generalizable reasoning. The method's insights into compute allocation and abstraction-driven exploration have significant implications for the future of LLM training and deployment, suggesting new directions for scaling, generalization, and interpretability in AI reasoning systems.