KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance

Abstract: RLVR improves reasoning in LLMs, but its effectiveness is often limited by severe reward sparsity on hard problems. Recent hint-based RL methods mitigate sparsity by injecting partial solutions or abstract templates, yet they typically scale guidance by adding more tokens, which introduce redundancy, inconsistency, and extra training overhead. We propose \textbf{KnowRL} (Knowledge-Guided Reinforcement Learning), an RL training framework that treats hint design as a minimal-sufficient guidance problem. During RL training, KnowRL decomposes guidance into atomic knowledge points (KPs) and uses Constrained Subset Search (CSS) to construct compact, interaction-aware subsets for training. We further identify a pruning interaction paradox -- removing one KP may help while removing multiple such KPs can hurt -- and explicitly optimize for robust subset curation under this dependency structure. We train KnowRL-Nemotron-1.5B from OpenMath-Nemotron-1.5B. Across eight reasoning benchmarks at the 1.5B scale, KnowRL-Nemotron-1.5B consistently outperforms strong RL and hinting baselines. Without KP hints at inference, KnowRL-Nemotron-1.5B reaches 70.08 average accuracy, already surpassing Nemotron-1.5B by +9.63 points; with selected KPs, performance improves to 74.16, establishing a new state of the art at this scale. The model, curated training data, and code are publicly available at https://github.com/Hasuer/KnowRL.

• The paper introduces a minimal-sufficient guidance framework that decomposes hints into atomic knowledge points to counter reward sparsity in RL-based LLM reasoning.
• The CSS and CBRS strategies enable efficient selection and curation of critical knowledge points, boosting accuracy by up to 15 points on challenging mathematical benchmarks.
• Empirical results validate the approach with state-of-the-art performance, improved convergence, and reduced computational overhead compared to conventional hint-based methods.

KnowRL: Minimal-Sufficient Knowledge Guidance for Reinforcement Learning in LLM Reasoning

Motivation and Limitations of Conventional RLVR and Hint-Based Training

KnowRL addresses persistent challenges in RLVR-based LLM reasoning, particularly reward sparsity encountered on complex problems. RLVR frameworks optimize LLM outputs against rule-based verification, promoting scalability without human annotations. However, the inherent reward sparsity—where most rollout trajectories fail to attain correctness on hard samples—substantially limits the gradient signal for effective optimization, especially as task complexity escalates.

Recent hint-based RL approaches attempt to mitigate this by injecting auxiliary guidance. Prior strategies include solution-prefix hints (fixed/adaptive ratio), abstraction-based hints (conceptual templates or teacher-derived reasoning patterns), and hybrid pipelines combining supervised and RL-generated rollouts. While these methods increase the probability of reward-positive trajectories, their guiding principle has primarily focused on increasing hint quantity, whether via longer prefixes or richer abstractions, neglecting guidance redundancy and introducing inconsistency and training overhead.

Crucially, the paper identifies three systemic issues with conventional hinting:

• Critical-segment effect: Only a minimal segment of knowledge yields sudden performance gains; further guidance provides diminishing benefits.
• Cross-hint inconsistency: Longer or more abstract hints expand the search space or induce branching ambiguity, destabilizing policy updates.
• Guidance-efficiency trade-off: Heavy abstraction templates or teacher hints disrupt training efficiency with added complexity.

These findings collectively indicate that maximizing guidance length or abstraction is suboptimal; the decisive factor is the allocation of minimal, coherent, and sufficient knowledge units.

KnowRL Framework and KP Selection Methodology

KnowRL formulates hint design as a minimal-sufficiency guidance problem. Instead of indiscriminate prefix expansion, it decomposes hints into atomic knowledge points (KPs), selects a compact subset that is sufficient to unlock reward learning, and injects hints only for hard samples. The pipeline comprises:

• KP Curation: For each problem, correct solutions are sampled (e.g., from DeepSeek-R1), and raw KPs are extracted by prompting for indispensable mathematical principles. Leakage verification ensures KPs are general—i.e., not instance-specific or answer-coupled.
• Problem-Wise KP Subset Selection: KnowRL rigorously searches for the most beneficial KP configuration. Offline ablations estimate performance for all, none, or leave-one-out KP removals per problem. However, naive pruning (e.g., Max-Score or strict Leave-One-Out, S-LOO) is confounded by the pruning interaction paradox: removing single KPs may improve accuracy, but removing multiple simultaneously often degrades it due to inter-KP dependencies.

To resolve this, KnowRL introduces Constrained Subset Search (CSS). CSS identifies non-degrading KPs, removes near-optimal candidates, and enumerates configurations only within a tractable subset space. Robustness is further enhanced by the Consensus-Based Robust Selection (CBRS) strategy, which aggregates near-optimal configurations across independent runs and resolves ties via variance minimization. Empirically, CSS achieves the best trade-off between accuracy and KP compactness (average ≈2.5 KPs per problem).

Empirical Results and State-of-the-Art Performance

KnowRL-Nemotron-1.5B, trained with the CSS-guided KP pipeline, establishes new state-of-the-art performance on eight mathematical reasoning benchmarks, including AIME25, HMMT25, CMIMC25, MATH-500, and Olympiad-Bench. Without inference-time hints, KnowRL-Nemotron-1.5B achieves an average accuracy of 70.08 (+9.63 over the Nemotron-1.5B baseline) and +1.50 over JustRL. Incorporation of KP hints at inference further raises performance to 74.16 with CSS-selected KP configurations.

The improvement is especially pronounced on harder competition-style tasks, with gains of +15.11 on AIME25, +12.98 on HMMT25, and +15.49 on CMIMC25. These results demonstrate that interaction-aware KP selection substantially enhances compositional and long-horizon reasoning. Notably, gains persist even in the absence of inference-time hints, indicating robust internalization of reasoning skills rather than mere prompt conditioning.

Difficulty-bucket analysis shows that full-KP injection can regress accuracy in some samples, while CSS selection consistently delivers positive gains across buckets. Random KP selection, matched for cardinality, performs significantly worse, emphasizing the criticality of interaction-aware KP selection.

Analysis of KP Selection Strategies and Optimization Dynamics

Comparative experiments validate that CSS achieves higher training accuracy and more stable optimization than CBRS. CSS produces smoother clip ratio trajectories, indicating less aggressive and more controlled policy refinement. It generalizes better under matched training budgets, confirming its effectiveness in harnessing minimal-sufficient guidance.

Per-query accuracy distributions further demonstrate KnowRL's effect: reward-sparse baselines (OpenMath-Nemotron-1.5B) register a large zero-correct fraction (41.21%), with average accuracy at 22.40%. KnowRL training reduces zero-correct cases to 13.00% and raises all-correct cases to 34.28%, achieving an average of 64.30% without KP hints. KP hints at inference concentrate correct counts further, achieving 77.04% average.

Ablations confirm that entropy annealing accelerates convergence and further improves final scores.

Practical and Theoretical Implications

KnowRL positions minimalist and structured guidance as a scalable principle for RL-based LLM reasoning. By internalizing critical knowledge structures rather than expanding guidance length or abstraction, KnowRL achieves superior performance, improved optimization stability, and reduced computational overhead. The minimal-sufficiency paradigm, and the pruning interaction paradox, provide new theoretical frameworks for hint design and guidance selection.

Practically, KnowRL's compact KP selection significantly reduces hint length per problem, lowers training costs, and enables robust generalization across heterogeneous benchmarks. This approach is extensible to broader reasoning domains beyond mathematics, motivating further research in minimal-sufficient guidance extraction, interaction modeling, and reward sparsity resolution.

Future developments may focus on algorithmic KP extraction for domains with implicit or multimodal knowledge, scalable subset search optimization, and integration with active curriculum pipelines. Extending KnowRL to agentic tasks and multimodal reasoning offers strong potential for robust, efficient RLVR.

Conclusion

KnowRL introduces a minimal-sufficient guidance framework for RLVR, emphasizing atomic KP selection and robust interaction-aware subset curation. Across a suite of mathematical reasoning benchmarks, KnowRL-Nemotron-1.5B achieves new state-of-the-art accuracy, internally improving policy performance and reducing reliance on inference-time scaffolding. Theoretical contributions include formalizing hint design as a minimal sufficiency problem and characterizing the pruning interaction paradox. Practically, KnowRL establishes compact, structured guidance as a scalable principle for reward-sparse RL, motivating future generalization across complex reasoning domains.