---
title: ReasonFlux Model
url: https://www.emergentmind.com/topics/reasonflux-model
type: topic
---

# ReasonFlux Model

ReasonFlux is a family of frameworks for hierarchical reasoning in large language models (LLMs), distinguished by the use of reusable thought templates, structured hierarchical planning via reinforcement learning (RL), and adaptive inference-time feedback mechanisms. The suite encompasses systems for mathematical reasoning, code generation, and trajectory-aware process reward modeling, each advancing the state of the art in computational efficiency and empirical accuracy relative to prior chain-of-thought (CoT), tree-of-thought (ToT), and reward modeling baselines [2502.06772, 2506.03136, 2506.18896].

## 1. Hierarchical Template-Driven Reasoning in LLMs

ReasonFlux-32B approaches mathematical reasoning via a three-stage hierarchical workflow:

1. **Thought Template Library**: Approximately 500 high-level, domain-agnostic templates $\mathcal{D}_{\mathrm{temp}} = \{T_1, \ldots, T_m\}$, each formalizing abstract solution principles (e.g., “trigonometric substitution,” “invariant principle”) through compact, metadata-rich modules.
2. **Navigator LLM ($\pi_\theta$)**: A 32B-parameter transformer leverages hierarchical RL to plan a *template trajectory* for each input problem $x$, selecting and sequencing optimal templates from the library.
3. **Inference LLM ($\pi_{\mathrm{inf}}$)**: Instantiates individual steps by applying structured templates to subproblems, with navigator-driven iterative feedback for correction and refinement.

This Problem $\rightarrow$ Template Trajectory $\rightarrow$ Template Instantiation paradigm constrains the combinatorial space explored during reasoning and has been empirically shown to significantly improve both accuracy and computational efficiency over flat CoT or ToT prompts [2502.06772].

## 2. Structure and Principles of the Thought Template Library

Each template $T_i$ anonymizes domain expertise via the following fields:

| Field            | Description                                              | Example                                          |
|------------------|---------------------------------------------------------|--------------------------------------------------|
| $T_{\mathrm{nam}}$ | Template name                                            | “$\sqrt{R^2 - x^2}$ Type Trigonometric Substitution” |
| $T_{\mathrm{tag}}$ | Keywords for retrieval                                   | {"Trigonometric Substitution", "Irrational Function Optimization"} |
| $T_{\mathrm{des}}$ | NL description & usage context                           | Converts $\sqrt{R^2 - x^2}$ to trigonometric form |
| $T_{\mathrm{sco}}$ | Formal applicability scope                               | Integrals of form $\int\sqrt{R^2 - x^2}\,dx$     |
| $T_a$             | List of application steps                                | Recognize radical $\rightarrow$ subst. $x=R\sin\theta$ $\rightarrow$ backsubstitute |
| $T_{\mathrm{exa}}$ | Worked examples                                         | Provided for typical use cases                   |

Templates are strictly designed for generality and compactness, supporting precise retrieval and cross-domain transfer. Template-augmented inference consistently yields 20–27 percentage point accuracy improvements across algebra, calculus, combinatorics, and geometry benchmarks [2502.06772].

## 3. Hierarchical Reinforcement Learning and Preference Optimization

ReasonFlux employs hierarchical RL to optimize the selection and ordering of templates, formulated as:

- **State** $s_t$: Pair $(x, \{T_1, \dots, T_{t-1}\})$, the original problem with chosen templates to date.
- **Action** $a_t$: Next template $T_{a_t}$ to add to the trajectory.
- **Trajectory** $\tau = (a_1, \dots, a_n)$: Ordered sequence of template indices.
- **Reward** $R(\tau)$: Averaged accuracy of $\pi_{\mathrm{inf}}$ applying $\tau$ to a set $\mathcal{X}_{\mathrm{sim}}$ of similar problems.

Objective is the expected trajectory reward:
\[
J(\theta) = \mathbb{E}_{x, \tau \sim \pi_\theta(\cdot|x)} [ R(\tau) ]
\]
A two-stage finetuning is performed: structure-based supervised fine-tuning (predicting template descriptions/scopes from names/tags) and preference learning using DPO-style loss on trajectory pairs $(\tau^+, \tau^-)$ where $R(\tau^+) > R(\tau^-)$.

**Algorithmic innovations** include hierarchical credit assignment at the trajectory level, curriculum scheduling from short to long template plans, and distillation from frontier solvers into reward underpinnings [2502.06772].

## 4. Adaptive Inference Scaling and Feedback Interplay

At test time, ReasonFlux adaptively scales the number and type of templates based on estimated input complexity:

- Navigator $\pi_\theta$ proposes a trajectory $\mathbb{T}^* = \{T^*_1, ..., T^*_n\}$.
- Templates are retrieved by keyword/tag, and instantiated stepwise via $\pi_{\mathrm{inf}}$.
- An interactive feedback loop refines trajectory decisions: navigator inspects outputs, either accepting, rejecting, or re-planning residual steps.
- The scaling of both number of templates ($n$) and rounds of feedback ($R$) is parameterized as a learned affine function of a complexity score $c(x)$.

This dynamic interplay is critical for handling long-horizon problems and enables the system to allocate greater compute to more challenging inputs. Harder problems empirically require more templates and interaction rounds (see Figure 10 in [2502.06772]).

## 5. Empirical Performance and Ablations

ReasonFlux-32B sets new empirical state-of-the-art results with modest compute (8×A100 GPUs):

| Benchmark         | ReasonFlux-32B | o1-preview | DeepSeek-V3 | Relative Gain         |
|-------------------|:--------------:|:----------:|:-----------:|----------------------|
| MATH              | 91.2%          | 85.5%      | —           | +6.7% over o1-preview |
| AIME 2024         | 56.7%          | 44.6%      | 39.2%       | +27%/+45%             |
| OlympiadBench     | 63.3%          | —          | 55.4%       | +7.9%                 |

Ablation results highlight the necessity of all major components:

- **No templates**: Large 32B flat CoT reaches only 76.6% (MATH), 9.3% (AIME).
- **Templates only**: Non-RL template retrieval improves to ≈83% MATH.
- **HRL only**: RL-planned trajectories without inference feedback yield 88% MATH.
- **Full ReasonFlux**: All stages combined deliver 91.2% on MATH.

Statistical significance is confirmed by 16× repeated runs, $p<0.01$ for main comparisons [2502.06772].

## 6. ReasonFlux Extensions: Co-Evolving Code and Process Reward Models

ReasonFlux methodology generalizes to code generation (ReasonFlux-Coder) and trajectory-aware process reward modeling (ReasonFlux-PRM):

- **ReasonFlux-Coder** applies a co-evolutionary RL framework (CURE) to jointly optimize code and unit-test generation, without ground-truth code, via clipped PPO on both coder and tester branches. It demonstrably boosts code and test accuracy, Best-of-N scaling, and agentic coding pipelines relative to class-leading Qwen baselines [2506.03136].
- **ReasonFlux-PRM** introduces trajectory-aware process reward models enabling both step-level and template-guided trajectory-level supervision. This allows for better selection of high-quality intermediate reasoning traces, more effective RL reward shaping, and enhanced test-time Best-of-N scaling. Empirical gains of 12.1% (SFT), 4.5% (RL), and 6.3% (test-time BoN) are documented across AIME, MATH500, and GPQA-Diamond benchmarks [2506.18896].

## 7. Practical Considerations, Limitations, and Future Directions

- **Compute and Latency**: Full training (structure-SFT + trajectory RL) requires about one week on 8×A100s. Inference latency for a moderate Olympiad problem is 3–5 seconds, using 4–8k tokens and multiple template steps.
- **Failure modes** include template mismatch (navigator picks misaligned template), incomplete library coverage (advanced disciplines unaddressed), and the need to extend template catalogues and retrain for domains outside math/code.
- **Scalability**: Current libraries focus on high-school/olympiad math; applications to QA, code, and science in ReasonFlux-PRM are effective when domain-specific templates and PRMs are constructed [2506.18896].
- **Open research**: Adaptive template library extension, dynamic weighting of reward aggregation ($\alpha, \beta$ in PRM), and self-supervised generator–PRM co-training are open directions.

ReasonFlux frameworks collectively demonstrate that hierarchical decomposition via reusable templates, combined with trajectory-level RL and adaptive scaling, can dramatically optimize reasoning search space and deliver state-of-the-art results with reduced parameter and compute budgets [2502.06772, 2506.03136, 2506.18896].

Source: https://www.emergentmind.com/topics/reasonflux-model