Agentic-R1: Dual-Strategy Distillation Model

• Agentic-R1 is a 7B-parameter open-source language model that integrates dual strategies—tool-augmented computation and chain-of-thought reasoning—to optimize complex problem solving.
• The DualDistill framework employs heterogeneous teacher trajectories and self-distillation to enhance strategy selection, yielding performance gains of 4–10 points across benchmarks.
• Empirical results demonstrate that Agentic-R1 improves mathematical and symbolic reasoning efficiency, with tool integration reducing multi-step solution latency to under 30 seconds.

Agentic-R1 refers to a 7B-parameter open-source LLM agent trained to integrate and dynamically select between tool-augmented computation and pure chain-of-thought (CoT) reasoning, using a dual-strategy distillation framework termed DualDistill. It enables the model to invoke external tools for algorithmic or arithmetic subproblems while retaining the ability to fall back on exclusively text-based reasoning for abstract or logic-intensive steps. Agentic-R1’s design and training pipeline establish a methodology for robust, efficient mathematical and symbolic reasoning, extending prior tool-augmented agent architectures.

1. Model Architecture and Dual-Strategy Reasoning

Agentic-R1 is based on Deepseek-R1-Distill-7B, a decoder-only transformer, but is augmented with a tool-calling interface and special token tags for structured reasoning. At inference time, the model interleaves:

• > … for chain-of-thought explanations,
• <[code](https://www.emergentmind.com/topics/karpathy-agent-code)>…</code> for Python code execution,
• <executor>…</executor> to embed execution results,
• <answer>…</answer> for final output.

When emitting a <code> block, an external Python interpreter is invoked with outputs fed back as <executor> content. The model supports context lengths up to 32K tokens, aligning well with long-horizon reasoning and iterative tool use. The agent implicitly decides when to switch between CoT and agentic strategies; no discrete gating network is present. This dynamic invocation is learned from composed demonstration trajectories, not prescribed by architectural hard-coding (Du et al., 8 Jul 2025).

2. DualDistill: Heterogeneous Multi-Teacher Distillation

The DualDistill framework is the core training technique. It operates in two stages:

2.1 Teacher Trajectory Composition

Given training data $(x,a)$, trajectories are composed from two distinct teacher agents:

• $\pi_A$ (agentic teacher, e.g., based on Claude-3.5-Sonnet, with code execution capability),
• $\pi_R$ (pure-reasoning teacher, e.g., Deepseek-R1 without code).

For each example, trajectories $y_1$ and $y_2$ are generated in randomized teacher order:

• $$y_1\sim z\cdot\pi_A(\cdot|x) + (1-z)\cdot\pi_R(\cdot|x)$$,
• $$y_2\sim (1-z)\cdot\pi_A(\cdot|x,y_1) + z\cdot\pi_R(\cdot|x,y_1)$$,

where $z\sim$Bernoulli(0.5). Outcomes are scored by a rule-based grader $G(y, a)$ indicating correctness. Final training trajectories are composed conditionally:

• If $G(y_1)=0$, $\pi_A$0, then $\pi_A$1,
• If both correct, $\pi_A$2,
• If only $\pi_A$3 is correct, $\pi_A$4,
• If both incorrect, discard.

Small, explicit transition phrases (e.g. "Let us switch from text to code") are inserted at strategy-switch junctures.

2.2 Teacher-to-Student and Self-Distillation

The student is initially fine-tuned to maximize the log-likelihood of these composed trajectories,

$\pi_A$5

Optionally, a temperature-smoothed KL loss may be used, but hard cross-entropy suffices in this work.

A self-distillation phase further adapts the student to its own weaker tool competencies: for each input $\pi_A$6, $\pi_A$7 samples are drawn from the student, scored, and if mean success $\pi_A$8 exceeds or falls below learned thresholds, examples are replayed with supplementary teacher verification or correction. Further fine-tuning on this self-distilled buffer yields the final student policy.

3. Strategy Selection and Tool Invocation

Agentic-R1’s student implicitly learns to select between agentic (tool) strategy—emitting <code>…</code> blocks, awaiting external execution, then continuing—and pure CoT text blocks. The policy does not use explicit gating losses; transition points are derived solely from composed distillation data. Empirically, tool use dominates on arithmetic-dense tasks (e.g., ≈79% usage on Combinatorics300) but drops to ≈52% for symbolic or logic-heavy tasks like AMC. The system learns to weigh overhead and reliability in invoking external code (Du et al., 8 Jul 2025).

4. Tool Integration: External Python REPL and Execution

The agent integrates a Python REPL supporting standard math, numpy, and fractions operations (capped at 10 calls per query, 3s per call). Tool calls occur exclusively within <code>/</code> delimiters; when a code block is completed, inference halts, the code is executed, and output is appended in an <executor> tag. Segments with runtime errors are masked out of the training loss. This forms a tight, hybrid loop between symbolic execution and neural reasoning, adhering to the agentic paradigm.

5. Empirical Performance and Benchmarks

Agentic-R1 was evaluated across a suite of computation-heavy and standard math reasoning datasets under varying token budgets (S=4096, L=32768):

Model	DeepMath-L S/L	Combinatorics300 S/L	MATH500 S/L	Avg (L)
Qwen2.5-7B (w/o tool)	17.2 / 17.5	21.8 / 21.8	75.1 / 75.2	33.1
Qwen2.5-7B (w/ tool)	34.7 / 34.7	28.9 / 28.9	70.2 / 70.2	39.9
Deepseek-R1-Distill-7B	34.7 / 56.3	34.7 / 44.5	83.1 / 89.2	63.1
Agentic-R1-7B	37.0 / 59.3	36.9 / 49.4	80.0 / 82.4	62.8
Agentic-R1-7B-SD	40.0 / 65.3	38.2 / 52.0	82.5 / 93.3	67.4

Agentic-R1 and its self-distilled variant consistently outperformed both pure tool-based and pure text-based methods on tool-intensive datasets by 4–10 points. Self-distillation further improved large-budget accuracy (+6–9 points). On standard benchmarks (MATH500/AMC/AIME), performance matched state-of-the-art long CoT, with tool calls reducing multi-step solution latency to under 30s—contrasting with multi-minute pure CoT solutions.

Trajectory composition in DualDistill provided a +20–30% gain in large-context settings against non-composed teacher distillation strategies.

6. Analysis, Ablations, and Error Patterns

Ablation studies show that:

• DualDistill’s composition step is crucial; its absence leads to large performance drops.
• Agentic-R1 sometimes over-invokes tools for problems solvable by CoT, causing unnecessary code errors. Self-distillation mitigates this overuse.
• On high-logic tasks requiring proof-style reasoning, pure text-based approaches have a slight edge, indicating limits in current tool-augmented action selection.

The training set remains relatively small (~2.6 K examples), suggesting significant headroom for improvement via scaling up multi-strategy trajectory data.

7. Limitations and Directions for Improvement

Key limitations are the manual design of transition segments (explicit strings separating strategies), the small size of the distilled training set, and restriction to a basic Python API. Extending to richer tools (e.g., theorem provers, web search) is a clear next step. Automating or optimizing strategy hand-offs within the student is currently unresolved. Expanding the trajectory buffer to ~100 K multi-strategy samples is likely to yield further accuracy gains on larger or more complex tasks. More advanced adversarial or automated teacher pipelines could further generalize the dual-strategy distillation approach (Du et al., 8 Jul 2025).