Draft-and-Follow Policy Optimization (DFPO)

• DFPO is a hierarchical reinforcement learning framework that decouples high-level planning ('draft') from fine-grained execution ('follow') in Paper-QA tasks.
• It jointly optimizes draft quality and final answer accuracy to close the knowing–doing gap observed in small LLM agents.
• The method incorporates techniques like negative-sample masking and reward routing to enhance efficiency, stability, and overall performance.

Draft-and-Follow Policy Optimization (DFPO) is a bespoke reinforcement learning algorithm introduced in the context of training small language-model agents for long-horizon scientific-paper question-answering (Paper-QA). DFPO explicitly decomposes the agent’s workflow into high-level planning (“knowing”) and fine-grained tool-use execution (“doing”), with a unified objective that jointly optimizes both the plan quality and the correctness of the final answer. This approach, presented within the PaperGuide framework, is motivated by the observed “knowing–doing” gap in small LLM agents, which can produce reasonable rationales but then perform poorly in following through with actions. DFPO is characterized by its hierarchical structure, theoretical guarantees, and empirical improvements in efficiency and reliability over previous RL baselines (Wang et al., 19 Jan 2026).

1. Motivation and Problem Setting

Recent papers have identified a core deficiency in small LLM-based agents: a disconnect between reasoning and execution, especially evident in Paper-QA tasks where agents may repeatedly search or retrieve information unnecessarily after gathering adequate evidence (Paglieri et al., 2024; Ruoss et al., 2024). Standard flat policies tend to either over-explore, invoking excessive tool calls, or over-exploit, leading to premature and looped actions. DFPO addresses this by imposing a two-stage pipeline:

• Draft phase (“knowing”): The agent produces a high-level plan as a sequence of natural-language instructions.
• Follow phase (“doing”): The agent instantiates these instructions through structured tool calls, following the ReAct paradigm (steps such as Retrieve, Analyze, GenerateAnswer).

This separation is reinforced by a single reinforcement learning objective, rewarding both the coherence of the draft and the accuracy of answers, thus reducing the knowing–doing gap.

2. Formal Framework and Policy Objective

Paper-QA is modeled as a multi-turn Markov Decision Process (MDP) wherein, for a given user question $q$, the agent trajectory consists of an output string $o = (d_1, …, d_{|d|}, y_{|d|+1}, …, y_{|o|})$, with $d$ as draft tokens and $y$ as solution tokens. The policy $\pi_\theta(o|q)$ factorizes token generation sequentially over this string.

State structure: At step $t$, the state $s_t$ comprises $(q, o_{<t})$.

Action structure: Actions are next-token selections, partitioned into draft or solution vocabularies according to phase.

Reward assignment (at trajectory end):

• $r^{draft} = p(d) \in [0, 1]$, a dense draft quality signal, operationalized by normalized trajectory-level entropy:

$$p(d) = -\frac{1}{|d|}\sum_{t=1}^{|d|}\log \pi_\theta(d_t \mid q, d_{<t})$$

• $o = (d_1, …, d_{|d|}, y_{|d|+1}, …, y_{|o|})$0, a binary solution correctness indicator.

DFPO Objective: The joint optimization target is:

$o = (d_1, …, d_{|d|}, y_{|d|+1}, …, y_{|o|})$1

where $o = (d_1, …, d_{|d|}, y_{|d|+1}, …, y_{|o|})$2 and $o = (d_1, …, d_{|d|}, y_{|d|+1}, …, y_{|o|})$3 are group-centered, trajectory-level advantages, with negative-sample masking.

3. Training Algorithm and Stability Techniques

DFPO training proceeds as follows:

1. Initialization: Parameters $o = (d_1, …, d_{|d|}, y_{|d|+1}, …, y_{|o|})$4 are seeded via Draft-&-Tool-Use SFT (DTFT).
2. Rollout collection: For a batch of questions, trajectories $o = (d_1, …, d_{|d|}, y_{|d|+1}, …, y_{|o|})$5 are sampled on-policy, with rewards $o = (d_1, …, d_{|d|}, y_{|d|+1}, …, y_{|o|})$6 and $o = (d_1, …, d_{|d|}, y_{|d|+1}, …, y_{|o|})$7 calculated.
3. Reward routing: The reward router chooses an appropriate evaluation function (exact match, semantic judge, etc.) per question type.
4. Advantage estimation:
   • Compute group means $o = (d_1, …, d_{|d|}, y_{|d|+1}, …, y_{|o|})$8, $o = (d_1, …, d_{|d|}, y_{|d|+1}, …, y_{|o|})$9.
   • For each trajectory: $d$0, $d$1; negative-sample masking zeroes $d$2 if $d$3.
5. Gradient computation: Aggregate token-wise gradients, split by phase, weighted by advantages.
6. Parameter update: Apply optimizer (RMSProp/Adam) with computed gradient.

Stability is further enforced by:

• Fully on-policy rollouts (excluding KL clipping).
• Negative-sample masking to avoid reinforcing drafts from failed trajectories (Theorem 3.2).
• Reward routing to reduce evaluation signal noise.

4. Theoretical Properties

DFPO’s optimization properties are established through two main results:

Proposition 3.1 (Gradient Decomposition)

In the on-policy regime (without KL regularization),

$d$4

Here, the DFPO gradient bifurcates into the standard multi-turn GRPO update plus a draft-phase bias proportional to the centered draft reward.

Theorem 3.2 (Relative Advantage Bounds)

Partitions trajectories into successful ($d$5, $d$6) and failed ($d$7, $d$8) groups:

• For failures, the draft is non-negatively reinforced ($d$9 for $y$0).
• For successes with low-quality drafts, the policy suppresses such plans ($y$1 if $y$2).
• Exceptional drafts leading to success are amplified ($y$3 at highest $y$4).

This suggests that DFPO avoids reinforcing spurious or by-chance solutions, while consolidating reliable planning patterns.

5. Empirical Performance

DFPO is validated on Paper-QA benchmarks:

• AirQA-Real (Cao et al., 2025): 553 long-form AI papers with diverse modalities.
• SciDQA (Singh et al., 2024): 2,900 peer-review questions, tested on a challenging subset.

Metrics:

• Avg: standard task accuracy (0–100).
• I-Avg: efficiency-aware metric penalizing excessive tool use:

$y$5

where $y$6 is mean tool-call turns, $y$7 the allowed maximum.

Main Results (Qwen2.5-3B and 7B backbones):

• DFPO increases I-Avg by +5.5 pp (3B) and +3.2 pp (7B) over SFT-only, matching larger (32B) LLMs in accuracy.
• Versus SFT+M-GRPO, gains are +4.9 pp (I-Avg) and +0.5 pp (Avg) for 3B; +3.5 pp/+1.7 pp for 7B.
• DFPO outperforms proprietary GPT-4o-mini on AirQA-Real (7B agent).

Ablations:

• Negative-sample masking (NSM) and reward router (RR) are both critical: removing NSM destabilizes draft learning; disabling RR reduces both accuracy and efficiency.
• Initializing other RL algorithms (M-GRPO, DAPO) from DTFT recovers much of the draft’s benefit, but DFPO achieves further tool-use efficiency by directly optimizing draft quality.

Entropy dynamics reveal that DFPO uniquely reduces both draft and solution-phase entropy for correct answers, while other baselines (M-GRPO) demonstrate overconfidence in errors or insufficient draft-follow integration.

Knowing–doing probe: Using a UCB bandit diagnostic (Schmied et al., 2025), DFPO closes the knowing–doing gap: when values are correct, PaperGuide-7B selects the optimal arm 50.3% of the time, versus 41.6% for the base model.

6. Advantages, Limitations, and Generalization

Advantages:

• Significantly improved efficiency in tool-use, without accuracy degradation.
• Principled hierarchical architecture, directly targeting the knowing–doing gap via joint optimization of both planning and execution.
• Empirically and theoretically stable training through negative-sample masking, reward router, and on-policy design.

Limitations:

• The scale of the model remains a bottleneck for deep semantic understanding; DFPO does not resolve core comprehension limitations.
• Implementation requires custom reward routing and synthetic expert trajectories, adding to engineering complexity.

Generalization:

The DFPO approach—externalizing high-level plans (drafts) and optimizing a hierarchical RL objective—extends by direct analogy to other structured tasks:

• Robotics: draft subgoal sequence, then execute motor primitives.
• Task-oriented dialogue: outline conversational steps, generate utterances.
• Multi-step code synthesis: draft pseudocode prior to API invocation.

This suggests a broader relevance for hierarchical RL domains where sparse rewards or knowing–doing misalignment challenge monolithic agents. DFPO’s gradient structure and relative-advantage bound apparatus persist in these settings, reinforcing robust planning even under weak task signals.