Planner-R1: Efficient Agentic RL

• The paper demonstrates that dense reward shaping yields a 56.9% final pass rate—2.7× improvement over GPT-5’s baseline—with only 180 training queries.
• It introduces a reward architecture combining schema conformity, commonsense, and hard constraints to guide token-level planning and tool use in the TravelPlanner environment.
• Empirical results reveal that smaller 8B models, with dense intermediate rewards, achieve competitive performance and improved compute and memory efficiency compared to larger models.

Searching arXiv for the specified Planner-R1 paper and closely related planning/RL work to ground the article in current literature. Planner-R1 is a reinforcement-learning approach for training language-model agents to perform long-horizon, tool-using planning on the TravelPlanner benchmark, where the model must gather information from several APIs and then produce a fully constraint-satisfying itinerary. Its central claim is that agentic RL becomes substantially more effective when the reward is not only sparse at the end of the episode, but is also shaped with process-level signals that provide intermediate credit for partial progress. On this benchmark, Planner-R1 reports a 56.9\% final-pass rate with only 180 training queries, a $2.7\times$ improvement over GPT-5’s 21.2\% baseline, while also arguing that smaller 8B models can become competitive, $3.5\times$ more compute-efficient, and 1.5\times more memory-efficient than 32B models under dense reward shaping (Zhu et al., 30 Sep 2025).

1. Task setting and formalization

Planner-R1 is defined over TravelPlanner as an episodic MDP. The state is the full conversation history, including the system prompt, user prompt, prior model actions, and all tool outputs. Actions are token-level emissions that may encode either natural-language planning text or structured tool calls. The environment exposes seven tools: search_flights, search_accommodations, search_restaurants, search_attractions, search_ground_transportation, get_cities, and calculator. The model interacts step by step, issuing tool calls, reading results, and eventually outputting a final itinerary inside a strict JSON <answer>...</answer> block (Zhu et al., 30 Sep 2025).

The benchmark’s original success criterion is preserved. A plan is successful only if it is schema-valid and satisfies both explicit user constraints and hidden commonsense constraints. The policy is written as $\pi_\theta(a \mid s)$, and the base optimization target is the terminal-success objective

$\max_\theta \; \mathbb{E}_{\pi_\theta}[r_T].$

The paper’s position is that this objective is too sparse for stable learning on TravelPlanner, because the final reward is 1 only if the whole itinerary is valid and all constraints are satisfied, and 0 otherwise. This motivates the introduction of denser auxiliary rewards derived from the benchmark’s own evaluation components (Zhu et al., 30 Sep 2025).

2. Reward shaping architecture

Planner-R1’s defining mechanism is shaped terminal reward. The auxiliary terms are based on schema conformity, the fraction of satisfied commonsense constraints, the fraction of satisfied hard constraints, indicators for complete commonsense and hard-constraint satisfaction, and the final pass indicator. These terms are combined as

$$r = r_{\text{schema}} \Big( \lambda_{1} r_{\text{cs}^{\text{micro}}} + \lambda_{2} r_{\text{hard}^{\text{micro}}} + \lambda_{3} r_{\text{cs}^{\text{macro}}} + \lambda_{4} r_{\text{hard}^{\text{macro}}} + \lambda_{5} r_{\text{pass}} \Big).$$

The important property claimed in the paper is that these are “proper” reward-shaping terms in the sense that they preserve the same optimal policy as the original final objective, while changing the learning dynamics by providing denser feedback (Zhu et al., 30 Sep 2025).

The implementation uses three reward stages, plus a curriculum that transitions across them during training.

Stage	$\lambda$	Feedback type
Stage 1	$[1,1,1,1,1]$	dense feedback
Stage 2	$[0,0,1,1,1]$	category-level feedback
Stage 3	$[0,0,0,0,1]$	sparse final-pass reward

The curriculum is intended to begin with dense guidance and then move gradually toward the sparse objective. The reported result, however, is that curriculum learning offered no significant benefit over directly using a good shaped reward. The dense auxiliary terms mattered much more than the schedule itself. This suggests that, in this setting, reward density is the decisive variable, whereas stage scheduling is secondary (Zhu et al., 30 Sep 2025).

3. Optimization procedure and training protocol

Planner-R1 uses GRPO, described in the paper as a clipped PPO-style objective without KL regularization. For each query, it samples $G$ trajectories from the old policy and normalizes the trajectory returns into advantages:

$3.5\times$0

The GRPO loss is

$3.5\times$1

with importance ratio

$3.5\times$2

The training recipe is therefore standard modern RL for LLMs, but the paper treats the task-specific reward decomposition as the critical ingredient (Zhu et al., 30 Sep 2025).

The experimental setup is tightly constrained. The authors fine-tune Qwen3 base models at two scales, 8B and 32B. The training budget is only 180 queries, obtained by merging the official 45/180 train-validation split into 180 training and 45 validation queries while preserving the easy/medium/hard ratio. They run five independent seeds. Training is done with verl + GRPO, using sglang for rollouts, on two nodes with 8 H200 GPUs each, for 16 total GPUs. The system uses up to 8 rollouts per query, a 30-turn cap, tool-response limits of 8,192 tokens, and model-response limits of 30,500 tokens. The authors explicitly do not enable “thinking-in-context,” because in their pilots it increased context length without helping task metrics. Evaluation is performed on the official TravelPlanner test split through the benchmark’s leaderboard pipeline, reporting delivery rate, commonsense micro/macro, hard-constraint micro/macro, and final pass rate; transfer is evaluated on NaturalPlan, Multi-IF, and $3.5\times$3-Bench (Zhu et al., 30 Sep 2025).

4. Empirical performance on TravelPlanner

The headline result is Planner-R1-32B at 56.9\% final pass on TravelPlanner, which the paper identifies as the strongest agentic result on the public leaderboard. Relative to GPT-5’s 21.2\% baseline, this is a $3.5\times$4 improvement. The paper also reports GPT-o3 at 11.3\%, while untuned Qwen3 baselines are substantially weaker: Qwen3-8B at 0.0\% and Qwen3-32B at 0.6\% final pass. The reported interpretation is that the base models already display some competence on partial metrics such as delivery, but fail badly at full constraint satisfaction; RL tuning substantially closes that gap (Zhu et al., 30 Sep 2025).

The most salient pattern concerns model-scale dependence on reward sparsity. The 8B model is described as extremely sensitive to sparse rewards: with Stage 1 dense rewards it reaches 39.9\% final pass, but Stage 2 and especially Stage 3 collapse frequently. The paper reports $3.5\times$5 collapses for Stage 2 and $3.5\times$6 collapses for Stage 3 in the 8B setting. By contrast, the 32B model is much more robust under sparse rewards, remaining in the low-to-mid 40s final pass across stage variants: 42.3\% for Stage 1, 44.1\% for Stage 2, 44.3\% for Stage 3, and 47.0\% for curriculum. The authors present this as evidence that larger models can tolerate weaker reward signals better, whereas smaller models require dense shaping to learn effectively (Zhu et al., 30 Sep 2025).

The qualitative analysis follows the same progression. Early in training, both models show repetitive tool loops, poor sequencing, and failures to gather the right information. As training advances, the failure modes become more semantic than syntactic: the models can generate valid JSON and coherent itineraries, but still miss one or two constraints. Eventually, the 32B model in particular learns more deterministic and structured tool sequences, and the frequency of major errors such as hallucinations and repeated tool calls drops. Reward-progression plots show an ordered acquisition pattern in which schema validity is learned first, then commonsense and hard-constraint micro rewards, then macro-category success, and finally the full pass metric. The paper presents this ordering as precisely the behavior induced by the shaping scheme (Zhu et al., 30 Sep 2025).

5. Efficiency, variance, and generalization

A major conclusion of Planner-R1 is that dense reward shaping changes the efficiency frontier. When training is extended to 3,000 steps for 8B and 2,000 steps for 32B, the 8B model reaches 90\% of the 32B model’s peak performance with only $3.5\times$7 FLOPs, compared with $3.5\times$8 FLOPs for 32B, yielding a $3.5\times$9 compute-efficiency advantage. The paper also claims about 1.5\times memory efficiency for the 8B model. In absolute peak accuracy, 32B is only slightly higher, at 56.9\% versus 56.4\% for 8B, and this difference is reported as not statistically significant. Larger models nevertheless retain an advantage in robustness under sparse signals and slightly higher peak accuracy, while also exhibiting higher variance across runs, including some sharp dips in pass rate; once shaped rewards are in place, the 8B model is described as more stable (Zhu et al., 30 Sep 2025).

The paper directly tests overfitting by evaluating fine-tuned models on NaturalPlan, Multi-IF, and $\pi_\theta(a \mid s)$0-Bench, all unseen during training. The main reported result is that RL-tuned models mostly maintain or improve over the base models. For the 8B model, improvements are reported on NaturalPlan’s Meeting and Calendar subtasks, along with slight improvement on later Multi-IF turns. For the 32B model, improvements are reported on all NaturalPlan subtasks and all Multi-IF turns, and $\pi_\theta(a \mid s)$1-Bench pass@1 rises from 28.0 to 33.9 after 2,000 steps. There are some small regressions at longer training for the 8B model on NaturalPlan Trip and Calendar, but the paper states that the overall picture is not one of overfitting to TravelPlanner. The authors interpret this as evidence that JSON-gated structured-output training and tool-conditioned behavior transfer across tasks (Zhu et al., 30 Sep 2025).

This pattern is central to the paper’s broader claim. Sparse terminal rewards alone are characterized as too weak, especially for smaller models, because TravelPlanner requires coordinated tool use, bookkeeping over many constraints, and long-horizon sequencing. Dense process-level signals give the policy traction early, allow it to learn which subgoals matter, and then support refinement toward full task success. A plausible implication is that Planner-R1 is less a new optimization algorithm than a demonstration that reward design can dominate scaling effects in agentic RL on benchmarked planning tasks.

6. Naming distinctions and relation to adjacent work

Planner-R1 is specifically the TravelPlanner agentic RL method described in “Reward Shaping Enables Efficient Agentic RL with Smaller LLMs” (Zhu et al., 30 Sep 2025). It should not be conflated with the similarly named “Plan-R1: Safe and Feasible Trajectory Planning as Language Modeling,” which addresses autonomous driving on nuPlan through next-motion-token prediction followed by GRPO fine-tuning with rule-based rewards such as collision avoidance and speed limits (Tang et al., 23 May 2025). The naming overlap can obscure the fact that the two systems solve different planning problems, operate over different action spaces, and use different environment models.

Planner-R1 also sits near, but is not identical to, later planner-centric work on long-horizon agents. “Planner Matters! An Efficient and Unbalanced Multi-agent Collaboration Framework for Long-horizon Planning” argues that planning is the dominant bottleneck in modular LM/VLM agents and proposes RL that exclusively optimizes the planner using trajectory-level rewards from a VLM-as-judge, while freezing the actor and memory manager (Wu et al., 4 May 2026). By contrast, Planner-R1 trains a single tool-using policy on TravelPlanner and derives shaped rewards from benchmark evaluation components rather than a judge model. A plausible implication is that both lines of work converge on the same systems-level thesis: better planning behavior may depend more on where reward is applied and how it is structured than on uniformly scaling every component.

Within travel planning specifically, “TourPlanner: A Competitive Consensus Framework with Constraint-Gated Reinforcement Learning for Travel Planning” is closer in task domain, but architecturally different. TourPlanner combines PReSO retrieval, Competitive consensus Chain-of-Thought, and sigmoid-based constraint-gated GSPO RL for single-turn itinerary generation on TripTailor (Wang et al., 8 Jan 2026). Planner-R1 instead treats TravelPlanner as an episodic tool-use environment, with token-level actions, API interaction, and shaped terminal reward tied to schema, commonsense, and hard-constraint evaluation. The contrast is useful because it distinguishes two strands of travel-planning research: pipeline planning with multi-path reasoning and constraint-gated refinement, and agentic RL over long-horizon interactive tool use.

Taken together, these distinctions clarify the specific contribution of Planner-R1. Its core result is not merely that RL can improve itinerary generation, but that carefully engineered reward shaping can make smaller LLMs competitive in long-horizon agentic planning without sacrificing generalization, thereby making reward design a primary lever for scalable agentic RL (Zhu et al., 30 Sep 2025).