PilotRL: Global Planning-Guided RL Agents

• PilotRL is a global planning-guided reinforcement learning framework that unifies high-level planning and action execution for long-horizon decision-making.
• It employs a three-stage progressive training scheme—executor enhancement, global planner cultivation, and joint optimization—to boost performance over traditional ReAct methods.
• Beyond LLM agents, PilotRL principles extend to adaptive control applications such as drone navigation and flight recovery, enhancing efficiency and safety.

PilotRL most specifically denotes the global planning-guided training framework for LLM agents introduced in "PilotRL: Training LLM Agents via Global Planning-Guided Progressive Reinforcement Learning" (Lu et al., 1 Aug 2025). In that formulation, an agent is trained to act under an explicit, adaptive global plan rather than relying only on ReAct-style single-step reasoning. In a broader PilotRL-style reading, adjacent literature applies reinforcement learning, inverse reinforcement learning, or imitation-derived reference policies to piloting problems such as adaptive drone control, flight-envelope protection, aircraft recovery, teleoperation arbitration, and pilot training. This suggests a broader research motif organized around learned planning, adaptive control, authority allocation, and pilot augmentation, even though not all such systems use PilotRL as a formal method name.

1. Specific meaning: PilotRL as a global planning-guided LLM-agent framework

In its strictest sense, PilotRL is a training framework built on the AdaPlan paradigm, an adaptive global plan-based agent architecture for long-horizon decision-making (Lu et al., 1 Aug 2025). The paper motivates PilotRL through three criticisms of existing agent design. First, ReAct is described as too local: it interleaves “thought” and “action” at each step, which limits effectiveness on tasks requiring long-term strategic planning, multi-step coordination, and handling delayed consequences. Second, planner–executor systems are often disconnected, creating a mismatch between generated plans and executable actions. Third, supervised fine-tuning is said to memorize successful traces and therefore generalize poorly when task contexts change.

AdaPlan addresses these issues by giving the agent two roles inside one model: a global planner and an executor. For a task instruction $G$ and initial context $\mathcal{C}^{(0)}$, the planner generates an initial plan

$$\mathcal{P}^{(0)} = [p_1^{(0)}, p_2^{(0)}, ..., p_{N_0}^{(0)}].$$

At step $t$, the executor selects an action $a^{(t)} \in \mathcal{A}$ conditioned on the accumulated context $\mathcal{C}^{(t-1)}$ and the current global plan $\mathcal{P}^{(t-1)}$, after which the planner updates the remaining future plan. The adaptive update rule is written as

$$p_i^{(t)} = \begin{cases} p_i^{(t-1)}, \text{if } i \le t \ \pi (p_i^{(t)} | G, \mathcal{C}^{(t)}, \mathcal{P}^{(t-1)}, i), \text{if } i > t. \end{cases}$$

The central distinction from ReAct is therefore not merely additional reasoning text, but a persistent high-level roadmap that can be revised after each interaction.

2. Progressive reinforcement learning and planner–executor coordination

PilotRL trains the unified planner–executor model in three stages, ordered to build capabilities progressively rather than optimizing all objectives simultaneously (Lu et al., 1 Aug 2025). Stage 1, “Executor Enhancement,” teaches the model to follow explicit global plans. In this phase, the reward is the sum of normalized format, adherence degree, and end-to-end performance signals: $$\mathcal{R}_{format} = \begin{cases} 1, \text{if the format is correct} \ 0, \text{if the format is incorrect} \end{cases}$$

$$\mathcal{R}_{adherence} = \begin{cases} 2, \text{if completely compliant} \ 1, \text{if partially compliant} \ 0, \text{if noncompliant} \end{cases}$$

$\mathcal{C}^{(0)}$0

Stage 2, “Global Planner Cultivation,” shifts emphasis to plan quality. Candidate plans are generated and selected, and the reward includes format, end-to-end performance, and a plan quality term

$\mathcal{C}^{(0)}$1

where each component is scored from 1 to 5 by DeepSeek-V3 according to correctness, executability, and standardization. Stage 3, “Joint Optimization,” then removes direct plan-quality supervision and optimizes planning and execution together using only format and end-to-end performance, so that final training pressure is placed on planner–executor synergy rather than on isolated plan aesthetics.

The optimization algorithm is GRPO, implemented with the verl framework. For each input $\mathcal{C}^{(0)}$2, the old policy samples a group of $\mathcal{C}^{(0)}$3 completions, and the objective combines a clipped policy ratio term with a KL penalty against a reference policy. The group-normalized advantage is

$\mathcal{C}^{(0)}$4

This design makes improvement relative within each sampled response group rather than relying on a global scalar baseline. Operationally, planner and executor are not separate models in the main method; they are two behaviors learned inside a unified model, with observations fed back to both plan revision and action selection.

3. Benchmarks, baselines, and empirical profile of the LLM-agent PilotRL

The reported training setup uses 5725 samples, 16 rollouts per sample, batch size 256, rollout batch size 64, and 4 epochs total: 1 epoch for Stage 1, 2 epochs for Stage 2, and 1 epoch for Stage 3, with learning rate $\mathcal{C}^{(0)}$5 (Lu et al., 1 Aug 2025). The implementation also masks environmental observation text enclosed in <observation>...</observation> so that these non-policy-generated segments do not contribute gradients.

Training uses ALFWorld, IQA, TextCraft, and Wordle, while evaluation covers those four in-domain benchmarks plus BabyAI and MAZE as out-of-domain tests. Baselines include GPT-4o, GPT-4o-mini, Agent-FLAN-7B, LLaMA-xLAM-2-8B-fc-r, DeepResearcher-7B, Naive Response, ReAct, MPO, SFT, and Vanilla RL. The abstract reports that LLaMA3.1-8B-Instruct + PilotRL surpasses GPT-4o by 3.60% and GPT-4o-mini by 55.78%; the main results section also summarizes average improvements over GPT-4o of 2.35% and over GPT-4o-mini of 53.90%. Reported model-specific averages include 68.53 for Qwen2.5-7B-Instruct + PilotRL, 70.43 for LLaMA3.1-8B-Instruct + PilotRL, and 69.77 for Qwen3-8B + PilotRL.

The ablation results are structurally important. Merging the three-stage rewards rather than using progressive training causes a 3.32% drop. Removing Stage 1 reduces performance by 4.20%, removing Stage 2 by 4.33%, and removing Stage 3 by 2.54%. Reordering the stages from $\mathcal{C}^{(0)}$6 to $\mathcal{C}^{(0)}$7 also degrades performance. AdaPlan consistently outperforms ReAct, with reported average gains of +18.64% on Qwen2.5-7B-Instruct, +13.58% on LLaMA3.1-8B-Instruct, and +7.19% on Qwen3-8B. A unified planner–executor architecture also exceeds isolated planner–executor systems by 6.48%, 7.51%, and 3.96% on the same three backbones. The article’s core empirical claim is therefore not only that planning helps, but that adaptive global planning and unified planner–executor training help more than local reasoning, supervised imitation, or vanilla RL alone.

4. PilotRL-style flight control and autonomous piloting

Outside the LLM-agent setting, several papers instantiate what can plausibly be read as PilotRL-style flight-control systems. "AirPilot: Interpretable PPO-based DRL Auto-Tuned Nonlinear PID Drone Controller for Robust Autonomous Flights" presents a PPO-based DRL drone position controller for autonomous flight in which the policy observes current position error, derivative of the position error, and integral of the position error, and outputs PID gains and/or velocity setpoints (Zhang et al., 2024). The controller is trained in Gazebo and deployed on a COEX Clover drone with Raspberry Pi 4, PX4/MAVROS-style integration, a 14-camera Vicon indoor tracking system, and a 3D A* planner. Reported improvements include reducing the navigation error of the default PX4 PID position controller by 90%, improving effective navigation speed of a fine-tuned PID controller by 21%, and reducing settling time and overshoot by 17% and 16% respectively.

A related but more focused formulation appears in "Reinforcement Learning Based Prediction of PID Controller Gains for Quadrotor UAVs," where DDPG performs online fine-tuning of PID/PD controller gains for the quadrotor inner loop rather than replacing the controller outright (Sönmez et al., 6 Feb 2025). The action is a 5-dimensional continuous vector of normalized gain adjustments, the search rate is set to $\mathcal{C}^{(0)}$8, and deployment targets a Pixhawk 2.1 Cube Black running PX4 Autopilot. The reported attitude-error RMSE improves from $\mathcal{C}^{(0)}$9 to $$\mathcal{P}^{(0)} = [p_1^{(0)}, p_2^{(0)}, ..., p_{N_0}^{(0)}].$$0 in numerical simulation and from $$\mathcal{P}^{(0)} = [p_1^{(0)}, p_2^{(0)}, ..., p_{N_0}^{(0)}].$$1 to $$\mathcal{P}^{(0)} = [p_1^{(0)}, p_2^{(0)}, ..., p_{N_0}^{(0)}].$$2 in outdoor flight, corresponding to about 12.4% and 33.6% reductions.

"Enhanced Flight Envelope Protection: A Novel Reinforcement Learning Approach" moves from gain tuning to pilot-command limiting and augmentation (Catak et al., 2024). Using a nonlinear F-16 model with an INDI control law, a DDPG agent observes

$$\mathcal{P}^{(0)} = [p_1^{(0)}, p_2^{(0)}, ..., p_{N_0}^{(0)}].$$3

and outputs a continuous restorative pitch-rate command

$$\mathcal{P}^{(0)} = [p_1^{(0)}, p_2^{(0)}, ..., p_{N_0}^{(0)}].$$4

The reward combines survival, tracking, angle-of-attack excess, load-factor excess, pitch-rate excess, and terminal penalties. Compared with a classical flight envelope protection algorithm, the RL method is reported to be especially better in pitch-roll coupled maneuvers, where the classical method fails to maintain envelope protection but the RL controller succeeds.

At a more flight-critical end of the spectrum, "Deep Reinforcement Learning based Control Design for Aircraft Recovery from Loss-of-Control Scenario" formulates spin recovery as a continuous-state, continuous-action Markov Decision Process on a nonlinear six-degree-of-freedom F-18/HARV model with actuator saturation and nonlinear aerodynamics (Sayyed et al., 10 Jan 2026). A PPO policy acts on

$$\mathcal{P}^{(0)} = [p_1^{(0)}, p_2^{(0)}, ..., p_{N_0}^{(0)}].$$5

while a two-phase reward first penalizes large angular rates and then enforces trimmed flight, with potential-based reward shaping added to accelerate learning. After 6000 episodes, the learned policy successfully arrests the angular rates and stabilizes the angle of attack, and is compared against a state-of-the-art sliding mode controller. In the paper’s framing, this establishes a pathway toward real-time loss-of-control mitigation.

Model-based RL adds a complementary strand. "Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flight" uses DreamerV3 to learn policies from scratch for autonomous drone racing by mapping raw onboard RGB images directly to collective thrust plus body-rate commands

$$\mathcal{P}^{(0)} = [p_1^{(0)}, p_2^{(0)}, ..., p_{N_0}^{(0)}].$$6

without explicit state estimation, intermediate vision preprocessing, or imitation-learning bootstrapping (Romero et al., 24 Jan 2025). The paper reports that PPO fails to learn meaningful pixel-to-control drone racing under the same conditions, while DreamerV3 converges on Circle, Kidney, and Figure 1 tracks and transfers to a real-world Figure 1 setup through a hardware-in-the-loop deployment. In a broader PilotRL-style sense, this work replaces hand-designed perception abstractions with latent world-model planning for visuomotor flight.

5. Human pilots, co-pilots, and reference policies

PilotRL-style work also includes systems in which the learned policy does not replace the human pilot, but instead teaches, models, assists, or arbitrates human behavior. "Augmenting Flight Training with AI to Efficiently Train Pilots" proposes an AI-based pilot trainer in which a behavioral cloning agent learns straight-and-level flying from 12.5 minutes of instructor demonstrations split into 25 trials of 30 seconds each in X-Plane (Guevarra et al., 2022). The learned policy predicts pitch and roll, is evaluated by average heading error over 10 evaluation trials, and is then used as a teacher model for student guidance. Student errors are detected through thresholded pitch and roll mismatches, and feedback is delivered as a black square with two lines whose position corresponds to $$\mathcal{P}^{(0)} = [p_1^{(0)}, p_2^{(0)}, ..., p_{N_0}^{(0)}].$$7 and whose slope represents the roll angle. The paper explicitly presents this as an imitation-learning prototype rather than a full reinforcement-learning system, while also suggesting future replacement of the BC teacher with reinforcement learning.

"Multiple-Pilot Collaboration for Advanced Remote Intervention using Reinforcement Learning" moves closer to a literal pilot/co-pilot RL formulation (Wang et al., 2021). Two DDPG agents are used: DDPG1 restores the master operator’s intent under communication delay, while DDPG2 performs online arbitration of authority between the fused master command and a co-pilot command through a learned weight matrix $$\mathcal{P}^{(0)} = [p_1^{(0)}, p_2^{(0)}, ..., p_{N_0}^{(0)}].$$8. The framework also includes Kalman-filter-based multi-operator command fusion and interval type-2 Takagi–Sugeno fuzzy force reconstruction for force-sensor-free telepresence. In the reported dual-pilot setup, the final arbitrated command reduces x-direction RMSE to 0.022571 from 0.057569 for the master and 0.058192 for the co-pilot; under triple-pilot collaboration with 0.5 s delay, the final command reaches RMSE 0.0051387 relative to the reference.

A different perspective is given by "Pilot Performance modeling via observer-based inverse reinforcement learning," which treats the pilot as an optimal controller minimizing an unknown quadratic cost functional (Town et al., 2023). Using a regularized history stack observer on a linearized quadcopter model, the method estimates $$\mathcal{P}^{(0)} = [p_1^{(0)}, p_2^{(0)}, ..., p_{N_0}^{(0)}].$$9, $t$0, and $t$1 such that the learned feedback gain converges to the expert or surrogate pilot gain. The paper emphasizes that the inverse reinforcement learning problem is non-unique and therefore aims for an equivalent cost functional rather than exact recovery. In PilotRL-style terms, this is pilot reinforcement learning in inverse form: learning the hidden objective that explains observed pilot actions.

The same co-pilot theme appears, without classical RL, in "Piloting Planetarium Visualizations with LLMs during Live Events in Science Centers" (Brossier et al., 28 Jan 2026). There the “AI-pilot” is a silent, speech-driven conversational agent in OpenSpace that dispatches tool calls for traveling to scene nodes, toggling visibility, changing simulation time, changing simulation speed, or issuing a no-op. The study concludes that AI pilots are not replacements for human counterparts in live planetarium shows, but may become useful co-pilots to reduce workload and allow multitasking. Although not a reinforcement-learning paper, it sharpens the distinction between full piloting autonomy and assistive co-piloting.

6. Boundaries, related paradigms, and recurrent misconceptions

A recurrent misconception is to conflate PilotRL with end-to-end visual steering systems such as NVIDIA’s PilotNet. "The NVIDIA PilotNet Experiments" describes a single deep neural network that takes pixels as input and produces a desired vehicle trajectory or steering output, with training based on supervised imitation learning from human driving data rather than reward shaping, value functions, policy gradients, or exploration-based control (Bojarski et al., 2020). Its core regression form,

$t$2

and mean-squared-error loss make it historically important for end-to-end autonomy, but not a reinforcement-learning PilotRL method.

Another adjacent direction treats the “pilot” as a high-level evaluator rather than as a controller. "SFO: Piloting VLM Feedback for Offline RL" introduces Sub-Trajectory Filtered Optimization and its concrete variant, Sub-Trajectory Filtered Behavior Cloning, in which a vision-LLM judges whether sub-trajectories look successful and those judgments are used to filter and weight offline learning data (Beck, 2 Mar 2025). The training loss is

$t$3

The paper’s central claim is that filtered and weighted behavior cloning can outperform more complex reward-learning approaches, with SFBC reporting a 73% success rate and mean return $t$4 on Pendulum-v1. In a broader PilotRL-style interpretation, the VLM functions as a pilot or guide that identifies promising trajectory segments rather than flying the system directly.

Taken together, these distinctions show that PilotRL is best understood as a family resemblance rather than a single invariant algorithmic template. In the strict sense, it is the AdaPlan-based, global planning-guided progressive reinforcement learning framework for LLM agents (Lu et al., 1 Aug 2025). In the looser sense suggested by adjacent work, PilotRL names a recurring design ambition: to replace fixed heuristics, hand-tuned gains, or purely local reaction with learned planning, adaptive authority allocation, nonlinear policy synthesis, or learned reference behavior. The common thread is not a shared state space or optimizer, but a shift from manually specified piloting logic toward data-driven policies that either act, arbitrate, recover, tune, or instruct under long-horizon and safety-critical constraints.