DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum

Abstract: Executable environments enable LLM agents to learn from the consequences of their actions. For embodied agents, those consequences extend beyond whether the current task succeeds: completing a delivery can consume the time, energy, or money needed for later work. Learning to plan therefore requires environments that preserve these dependencies and turn them into feedback across a complete trajectory. We introduce DeliveryGym, a 3D environment for evaluating and training agents on continuous courier shifts. It couples multimodal tool interaction with persistent world dynamics and computes trajectory rewards from simulator events, making the costs of an agent's decisions available for reinforcement learning (RL). The environment also adapts future training shifts to the policy's observed weaknesses while keeping evaluation fixed. Across six models and 13 city maps, evaluation exposes a gap between reliably executing assigned deliveries and choosing and sequencing work over a shift. On the unseen-city test set, RL improves Qwen3-VL-4B's net income by 54.3%, showing that learning from complete shifts improves performance under these coupled constraints. Adapting the training environment improves evaluation income by 18% over uniform sampling at 100 updates, indicating that which situations an agent practices also matters. DeliveryGym provides an executable setting for studying how agents learn to coordinate deliveries and preserve resources for later orders within an episode.

• The paper presents a novel reinforcement learning (RL) environment, DeliveryGym, designed to evaluate an agent’s ability to plan long-term tasks under real-world constraints, such as managing time, energy, and resources across multiple deliveries.
• The environment, based in a simulated urban setting, introduces a dynamic decision-making process for agents which teaches them to make route-level decisions genuine actionable insights on navigation, routing, and resource allocation. Evaluation results reflect the realism as agents increase profitability in a way that is actionable in their uncovered shifts.
• Delivered and real-time methods improve income performance across multiple models, showing the necessity of considering both immediate task success and long-horizon trade-offs for effective planning in real-world scenarios.

Problem formulation and contribution

DeliveryGym addresses a specific deficiency in embodied-agent benchmarks: successful completion of an isolated task does not measure whether an agent can allocate time, energy, money, and attention across a persistent episode. The paper formulates courier work as a continuing, multi-turn decision process in which order acceptance, routing, fulfillment, resource management, and safety events interact. A delivery can therefore be locally successful while reducing the feasibility or profitability of subsequent deliveries.

The environment is implemented in Unreal Engine 5 using SimWorld Studio and extends the delivery setting of DeliveryBench with an online RL interface, trajectory-level verification, and adaptive task generation (2609.19801). Agents operate across 13 fixed city maps, including Paris and New York, and are evaluated on 130 replayable shifts. Each shift preserves the world state after every action: the clock advances, orders remain active, stamina and scooter battery are depleted, food quality changes, balances are updated, and safety events accumulate.

Figure 1: DeliveryGym represents a courier shift as a persistent 3D decision process with shared temporal, resource, economic, food-quality, and safety constraints.

The central methodological claim is that long-horizon embodied planning should be trained from executable consequences rather than from isolated success labels or externally judged trajectories. DeliveryGym records simulator events and derives income and constraint penalties directly from the resulting trace. This provides a verifiable learning signal without action annotations or an LLM judge.

Environment and agent interface

A DeliveryGym episode consists of a stream of order opportunities. Each order has a pickup location, destination, payment, deadline, and food type. The agent decides which orders to accept, may carry several commitments simultaneously, navigates to restaurants and customers, and manages movement resources. Food categories introduce differentiated temporal and handling costs: ice cream decays quickly, fragile food is damaged by collisions, and hot or time-sensitive meals lose value under delay.

The environment exposes an egocentric RGB view, a route map, structured courier state, recent interaction history, and a persistent notebook. The structured state includes current time, active orders, energy, balance, and the result of the preceding action. Agents interact through tools for order inspection and selection, navigation, pickup and handoff, waiting, rest, charging, purchasing, item use, and note-taking. Invalid actions consume a turn, making action validity itself part of the planning problem.

Two movement interfaces separate high-level route selection from fine-grained visual grounding. In Waypoint mode, the agent selects a visible registered destination. In Any-Point mode, it selects an image pixel that is projected into 3D and checked for walkability. Waypoint observations are cached during most RL experiments, whereas Any-Point evaluation uses online rendering. This distinction is important: the paper evaluates not only strategic order planning but also whether the agent can translate a route decision into a valid visual action.

The shift objective is terminal net income, including base payment, tips, food and lateness reductions, resource purchases, and hospital recovery costs. Safety penalties for red-light violations and obstacle collisions can be added during training. Evaluation income does not include these artificial penalties, allowing the experiments to test whether explicit safety shaping improves economically measured behavior rather than merely optimizing a modified score.

Figure 2: The training loop connects multimodal observations, tool-mediated state transitions, event-based verification, policy updates, and weakness-driven task sampling.

The online training system uses asynchronous simulation workers, batched model inference, rollout collection, and policy optimization. Rendering and simulator overhead are treated as first-class systems constraints. Caching, Unreal Engine optimization, and asynchronous execution increase throughput from 2.1 to 18.4 valid shifts per hour, an 8.8-fold improvement. This systems result matters because terminal-reward RL over 60-turn multimodal episodes is otherwise dominated by environment latency.

Benchmarking existing models

The first experimental result is a substantial difference between execution competence and shift-level planning. Under Waypoint control, GPT-5.6 Sol achieves the highest model income at $18.3 per shift, followed closely by Claude Fable 5 at $18.1. The search-based reference reaches $24.0. Thus, even the strongest evaluated model remains 24% below the reference, despite operating with simplified waypoint navigation.

Waypoint agent	Net income	Delivered	On time	Red-light violations	Obstacle collisions
GPT-5.6 Sol	$18.3	4.1	88.4%	0.5	1.0
Claude Fable 5	$18.1	3.8	84.2%	0.7	1.2
Kimi K3	$16.0	3.2	78.3%	1.0	1.7
Qwen3.6-35B-A3B	$13.4	2.8	74.2%	1.3	2.0
GLM-5.3-Flash	$14.7	3.0	76.3%	1.2	1.9
Qwen3-VL-4B	$7.6	0.7	55.8%	2.3	3.5
Search reference	$24.0	6.2	99.0%	0.0	0.0

The gap is not reducible to failure on individual deliveries. GPT-5.6 Sol achieves 92% single-order success but only 76% of reference shift income. Qwen3.6 completes 87% of assigned deliveries while recovering 56% of reference income. The implication is direct: single-order success is an inadequate proxy for shift-level planning. An agent must evaluate the opportunity cost of accepting an order, sequence overlapping commitments, preserve resources, and account for future arrivals.

Figure 3: Single-order success substantially exceeds the fraction of oracle shift income recovered, exposing the distinction between local execution and global shift planning.

The Any-Point condition exposes an additional bottleneck. Switching from Waypoint to Any-Point reduces income by 50–89% across the six models. GPT-5.6 Sol falls from $18.3 to $9.2, completed deliveries decline from 4.1 to 1.7, and obstacle collisions increase from 1.0 to 3.3 per shift. The reference itself falls from $24.0 to $17.5, so the degradation reflects both increased control difficulty and model grounding failures. Recorded examples include selecting a pixel straight ahead when the route turns right and failing to identify blocked streets or traffic signals.

This comparison establishes that embodied planning has at least two separable failure modes: choosing an economically appropriate sequence of tasks and executing that sequence through visual, spatially grounded actions. DeliveryGym’s fixed harness is therefore consequential. The benchmark does not provide hidden route repair, automatic navigation, or post hoc action correction.

Online RL and reward design

The main RL experiments post-train Qwen3-VL-4B with outcome-based GRPO. Each group contains eight trajectories generated from the same shift specification, and advantages are computed from terminal simulator returns. The policy loss is applied only to generated action tokens; simulator responses and image tokens provide context but are not directly optimized.

Training on complete persistent shifts improves evaluation income substantially. Online RL raises income by 58% over initialization and by 13% over the SFT baseline. On the unseen-city test set, the safety-RL policy improves Qwen3-VL-4B’s income by 54.3% relative to initialization. The result supports the paper’s central hypothesis: retaining state across deliveries exposes the delayed economic consequences needed to learn shift-level behavior.

The paper provides a further diagnostic by comparing persistent multi-order training with isolated single-order training. Persistent training increases evaluation income from $9.1 to $12.0, a 32% improvement. On matched decision states evaluated under a common continuation controller, the persistent policy reduces mean regret from $3.5 to $2.6, a 26% reduction. These results indicate that the gain is not merely a consequence of learning more navigation or fulfillment patterns; it includes improved first decisions under competing-order, resource, and deadline conflicts.

Reward design produces a nontrivial result. Earnings-only RL increases income by 38% over initialization, but the resulting policy still averages 1.8 red-light violations and 2.8 obstacle collisions per shift. Adding explicit safety terms reduces red-light violations by 62% and obstacle collisions by 54%. More importantly, safety-aware training increases evaluation income by 14% over earnings-only RL, even though evaluation income excludes the artificial safety penalties.

Figure 4: Explicit safety terms reduce traffic and collision violations while also improving net income under the original economic evaluation.

The implication is that constraint handling is economically coupled to planning rather than merely an external compliance objective. Avoiding collisions and unsafe crossings prevents hospital recovery, preserves time, and maintains the resources needed for later orders. However, the result should not be interpreted as evidence that scalar reward shaping generally solves constrained RL. The safety weights are hand-specified, and the experiments do not establish that the chosen coefficients are optimal or that safety constraints are satisfied under distribution shift.

Adaptive curriculum

DeliveryGym adapts the training distribution using simulator assertions on fixed probes for three skill families: traffic compliance, obstacle recovery, and multi-order planning. Every 200 training trajectories, probe failures update smoothed weakness estimates. Seventy percent of subsequent training groups target the diagnosed weaknesses, while 30% remain sampled from the base distribution. A feasibility filter rejects generated shifts without a safe, profitable solution.

This design separates adaptive training from adaptive evaluation. Training configurations change, but evaluation shifts, held-out cities, and probe identities remain fixed. The curriculum therefore attempts to improve the learner without changing the test distribution.

At 100 policy updates, adaptive sampling improves evaluation income by 18% over uniform sampling, 10% over a static curriculum, and 14% over random targeting. The comparison with random targeting is particularly informative: the advantage is not explained solely by allocating 70% of trajectories to targeted practice. It depends on using measured policy weaknesses to determine which skill to practice.

Figure 5: Weakness-driven sampling improves evaluation income and raises probe performance for traffic compliance, obstacle recovery, and multi-order planning.

Probe scores improve by 16 percentage points for traffic compliance, 18 points for obstacle recovery, and 26 points for multi-order planning relative to uniform sampling at the relevant phase endpoints. The largest gain occurs on multi-order planning, consistent with the environment’s emphasis on cross-delivery dependencies. The result also gives operational meaning to the income improvement: the curriculum changes identifiable capabilities rather than only producing an undifferentiated score increase.

The curriculum remains relatively narrow. It diagnoses only three predefined failure families, uses fixed probe assertions, and samples within manually specified parameter ranges. Consequently, the experiment demonstrates adaptive practice within a designed skill taxonomy, not unrestricted discovery of all failure modes.

Scaling task and map diversity

The paper studies whether a fixed interaction budget becomes more useful when the training distribution contains more task configurations or cities. In the task-pool sweep, the number of available configurations within one city varies from 200 to 144,000 while the system collects the same 2,400 trajectories. Expanding the pool produces a 47% income gain overall; even increasing the pool from 3,200 to 144,000 yields an additional 11%.

Map scaling produces an even larger effect. Increasing the number of training cities from one to eight raises evaluation income by 61%, with the transition from four to eight maps contributing a further 14%.

Figure 6: Broader task pools and map coverage improve evaluation income despite fixed trajectory and optimizer budgets.

These findings suggest that training diversity has value independent of the number of collected trajectories. Nevertheless, the interpretation requires care. The map sweep measures broader training coverage on fixed evaluation maps, not a separate transfer test to unseen cities. Likewise, the task-pool size denotes available configurations, whereas only approximately 156, 250, 286, and 300 distinct specifications are expected to be encountered for pools of 200, 800, 3,200, and 144,000 under the stated sampling process. The gains therefore reflect the interaction between sampling diversity and repeated policy updates, not exposure to the entire nominal task pool.

Limitations and open questions

DeliveryGym models one occupation in simulated cities, and its operational rules are abstractions of courier work. The economic objective omits social, labor, and service-quality considerations that are not represented by income, ratings, food condition, or safety events. The search reference is also not a certified global optimum: it is a bounded receding-horizon beam search with simulator branch access, a width of 32, and at most 2,048 expanded nodes per decision. Comparisons to this reference consequently quantify performance relative to a strong planning baseline rather than to an exact optimum.

The strongest RL results use Waypoint control, while Any-Point remains substantially more difficult. Transfer from Waypoint-trained policies to Any-Point navigation is not established, and physical deployment is explicitly untested. The environment also relies on cached visual observations for the principal learning comparisons, although the reported audit finds small average cached-versus-live discrepancies: mean absolute income difference is $0.16, with a maximum of $3.00 across the audited shifts.

The evaluation suite contains 13 fixed maps and a finite collection of replayable shifts. Although three cities are held out from training in the principal benchmark, the paper does not establish transfer to real urban demand distributions, unseen simulator dynamics, or physical delivery platforms. The adaptive curriculum is likewise constrained by manually selected probes and skill families. Open questions include whether automatically discovered failure modes can improve on this taxonomy, whether turn-level or event-level credit assignment can outperform terminal GRPO, and whether safety-aware policies retain their gains under richer continuous-control interfaces.

Conclusion

DeliveryGym operationalizes long-horizon embodied planning as persistent courier-shift RL. Its principal contribution is the integration of multimodal tool interaction, stateful 3D dynamics, event-verifiable trajectory rewards, and weakness-driven curriculum adaptation within one environment. The experiments show that high single-order success can coexist with poor shift profitability, that online RL improves cross-order planning, that explicit safety rewards can increase both compliance and income, and that adaptive task allocation improves learning under fixed rollout budgets. The remaining question is whether these gains persist when the interface, city distribution, simulator dynamics, and economic objective are broadened beyond the controlled Waypoint-based setting.