---
title: 'DeliveryGym: Reinforcement Learning Environment'
url: https://www.emergentmind.com/topics/deliverygym
type: topic
---

# DeliveryGym: Reinforcement Learning Environment

DeliveryGym is an executable three-dimensional reinforcement-learning environment for training and evaluating multimodal embodied agents on long-horizon courier work [2609.19801]. It models continuous courier shifts rather than isolated deliveries: time, location, accepted orders, energy, battery, money, inventory, food condition, safety state, and route commitments persist across decisions. The environment combines Unreal Engine 5 simulation, multimodal tool interaction, simulator-verifiable event rewards, online RL, and an adaptive curriculum. Its primary objective is to maximize net income over an entire courier shift while preserving the resources required for later work.

## 1. Conceptual foundations and research scope

DeliveryGym addresses the distinction between **assigned-delivery execution** and **shift-level planning**. Assigned-delivery execution concerns navigating to a restaurant and customer and completing a known order. Shift-level planning additionally requires deciding whether to accept an order, which orders to sequence, how to coordinate overlapping commitments, and how to preserve time, energy, money, and food quality for subsequent opportunities.

The environment is designed around persistent consequences. Accepting a distant order may generate immediate revenue but consume the resources needed for later orders. Conversely, resting, recharging, or rejecting an individually attractive offer may reduce immediate earnings while increasing the value of future work. This structure creates delayed credit-assignment problems that are absent from episodic single-delivery benchmarks.

A shift is a multi-turn episode with a continuing clock. Orders arrive over time, may overlap temporally, and remain commitments after acceptance. The default task contract allows at most eight offers, three commitments, and three parcels. The usual episode limit is 60 turns, and every action consumes one turn, including invalid actions. Movement, waiting, rest, and recovery advance the simulated clock and therefore affect order arrivals and deadlines.

The environment’s central design principle is that rewards should be computed from simulator events over the complete trajectory rather than from an isolated success label. This permits RL policies to learn the consequences of decisions whose costs or benefits appear many turns after the original action.

## 2. Simulation architecture and embodied world

DeliveryGym builds on the DeliveryBench delivery scenario and uses **SimWorld Studio**, implemented with **Unreal Engine 5** [2609.19801]. The simulator supplies rendered street views, walkable areas, roads, intersections, traffic lights, obstacles, blocked streets, collisions, restaurants, customer locations, resource stations, registered navigation waypoints, and movement and physical-state updates.

The system has three conceptual layers:

1. **Unreal Engine 5**: rendering, camera geometry, locomotion, collisions, and low-level interactions.
2. **SimWorld city layer**: streets, map geometry, traffic, obstacles, restaurants, customers, resource stations, and navigation anchors.
3. **DeliveryGym task layer**: order arrivals, courier shifts, persistent resources, delivery rules, event accounting, rewards, and evaluation.

The paper evaluates 13 fixed city maps, including Paris and New York. A compatible map can receive the same courier task when its service locations and navigation data are supplied. A shift seed determines the order stream, locations, deadlines, hazards, and other exogenous conditions, making the shift replayable.

The simulator is a physical-looking urban environment rather than a complete high-fidelity model of real courier work. Its explicitly represented dynamics include movement, collisions, blocked routes, traffic-related events, resource depletion, food deterioration, order deadlines, and recovery. The paper identifies simplified economics and physics, limited city and demand diversity, and the simulation-to-reality gap as limitations.

### Courier and order state

For each order, the environment specifies or derives:

- pickup restaurant;
- customer destination;
- payment;
- deadline;
- food type;
- food-care requirements;
- possible tip and rating consequences.

The courier can walk or use a scooter. Default speeds are approximately 2 m/s for walking and 6 m/s for scooter travel, with an 80-meter movement cap per turn. Walking drains stamina by 0.08 percentage points per meter, while scooter travel drains battery by 0.04 points per meter. Pace changes both speed and consumption.

Rest and recharge restore resources at 7.5 percentage points per minute. Charging costs \$0.05 per percentage point. The courier begins with \$100 working capital. Resource exhaustion can trigger hospital recovery, costing \$5 and 900 seconds.

Food types impose additional constraints. Ice cream decays rapidly and benefits from thermal protection; hot meals, soup, and cake lose quality with delay; fragile food is damaged by rough handling or collisions; and time-sensitive meals lose value on long routes.

## 3. Agent interface, tools, and movement modes

Every evaluated model uses a fixed **DeliveryAgent** harness. The harness formats observations, validates one structured action, executes it in the simulator, and returns structured feedback. It does not select orders, plan routes, repair failed actions, or provide hidden navigation competence.

At each turn, the policy receives:

- an egocentric RGB image;
- a route map;
- visible orders and commitments;
- time and current location;
- energy and battery;
- money balance;
- inventory and carried food;
- the previous action result;
- recent interaction history;
- a persistent notebook.

The policy does not receive future orders, hidden geometry, internal scoring state, or the correct next action.

| Tool group | Representative actions | Function |
|---|---|---|
| Work | `view_orders`, `accept`, `reject`, `check_order` | Inspect offers, deadlines, fees, and commitments |
| Navigation | `navigate`, `look`, `move_point`, `move_waypoint` | Move, inspect views, and obtain route or pose feedback |
| Fulfillment | `collect`, `hand_over` | Collect parcels and complete verified handoffs |
| State and resources | `wait`, `rest`, `buy`, `recharge`, `use_item`, `note` | Advance time, manage resources, and maintain notes |

Malformed or invalid actions return typed errors and consume a turn. Valid actions can change the clock, position, resources, safety state, orders, and balance.

### Waypoint movement

Waypoint movement selects a visible registered destination or waypoint. It abstracts some local locomotion while requiring route selection and responses to traffic, obstacles, and blocked paths. Waypoint training and evaluation can use cached RGB views.

### Any-Point movement

Any-Point movement selects normalized image coordinates \((u,v)\in[0,1]^2\). The selected pixel is deprojected into the 3D scene and checked for reachability. This mode requires finer visual grounding because the agent must interpret the street image and select a physically valid target. Any-Point evaluation uses online rendering.

Both movement interfaces update the same underlying simulator state. Their distinction is therefore primarily one of action granularity and visual-grounding difficulty rather than task semantics.

## 4. Persistent dynamics and reward accounting

A generic transition can be represented as

$$
s_{t+1}=F(s_t,a_t,\xi_t),
$$

where \(s_t\) is the complete courier and world state, \(a_t\) is the selected tool action, \(\xi_t\) represents exogenous events such as order arrivals, and \(F\) is the simulator and task-layer transition function. The observable state is a projection,

$$
o_t=G(s_t),
$$

containing the RGB view, route map, visible order information, structured courier state, recent results, and notebook context.

The persistent constraints include:

- **Time and workload**: all active deadlines share one clock.
- **Energy and mobility**: walking drains stamina and scooter travel drains battery.
- **Money and resources**: rest, charging, supplies, and hospital recovery consume money or time.
- **Food and inventory**: food condition depends on delay, handling, and protection.
- **Safety and route**: traffic violations and collisions affect the shift and can trigger recovery.
- **Location and distance**: the courier’s post-delivery location determines the cost of subsequent work.

The default base payment for order \(o\) is

$$
p_o=10d_ou_o,
$$

where \(d_o\) is the shortest-path distance from pickup to destination in kilometers and \(u_o\sim\mathcal U[0.75,1.35]\) is a multiplicative random factor. Tips depend on the final rating: \$3, \$1.50, or \$0 for ratings of 5, 4, or 0–3 stars, respectively. Ratings of 0–1 also incur a \$2 deduction. Late or damaged food reduces payment, and sufficiently late delivery can reduce the base-pay multiplier to 0.5.

The event-based earnings function is

$$
E(\tau)=\text{base pay}+\text{tips}-\text{late/food reductions}-\text{direct expenses},
$$

where \(\tau\) denotes the complete trajectory. A more explicit shift ledger is

$$
E_H=\sum_{o\in\mathcal D_H}
\left(p_o+\operatorname{tip}_o-\operatorname{reduction}_o\right)-C_H.
$$

Here, \(\mathcal D_H\) is the set of unique orders with verified paid handoffs and \(C_H\) contains resource purchases and service costs, including hospital recovery. Unfinished commitments, remaining inventory, and unused energy have no terminal value. Net income is effectively final balance minus initial capital.

For training, the terminal return can include explicit safety penalties:

$$
R(\tau)=E(\tau)
-\lambda_{\mathrm{red}}N_{\mathrm{red}}(\tau)
-\lambda_{\mathrm{obs}}N_{\mathrm{obs}}(\tau),
$$

where \(N_{\mathrm{red}}\) and \(N_{\mathrm{obs}}\) count red-light violations and obstacle collisions. The evaluated penalty settings include

$$
(\lambda_{\mathrm{red}},\lambda_{\mathrm{obs}})
\in\{(0,0),(0.5,0),(0.5,0.5)\}.
$$

Safety penalties are training terms; evaluation reports simulator net income \(E(\tau)\). This separation permits analysis of whether safety shaping improves economic outcomes without redefining the principal evaluation metric.

## 5. Reinforcement learning and adaptive curriculum

DeliveryGym post-trains **Qwen3-VL-4B-Instruct** with outcome-based Group Relative Policy Optimization (GRPO). Each rollout group contains \(G=8\) trajectories from the same shift specification. For trajectory \(i\), the normalized group advantage is

$$
\widehat A_i=
\frac{R_i-\overline R}{s_R+10^{-6}},
$$

where \(\overline R\) and \(s_R\) are the group mean and standard deviation.

For generated action token \(y_{i,k}\), the likelihood ratio is

$$
\rho_{i,k}=
\frac{\pi_\theta(y_{i,k}\mid h_{i,k})}
{\pi_{\mathrm{old}}(y_{i,k}\mid h_{i,k})}.
$$

The objective uses clipped policy ratios and a KL regularization term, with clipping parameter \(\epsilon=0.2\) and KL coefficient \(\beta=0.01\). Simulator responses and image tokens are context rather than optimized action tokens.

The default RL configuration uses:

- 10 training maps;
- 3 maps excluded from training;
- 60 turns per rollout;
- 8 trajectories per group;
- 300 collected configurations or groups;
- 2,400 trajectories;
- AdamW;
- bfloat16;
- learning rate \(5\times10^{-7}\);
- 5% warmup;
- weight decay \(0.1\);
- effective batch size of 64 trajectories;
- four gradient-accumulation steps;
- maximum sequence length of 32,768 tokens.

### Adaptive curriculum

The adaptive curriculum changes which training situations are sampled according to measured weaknesses while keeping evaluation fixed. Its loop is

$$
\text{policy}
\rightarrow
\text{weakness probes}
\rightarrow
\text{targeted shifts}
\rightarrow
\text{RL updates}
\rightarrow
\text{new policy}.
$$

The principal diagnosed weaknesses are:

1. traffic compliance;
2. obstacle recovery;
3. multi-order planning.

The broader probe suite also includes deadline reasoning, resource management, food care, and conflicting-order rejection. Each target family has 16 fixed training-only probes, with up to 20 turns per probe. A multi-order probe can require completing a specified pair of orders, meeting both deadlines, and earning at least 90% of a bounded planning reference’s income.

After every 200 training trajectories, the system computes the mean assertion score \(p_{k,r}\), defines a deficit

$$
d_{k,r}=1-p_{k,r},
$$

and updates a smoothed weakness weight:

$$
w_{k,r}=0.8w_{k,r-1}+0.2d_{k,r}.
$$

The sampling probability is

$$
q_{k,r}=
\frac{w_{k,r}+0.05}
{\sum_{\ell}(w_{\ell,r}+0.05)}.
$$

Training groups consist of 70% targeted examples sampled according to \(q_{k,r}\) and 30% examples from the base distribution. Targeted generation varies parameters within predefined ranges: two to four controlled intersections for traffic practice, one to three blocked edges with passable detours for obstacle recovery, and two to three overlapping orders for multi-order planning. A feasibility filter rejects shifts without a safe and profitable solution, after which tasks are resampled within the same source and skill family.

The curriculum is compared with uniform sampling, a static curriculum based on the initial weakness profile, and random targeting. Its evaluation suite remains fixed: 130 held-out shifts, comprising 10 shifts on each of 13 maps, including 30 shifts on three cities excluded from training and validation. The evaluation suite is excluded from training, weakness diagnosis, adaptive generation, and checkpoint selection.

## 6. Empirical results, limitations, and significance

Six models are evaluated with the same harness, observations, tools, movement modes, and 60-turn limit: GPT-5.6 Sol, Claude Fable 5, Kimi K3, Qwen3.6-35B-A3B, GLM-5.3-Flash, and Qwen3-VL-4B. The primary metric is mean net income. Additional metrics include delivered orders, on-time delivery, red-light violations, obstacle collisions, single-order success, planning diagnostics, resource use, and exhaustion.

### Fixed-test performance

Under Waypoint control, GPT-5.6 Sol achieves the highest mean income among the evaluated models at \$18.3, followed by Claude Fable 5 at \$18.1. Qwen3-VL-4B obtains \$7.6, while the search oracle obtains \$24.0. GPT-5.6 Sol remains 24% below the oracle.

Under Any-Point control, income decreases by 50–89% across the six models. GPT-5.6 Sol falls to \$9.2, while Qwen3-VL-4B falls to \$0.8. Obstacle collisions increase for GPT-5.6 Sol from 1.0 under Waypoint control to 3.3 under Any-Point control. Even the search oracle loses 27% of its income, indicating that Any-Point introduces substantial fine-grained movement difficulty rather than merely exposing model weakness.

Single-order performance does not predict shift-level performance. GPT-5.6 Sol achieves 92% single-order success but only 76% of oracle income. Qwen3.6 completes 87% of assigned deliveries while obtaining 56% of oracle income. An online order-planning intervention increases mean income from \$14.68 to \$17.20, a 17.1% gain, showing that work selection and sequencing account for a substantial part of the performance gap.

### RL and persistent training

For Qwen3-VL-4B under Waypoint control:

- online RL increases validation income by 58% over initialization;
- income improves by 13% over supervised fine-tuning;
- completed deliveries increase from 0.7 to 3.2 under the same turn cap;
- the selected safety-aware policy improves fixed-test income by 54.3% over initialization.

Fixed-test income is \$7.60 for initialization, \$10.33 for earnings-only RL, and \$11.73 for safety RL. The safety-RL improvement over initialization is \$4.13, with a 95% interval of \([3.45,4.81]\).

Earnings-only RL increases income by 38% but produces 1.8 red-light violations and 2.8 obstacle collisions per shift. Adding safety penalties reduces red-light violations by 62% and obstacle collisions by 54%. Safety-aware RL also increases ordinary evaluation income by 14% over earnings-only RL, suggesting that safer behavior preserves resources and future delivery opportunities.

Persistent multi-order training increases validation income from \$9.1 to \$12.0, a 32% improvement. On 120 matched decision states, it reduces decision regret from \$3.5 to \$2.6, a 26% reduction, under a common continuation controller. Regret is defined as

$$
\operatorname{Regret}(s)
=
\max_{a\in\mathcal A(s)}V_C(s,a)-V_C(s,a_\pi).
$$

This evaluates the current decision while holding downstream execution constant.

### Curriculum and resource outcomes

Adaptive curriculum training improves validation income by 18% over uniform sampling, 10% over a fixed curriculum, and 14% over random targeting. On the fixed test suite, its income is \$11.65 versus \$10.00 for uniform sampling, a 16.5% gain with a 95% interval of \([1.02,2.28]\) dollars. Targeted probe scores improve over uniform sampling by 16 percentage points for traffic compliance, 18 points for obstacle recovery, and 26 points for multi-order planning.

The RL policy replenishes resources more frequently than initialization and substantially reduces exhaustion. Replenish shifts increase from 18.5% to 40.0%; recharge events increase from 0.12 to 0.32; battery-empty shifts decrease from 27.7% to 9.2%; and stamina-empty shifts decrease from 13.8% to 3.1%. These changes indicate that learned replenishment can be economically valuable despite its immediate time and monetary costs.

### Computational infrastructure

The training system contains simulation workers, a rollout service, a batched inference service, and a trainer. Throughput increases from 2.1 valid shifts per hour with live Unreal Engine rendering to 18.4 after cached Waypoint images, engine-side stepping optimization, faster image transfer, isolated simulator workers, asynchronous execution, and batched inference. This is an 8.8-fold speedup.

The final evaluation uses 130 fixed shifts, three repeats per shift, a 60-turn maximum, Waypoint and Any-Point movement, checkpoint selection on a separate validation suite, and three independent training runs. Six of 780 attempts encounter infrastructure failures and are retried once. Behavioral failures such as policy refusals, malformed actions, and zero-income shifts remain in the results rather than being silently removed.

### Limitations and research implications

DeliveryGym’s principal limitations are its focus on a single occupation, simplified economics and physics, incomplete representation of social and service values, limited city and demand diversity, absence of physical deployment, and dependence on a disclosed but consequential harness. Post-training is conducted with Waypoint control, and transfer to Any-Point control is not established. The search oracle has cloned-simulator access and computational privileges unavailable to ordinary agents, so it is a reference for achievable income rather than a deployable baseline or certified global optimum.

The environment’s principal research contribution is the conjunction of embodied tools, persistent state, verifiable event traces, trajectory-level rewards, and adaptive training experience. Its results indicate that profitable courier behavior requires more than reliable local execution: agents must evaluate future opportunity costs, preserve resources, handle safety constraints, manage food condition, recover from blocked routes, and sequence competing work over a continuous shift.

Source: https://www.emergentmind.com/topics/deliverygym