---
title: 'DeliveryBench: Multi-Domain Delivery Benchmark'
url: https://www.emergentmind.com/topics/deliverybench
type: topic
---

# DeliveryBench: Multi-Domain Delivery Benchmark

DeliveryBench denotes a rigorous, multi-dimensional benchmark ecosystem for evaluating algorithms and embodied agents in the context of real-world delivery logistics. It encompasses a spectrum ranging from classic static and dynamic vehicle routing to last-mile and embodied city-scale delivery, with strong emphasis on realistic constraints, data fidelity, and reproducibility. Major DeliveryBench resources span industrial-scale datasets, interactive high-fidelity simulators, formal task specifications, and standardized evaluation protocols. 

## 1. Formal Problem Domains and Mathematical Structure

DeliveryBench implementations instantiate diverse delivery optimization problems, united by precise mathematical frameworks.

- **City-Scale Embodied Delivery (Food Delivery Benchmark)**  
  The agent’s state at time $t$ is $s_t$:
  - Physical pose and current vehicle (walking, e-scooter, car, or bus)
  - Resource levels (stamina $m_t$, battery $b_t$, cash $w_t$)
  - Active order pool (statuses and deadlines), associated food-attribute tensors ($T$, fragility, odor)
  - Global map structure (road network, POIs)

  The action space $\mathcal{A}$ includes:
  - Navigation actions (high-level MOVE_TO, low-level step/turn)
  - Order manipulation (view, accept, pickup, deliver)
  - Resource and social actions (rest, recharge, buy, rent, help)
 
  The principal objective is profit maximization subject to domain constraints:
  $$
  \pi^\star \in \arg\max_{\pi\in\Pi_\mathcal{C}} \mathbb{E}_\pi[E - C]
  $$
  where $E$ is aggregate earnings (including delivery and rating-based bonuses), $C$ is total costs, and $\Pi_\mathcal{C}$ is the set of constraint-feasible policies [2512.19234].
  
- **Dynamic Pickup and Delivery (DPDP)**
  Formulated over a directed network $G=(F,A)$ among factories, with real-time arrival of orders $O = \{o_1,\ldots,o_N\}$, each $o_i = (F_{p_i}, F_{d_i}, q_i, t_{e_i}, t_{l_i})$. Fleet $V$ of $K$ vehicles (capacities $Q$) manages assignments $\Pi_k$. Key constraints include order fulfillment, vehicle capacity, strict time-windows, driver shifts, LIFO loading, and dock waiting.  
  The objective minimizes $f = \lambda f_1 + f_2$ with $f_1$ total lateness and $f_2$ average vehicle-distance, $\lambda\gg1$ [2202.01256].

- **Planar VRP and Multi-objective Benchmarks**
  In the classic static VRPBench scenario, the domain is a planar graph $G=(V,E)$ with depot $\pi$, customers $C=V\setminus\{\pi\}$, and $k$ vehicles. Routes $R_v$ must minimize $f(R) = \sum_{v=1}^k W(R_v)$ for total route length, or alternative objective mixes including minimizing routes or load variance.  
  VRPBench supports augmentation for time-windows, capacities, and lexicographic or Pareto multi-objective optimization [1610.05402].

- **Last-Mile Delivery (LaDe)**
  Instance data is structured over spatio-temporal events with courier trajectories, package location, time-windows, and city/region segmentation (graph $\mathcal{G}$). Formal definitions emphasize task sequences, time-window constraints, and region-level flow [2306.10675].

## 2. Datasets, Simulators, and Synthetic Environments

DeliveryBench encompasses a range of dataset and simulator resources, each targeting distinct facets of the delivery problem:

| Domain           | Benchmark/Dataset | Scale      | Format/Simulator    | Key Features                                   |
|------------------|-------------------|------------|---------------------|------------------------------------------------|
| Embodied Delivery| DeliveryBench     | 9 cities   | Unreal Engine sim   | Long-horizon city-scale, rich agent actions    |
| DPDP             | Huawei DPDP Comp. | 2k–5k jobs | JSON/event sim      | Dynamic, dock/shift, time-windows, LIFO        |
| Planar VRP       | VRPBench          | 1k–10k pts | 2D graph tools      | Real geography, multi-objective, visualization |
| Last-mile        | LaDe              | 10.7M pkgs | Annotated CSV/JSON  | Courier GPS, five cities, package events        |

- **DeliveryBench** utilizes a high-fidelity, Unreal Engine–powered simulator for 3D city environments, integrating procedural map generation, resource-constrained mobility, food quality physics, and social/teamwork actions [2512.19234].
- **DPDP Benchmark** orchestrates a discrete-event simulator, enforcing rigorous time constraints (600s per optimization tick), multi-pallet handling, and empirical factory networks [2202.01256].
- **VRPBench** provides planar city-graph extraction tools, density-driven sampling, and GUI solution overlays, producing instances up to 10,000 nodes, far exceeding typical TSPLib/VRP test sizes [1610.05402].
- **LaDe** is a real-world courier dataset with >10M packages, 21k couriers, and second-level spatiotemporal logs from multiple cities, standardizing last-mile task modeling [2306.10675].

## 3. Task Definitions, Constraints, and Evaluation Protocols

DeliveryBench distinguishes itself by formalizing a suite of demanding real-world tasks with explicit constraints and evaluation regimes:

- **Embodied Delivery**  
  - Task: Maximize net profit over episodic agent deployment, subject to delivery deadlines, food quality, stamina, battery, economic trade-offs, sequencing, and social constraints.
  - Evaluation: Hourly net profit, on-time rate, planning and resource metrics, violation fractions, food/customer satisfaction, stratified by city size and agent type [2512.19234].

- **Dynamic Pickup and Delivery**  
  - Tasks: Online dispatch, vehicle scheduling with full respect for capacity, time-windows, shift and dock availability, LIFO constraints.
  - Evaluation: Primary score $f = \lambda f_1 + f_2$, plus service rate, average/maximum lateness, computational resource tracking [2202.01256].

- **Static/Planar VRP**  
  - Task: Solve for cost-optimal k-vehicle routes under given customer distributions, with extensions for capacity and time-window.
  - Evaluation: Total/average route length, vehicle count, fairness, solution feasibility, with visualization and potential lower-bound gap reporting [1610.05402].

- **Last-Mile Datasets**  
  - Tasks: Route prediction (sequence learning), ETA prediction, spatio-temporal demand forecasting (region graph).
  - Evaluation: HR@k, Kendall’s $\tau$, LMD, edit distance for routing; MAE, RMSE, ACC@$\Delta$ for ETA; MAE/RMSE for forecasting, stratified by city, scenario, and AOI [2306.10675].

## 4. Baselines, Algorithmic Approaches, and Leaderboards

DeliveryBench supports meaningful comparison by providing baseline implementations, reproducible protocols, and (often) public leaderboards:

- **Embodied Agent Baselines**
  - VLM models: GPT-5, GPT-4o, Claude-3.7, Gemini, open-source Qwen and LLaMA variants.
  - Humans serve as empirical upper baselines: in DeliveryBench, humans earn \$50–63/hr versus best LLM \$31/hr in small/medium cities.
  - Planning ablations and context engineering show substantial gains (e.g., using explicit plan-annotation or “dynamic cheatsheet” raises performance).
  - Multi-agent settings: Small teams optimal, large-team performance sharply drops due to coordination brittleness [2512.19234].

- **DPDP Competition**
  - Winner: Variable Neighborhood Search; runner-ups rely on heuristics augmented for dynamic constraints.
  - RL/DRL approaches trail well-tuned combinatorial and threshold-based heuristics due to real-time and scalability limits.
  - Runtimes and real-time compliance are strictly enforced within the evaluation loop [2202.01256].

- **Planar VRP**
  - Instances vastly exceed traditional benchmarks (10k vs TSPLib/VRPTW max 1k).
  - Algorithms can optimize for distance, vehicle count, or fairness. Lower bounds are used when available for measuring solution quality [1610.05402].

- **LaDe Regional Models**
  - Classical (TimeGreedy, DistanceGreedy, Or-Tools), ML (OSquare, Graph2Route, FDNET), and GNN-based models (ASTGCN, DCRNN).
  - Example: Graph2Route achieves HR@3 ≈ 71.7 %, outperforming heuristic baselines in last-mile route prediction [2306.10675].

## 5. Domain-Specific Constraints and Realism

DeliveryBench environments and datasets are engineered for high-fidelity with respect to operational delivery constraints:

- **Resource Coupling:** All major environments enforce stamina/battery/cash/vehicle/routing interactions; depletions yield forced halts or cost penalties.
- **Food Quality Modeling:** Discrete modeling of thermal drift, fragility (damage by vibration), and odor transfer introduces cross-order dependencies not found in classic VRP/OR benchmarks.
- **Social/Competition Dynamics:** Multi-agent competition for order pools, charging stalls, and explicit cooperative help actions are implemented in embodied simulators.
- **Dock and Shift Realism:** In DPDP, vehicles must respect factory dock capacities, FCFS service, queueing, and work shift intervals. LIFO constraints on pallet loading restrict feasible delivery sequences.
- **Density and Heterogeneity:** Planar VRPBench leverages empirical street-class, type, and land-use-based density factors for realistic customer distributions [1610.05402]. LaDe spans five heterogeneous cities with distinct demand/supply patterns [2306.10675].

## 6. Extensions, Adaptability, and Comparative Strengths

DeliveryBench supports robust extensibility:

- **Scenario Expansion:** Embodied simulation, classic VRP, last-mile, and DPDP frameworks enable research on static/dynamic routing, multi-objective optimization, teamwork, and stochastic/demand-forecasting tasks.
- **Adapting to New Contexts:** VRPBench provides a documented pipeline for generating instances from arbitrary city street networks, supporting adaptation via GIS/OSM imports and land-use-based density modeling [1610.05402].
- **Open Toolchains:** DeliveryBench releases code, data, and configuration schemas for integration and public evaluation.
- **Comparative Strengths:** Relative to TSPLib, Solomon, and CVRPLib, DeliveryBench instances offer order-of-magnitude scaling, empirical geographical heterogeneity, multi-objective support, and context-rich simulation [1610.05402].

## 7. Impact, Current Limitations, and Future Directions

DeliveryBench has created a unified platform for evaluating logistics and embodied reasoning in realistic, constraint-dense settings.

- **Revealed Gaps:** Current agents (notably VLM-based) underperform humans, especially in parallel task management and resource foresight; they are prone to commonsense violations (e.g., letting food items spoil) and have brittle multi-agent coordination [2512.19234].
- **Algorithmic Insights:** Heuristic and hybrid methods remain competitive under dynamic constraints and real-time budgets; pure RL/DRL is not yet dominating at operational scales [2202.01256].
- **Forward Trajectory:** Future efforts will target real-time embodied reasoning, large-scale RL/imitation training, richer social/negotiation mechanisms, and removal of privileged spatial modal priors to test generalization [2512.19234].

DeliveryBench serves as a comprehensive, reproducible, and practical foundation for advancing research in multi-faceted delivery domains, closing the gap between theoretical optimization and deployment in complex, real-world environments [2512.19234][2306.10675][1610.05402][2202.01256].

Source: https://www.emergentmind.com/topics/deliverybench