---
title: Waymo Sim Agents Benchmark
url: https://www.emergentmind.com/topics/waymo-sim-agents-benchmark
type: topic
---

# Waymo Sim Agents Benchmark

The Waymo Sim Agents Benchmark, formalized through the Waymo Open Sim Agents Challenge (WOSAC), is a public benchmark for **closed-loop, multi-agent behavior simulation** in autonomous driving. Its central objective is not trajectory forecasting in isolation, but the evaluation of simulators whose **joint distribution of futures** matches real driving data while remaining compatible with arbitrary autonomous-vehicle policies through an explicit factorization between the AV policy and the environment model. Since its introduction, the benchmark has served simultaneously as a leaderboard, an evaluation methodology for distributional realism, and a research substrate for studying closed-loop drift, interaction modeling, robustness under perturbation, and planner evaluation [2305.12032][2508.01922].

## 1. Origin and formal problem statement

WOSAC was introduced as “the first public challenge to tackle this task and propose corresponding metrics,” with the stated goal of stimulating the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving [2305.12032]. The benchmark focuses on predicting **mid-level object states**—position, heading, speed, and related variables—for the AV and surrounding traffic participants, rather than replaying logs or rendering sensor data.

In the original formulation, the simulator receives a map, traffic-signal states, and $H=11$ steps of past observations for up to $A=128$ dynamic agents, and must output $T=80$ future steps, corresponding to $8$ seconds at $10$ Hz, sampled autoregressively in closed loop [2305.12032]. The factorization requirement is explicit:

$$
q^{\mathrm{world}}(o_t \mid o_{<t}^c)
=
\pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).
$$

This requirement is foundational: it is what permits a submitted simulator to be “hooked up with arbitrary AV planners and used to stress-test them” [2305.12032].

Later reports summarize the protocol in closely related but slightly different terms: an $8$ s rollout at $10$ Hz from a $1$ s “warm-up” history, or $10$ frames of history followed by $32$ joint $8$-second rollouts for all agents [2512.18537][2509.23993]. This suggests that the benchmark retained a stable closed-loop core while its public descriptions and leaderboard summaries were streamlined across challenge iterations.

## 2. Scenario construction, data basis, and submission protocol

The benchmark is based on the Waymo Open Motion Dataset. The original paper states the canonical split as Train: $486{,}995$ scenarios, Val: $44{,}097$, and Test: $44{,}920$, with each scenario a $9.1$ s slice comprising $1.1$ s of history and $8.0$ s of future [2305.12032]. Other reports round this to a standard $487$k/$44$k/$45$k split, reflecting the same large-scale evaluation setting [2509.23993].

Each scenario contains up to $128$ agents, one of which is the AV. The benchmark does not simulate object insertions or deletions during evaluation; newly appearing logged agents are ignored [2305.12032]. Later descriptions emphasize the same operational structure in challenge terms: generate **32 independently simulated rollouts per scenario**, maintain low collision and off-road rates, preserve kinematic and interactive realism, and maximize a single aggregate realism score [2306.15914].

The submission protocol is correspondingly strict. Entrants generate $32$ rollouts per test scenario, package them as serialized `SimAgentsChallengeSubmission` protos in shards of roughly $150$ scenarios each, and upload a single `.tar.gz` bundle to the WOSAC evaluation server, which returns composite and component scores on a hidden test set [2305.12032]. This server-side evaluation is essential to the benchmark’s distribution-matching design, because realism is assessed against held-out logged futures rather than by a manually specified reward.

## 3. Evaluation methodology and realism metrics

The defining methodological contribution of the benchmark is its use of **approximate negative log-likelihood** of real futures under the simulator’s rollout distribution. The ideal but infeasible objective is the dataset-level NLL of the full future under $q^{\mathrm{world}}$:

$$
\mathrm{NLL}^*
=
-\frac{1}{|\mathcal{D}|}
\sum_{i\in\mathcal{D}}
\log q^{\mathrm{world}}\big(o_{\ge 1,i}\mid o_{<1,i}\big).
$$

Because this full joint likelihood is intractable, WOSAC decomposes realism into one-dimensional summaries, builds histograms from $K=32$ simulated rollouts with Laplace smoothing, and evaluates the likelihood of the logged measurement under those empirical distributions [2305.12032].

The original benchmark defines **nine component metrics**. The kinematic group contains linear speed, linear acceleration, angular speed, and angular acceleration. The interaction group contains distance to nearest object, collision indicator, and time-to-collision. The map-based group contains distance to road edge and road departure [2305.12032]. These are then aggregated into a composite score,

$$
\mathcal{M} = \sum_{j=1}^9 w_j\,\overline{\mathrm{Metric}_j},
\qquad
\sum_j w_j = 1,
$$

with collision and road-departure components given double weight to emphasize safety [2305.12032]. Auxiliary open-loop metrics, especially ADE and minADE, are also reported.

Later leaderboard-oriented reports summarize the public outputs somewhat differently. TrafficBots V1.5 describes four metric groups—Realism meta-metric, Kinematic, Interactive, Map-based, and minADE [2406.10898]. The Multiverse Transformer report writes the realism meta-metric as a weighted sum of kinematic, interactive, and map-based groups with equal weights [2306.11868]. SMART-R1 reports a finer decomposition in which kinematic metrics have total weight $0.20$, interactive metrics $0.45$, and map-based metrics $0.35$, with map-based evaluation additionally including traffic-light violation [2509.23993]. A plausible implication is that the benchmark’s public reporting interface evolved across challenge versions even as the central distribution-matching logic remained intact.

## 4. Baselines and leaderboard progression

The original benchmark paper established a broad performance range. On the test split, the Random Agent baseline achieved composite realism of approximately $0.155$, Constant Velocity approximately $0.287$, and Constant Velocity plus Gaussian noise approximately $0.324$ [2305.12032]. Wayformer, when run with identical samples, reached approximately $0.338$ at $10$ Hz and approximately $0.575$ at $2$ Hz; the diverse-sampling variant at $10$ Hz reached approximately $0.421$ [2305.12032]. The Logged Oracle, which simply replays the logged future $32$ times, obtained $0.722$, described as a practical upper bound [2305.12032].

The 2023 leaderboard established the first wave of high-performing closed-loop simulators. The benchmark paper lists MVTE at composite realism approximately $0.645$, MVTA at approximately $0.636$, MTR+++ at approximately $0.608$, CAD at approximately $0.531$, and JointMultipath++ at approximately $0.533$ [2305.12032]. These systems already exhibited the design patterns that became characteristic of WOSAC entries: transformer backbones, explicit diversity mechanisms, and post hoc or learned collision mitigation.

Later challenge iterations pushed the public realism scores substantially higher. TrafficBots V1.5, described as a “minimal-novelty” but competitive baseline, reported on the 2024 leaderboard a Realism meta score of $0.6988$, Kinematic $0.4304$, Interactive $0.7114$, Map-based $0.8360$, and minADE $1.8825$, earning a 3rd-place ranking in WOSAC 2024 [2406.10898]. In 2025, TrajTok reported a realism score of $0.7852$, ranking 2nd, with a map-based score of $0.9207$ [2506.21618]. SMART-R1 reported Realism Meta $0.7858$, Kinematic $0.4944$, Interactive $0.8110$, Map-based $0.9201$, and minADE $1.2885$, ranking first on the leaderboard at the time of submission [2509.23993].

The benchmark has also been used to compare model-based and data-driven simulators under a common protocol. A systematic SUMO evaluation on WOSAC reported short-horizon realism meta metric $0.6532$, Kinematic $0.3294$, Interactive $0.7153$, Map-based $0.7585$, and minADE $5.8305$ m, while requiring fewer than $100$ tunable parameters [2512.18537]. This is notable because the same study found strong long-horizon stability for SUMO relative to representative data-driven simulators.

## 5. Methods and design patterns shaped by the benchmark

One persistent benchmark-level conclusion is that **closed-loop** modeling matters. The WOSAC overview states directly that closed-loop methods outperformed fully open-loop ones, that diversity through sampling or ensembles is critical, and that collision-minimization strategies can reduce collisions but may over-trim the tails of real data [2305.12032]. These observations shaped the architecture of subsequent top systems.

The Multiverse Transformer exemplifies the early closed-loop recipe. MVTA and MVTE use a transformer-based motion-prediction backbone, a receding-horizon mechanism that predicts $1.0$ s ahead but executes only $\Delta=0.1$ s, variable-length history aggregation to mitigate drift, and periodic top-$k$ sampling to balance diversity against unrealistic rollouts [2306.11868]. The authors explicitly attribute gains in kinematic stability to variable-length history, gains in multi-modal diversity and TTC to receding horizon, and gains in interactive realism to periodic stochastic sampling [2306.11868].

TrafficBots V1.5 represents a different strand: a **CVAE-based multi-agent policy** conditioned on per-agent destination and a latent “personality,” combined with HPTR and relative-pose encoding for scalable, agent-centric interaction modeling [2406.10898]. Its training recipe adds free-nats KL clipping and scheduled teacher-forcing, while inference samples $128$ full scenarios, ranks them by total collision count, and selects the $32$ safest simulations [2406.10898]. The report attributes its relatively low interactive score to the absence of an explicit collision loss and its comparatively large minADE to CVAE-based regression on continuous trajectories [2406.10898].

By 2025, tokenized next-token simulators had become dominant. TrajTok introduces a trajectory tokenizer with explicit symmetry and coverage construction, plus spatial-aware label smoothing for cross-entropy training, and reports that replacing SMART’s tokenizer with TrajTok improved the realism score from $0.7814$ to $0.7852$ and set a new state of the art on the map-based metric [2506.21618]. SMART-R1 then extends this line by applying “R1-style” reinforcement fine-tuning to a $7$M-parameter next-token model, directly optimizing Realism Meta through metric-oriented policy optimization and an SFT–RFT–SFT schedule [2509.23993].

The benchmark has also been used beyond leaderboard ranking. Waymax provides a JAX/XLA, hardware-accelerated simulator built on WOMD with in-graph `reset(state)` and `step(state, action)` primitives, learned and hard-coded behavior models, and benchmarking of imitation-learning and reinforcement-learning agents [2310.08710]. RL fine-tuning work based on MotionLM showed that closed-loop policy-gradient updates with explicit collision penalties improved WOSAC metrics, for example raising the $10$M-parameter model’s composite score from $0.549$ to $0.608$ while improving collision and offroad scores and reducing minADE [2409.18343]. This suggests that the benchmark has become both an evaluation target and a driver of training methodology.

## 6. Limitations, controversies, and benchmark extensions

The original WOSAC paper already identified several unresolved problems: object birth and death, time-varying agent attributes, rare events and tail collision data, enforcing true closed-loop factorization, richer world models beyond mid-level boxes, and exploration of generative architectures such as flows, VAEs, GANs, and diffusion models [2305.12032]. Many subsequent critiques and extensions can be read as direct responses to that list.

A central criticism is that the standard metametric may not be sufficiently sensitive to failures under **partial replay**, where the ego is frozen to its ground-truth trajectory while the simulator controls the remaining agents. Schofield et al. define scenario-wise deltas
$\Delta\mathcal{M}_i=\mathcal{M}_i-\hat{\mathcal{M}}_i$ and
$\Delta\mathcal{M}^{\mathrm{sim}}_i=\mathcal{M}^{\mathrm{sim}}_i-\hat{\mathcal{M}}^{\mathrm{sim}}_i$,
together with confusion rates $C_s$ and $C_p$, to separate simulator sensitivity from policy effects [2508.01922]. In their evaluation on approximately $4{,}393$ hold-out scenarios, GUMP had the best full-replay score but was the most sensitive to a frozen ego, SMART was almost unaffected, and TrafficBots V1.5 lay in between [2508.01922]. The same work argues for extending evaluation to a causal-agent domain using labels from CausalAgents [2508.01922][2207.03586].

The CausalAgents benchmark provides a complementary robustness perspective. It annotates which agents in WOMD are causal to the self-driving car, finds that on average only $13\%$ of agents in a scene are causal, and shows that deleting non-causal agents still causes large relative changes in minADE—between $25\%$ and $38\%$ across several forecasting architectures [2207.03586]. This suggests that strong benchmark performance can coexist with substantial sensitivity to spurious context.

Another limitation is horizon length. The official WOSAC protocol is short-horizon—$8$ s at $10$ Hz from a short initialization window—whereas the SUMO comparison extends rollouts to $60$ s and reports collision rate and off-road rate as stability metrics [2512.18537]. In that study, SUMO maintained collision rate $0.0047$ and off-road rate $0.0073$, while TrafficBots V1.5 exhibited collision $0.2507$ and offroad $0.1526$ in long-horizon rollouts [2512.18537]. A plausible implication is that short-horizon realism and long-horizon stability probe different simulator properties.

Taken together, these extensions show that the Waymo Sim Agents Benchmark is not a fixed scalar leaderboard. It is a layered evaluation ecosystem centered on closed-loop realism, but increasingly supplemented by causal robustness tests, partial-replay sensitivity analysis, long-horizon stability studies, and downstream planner-evaluation protocols [2508.01922][2409.18343].

Source: https://www.emergentmind.com/topics/waymo-sim-agents-benchmark