Waymo Sim Agents Benchmark

• The Waymo Sim Agents Benchmark is a public evaluation framework that measures closed-loop multi-agent simulation realism by matching the joint distribution of future trajectories to real data.
• It employs strict protocols using the Waymo Open Motion Dataset, simulating up to 128 agents over an 8-second rollout with metrics covering kinematics, interactions, and map-based behavior.
• Advanced architectures like transformer backbones and CVAE models have driven improvements in simulator stability, diversity, and overall realism scores on the leaderboard.

The Waymo Sim Agents Benchmark, formalized through the Waymo Open Sim Agents Challenge (WOSAC), is a public benchmark for closed-loop, multi-agent behavior simulation in autonomous driving. Its central objective is not trajectory forecasting in isolation, but the evaluation of simulators whose joint distribution of futures matches real driving data while remaining compatible with arbitrary autonomous-vehicle policies through an explicit factorization between the AV policy and the environment model. Since its introduction, the benchmark has served simultaneously as a leaderboard, an evaluation methodology for distributional realism, and a research substrate for studying closed-loop drift, interaction modeling, robustness under perturbation, and planner evaluation (Montali et al., 2023, Schofield et al., 3 Aug 2025).

1. Origin and formal problem statement

WOSAC was introduced as “the first public challenge to tackle this task and propose corresponding metrics,” with the stated goal of stimulating the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving (Montali et al., 2023). The benchmark focuses on predicting mid-level object states—position, heading, speed, and related variables—for the AV and surrounding traffic participants, rather than replaying logs or rendering sensor data.

In the original formulation, the simulator receives a map, traffic-signal states, and $H=11$ steps of past observations for up to $A=128$ dynamic agents, and must output $T=80$ future steps, corresponding to $8$ seconds at $10$ Hz, sampled autoregressively in closed loop (Montali et al., 2023). The factorization requirement is explicit:

$$q^{\mathrm{world}}(o_t \mid o_{<t}^c) = \pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).$$

This requirement is foundational: it is what permits a submitted simulator to be “hooked up with arbitrary AV planners and used to stress-test them” (Montali et al., 2023).

Later reports summarize the protocol in closely related but slightly different terms: an $8$ s rollout at $10$ Hz from a $1$ s “warm-up” history, or $10$ frames of history followed by $A=128$0 joint $A=128$1-second rollouts for all agents (Liang, 20 Dec 2025, Pei et al., 28 Sep 2025). This suggests that the benchmark retained a stable closed-loop core while its public descriptions and leaderboard summaries were streamlined across challenge iterations.

2. Scenario construction, data basis, and submission protocol

The benchmark is based on the Waymo Open Motion Dataset. The original paper states the canonical split as Train: $A=128$2 scenarios, Val: $A=128$3, and Test: $A=128$4, with each scenario a $A=128$5 s slice comprising $A=128$6 s of history and $A=128$7 s of future (Montali et al., 2023). Other reports round this to a standard $A=128$8k/$A=128$9k/$T=80$0k split, reflecting the same large-scale evaluation setting (Pei et al., 28 Sep 2025).

Each scenario contains up to $T=80$1 agents, one of which is the AV. The benchmark does not simulate object insertions or deletions during evaluation; newly appearing logged agents are ignored (Montali et al., 2023). Later descriptions emphasize the same operational structure in challenge terms: generate 32 independently simulated rollouts per scenario, maintain low collision and off-road rates, preserve kinematic and interactive realism, and maximize a single aggregate realism score (Qian et al., 2023).

The submission protocol is correspondingly strict. Entrants generate $T=80$2 rollouts per test scenario, package them as serialized SimAgentsChallengeSubmission protos in shards of roughly $T=80$3 scenarios each, and upload a single .tar.gz bundle to the WOSAC evaluation server, which returns composite and component scores on a hidden test set (Montali et al., 2023). This server-side evaluation is essential to the benchmark’s distribution-matching design, because realism is assessed against held-out logged futures rather than by a manually specified reward.

3. Evaluation methodology and realism metrics

The defining methodological contribution of the benchmark is its use of approximate negative log-likelihood of real futures under the simulator’s rollout distribution. The ideal but infeasible objective is the dataset-level NLL of the full future under $T=80$4:

$T=80$5

Because this full joint likelihood is intractable, WOSAC decomposes realism into one-dimensional summaries, builds histograms from $T=80$6 simulated rollouts with Laplace smoothing, and evaluates the likelihood of the logged measurement under those empirical distributions (Montali et al., 2023).

The original benchmark defines nine component metrics. The kinematic group contains linear speed, linear acceleration, angular speed, and angular acceleration. The interaction group contains distance to nearest object, collision indicator, and time-to-collision. The map-based group contains distance to road edge and road departure (Montali et al., 2023). These are then aggregated into a composite score,

$T=80$7

with collision and road-departure components given double weight to emphasize safety (Montali et al., 2023). Auxiliary open-loop metrics, especially ADE and minADE, are also reported.

Later leaderboard-oriented reports summarize the public outputs somewhat differently. TrafficBots V1.5 describes four metric groups—Realism meta-metric, Kinematic, Interactive, Map-based, and minADE (Zhang et al., 2024). The Multiverse Transformer report writes the realism meta-metric as a weighted sum of kinematic, interactive, and map-based groups with equal weights (Wang et al., 2023). SMART-R1 reports a finer decomposition in which kinematic metrics have total weight $T=80$8, interactive metrics $T=80$9, and map-based metrics $8$0, with map-based evaluation additionally including traffic-light violation (Pei et al., 28 Sep 2025). A plausible implication is that the benchmark’s public reporting interface evolved across challenge versions even as the central distribution-matching logic remained intact.

4. Baselines and leaderboard progression

The original benchmark paper established a broad performance range. On the test split, the Random Agent baseline achieved composite realism of approximately $8$1, Constant Velocity approximately $8$2, and Constant Velocity plus Gaussian noise approximately $8$3 (Montali et al., 2023). Wayformer, when run with identical samples, reached approximately $8$4 at $8$5 Hz and approximately $8$6 at $8$7 Hz; the diverse-sampling variant at $8$8 Hz reached approximately $8$9 (Montali et al., 2023). The Logged Oracle, which simply replays the logged future $10$0 times, obtained $10$1, described as a practical upper bound (Montali et al., 2023).

The 2023 leaderboard established the first wave of high-performing closed-loop simulators. The benchmark paper lists MVTE at composite realism approximately $10$2, MVTA at approximately $10$3, MTR+++ at approximately $10$4, CAD at approximately $10$5, and JointMultipath++ at approximately $10$6 (Montali et al., 2023). These systems already exhibited the design patterns that became characteristic of WOSAC entries: transformer backbones, explicit diversity mechanisms, and post hoc or learned collision mitigation.

Later challenge iterations pushed the public realism scores substantially higher. TrafficBots V1.5, described as a “minimal-novelty” but competitive baseline, reported on the 2024 leaderboard a Realism meta score of $10$7, Kinematic $10$8, Interactive $10$9, Map-based $$q^{\mathrm{world}}(o_t \mid o_{<t}^c) = \pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).$$0, and minADE $$q^{\mathrm{world}}(o_t \mid o_{<t}^c) = \pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).$$1, earning a 3rd-place ranking in WOSAC 2024 (Zhang et al., 2024). In 2025, TrajTok reported a realism score of $$q^{\mathrm{world}}(o_t \mid o_{<t}^c) = \pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).$$2, ranking 2nd, with a map-based score of $$q^{\mathrm{world}}(o_t \mid o_{<t}^c) = \pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).$$3 (Zhang et al., 23 Jun 2025). SMART-R1 reported Realism Meta $$q^{\mathrm{world}}(o_t \mid o_{<t}^c) = \pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).$$4, Kinematic $$q^{\mathrm{world}}(o_t \mid o_{<t}^c) = \pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).$$5, Interactive $$q^{\mathrm{world}}(o_t \mid o_{<t}^c) = \pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).$$6, Map-based $$q^{\mathrm{world}}(o_t \mid o_{<t}^c) = \pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).$$7, and minADE $$q^{\mathrm{world}}(o_t \mid o_{<t}^c) = \pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).$$8, ranking first on the leaderboard at the time of submission (Pei et al., 28 Sep 2025).

The benchmark has also been used to compare model-based and data-driven simulators under a common protocol. A systematic SUMO evaluation on WOSAC reported short-horizon realism meta metric $$q^{\mathrm{world}}(o_t \mid o_{<t}^c) = \pi^{\mathrm{AV}}(o_t^{\mathrm{AV}} \mid o_{<t}^c)\cdot q^{\mathrm{env}}(o_t^{\mathrm{env}} \mid o_{<t}^c).$$9, Kinematic $8$0, Interactive $8$1, Map-based $8$2, and minADE $8$3 m, while requiring fewer than $8$4 tunable parameters (Liang, 20 Dec 2025). This is notable because the same study found strong long-horizon stability for SUMO relative to representative data-driven simulators.

5. Methods and design patterns shaped by the benchmark

One persistent benchmark-level conclusion is that closed-loop modeling matters. The WOSAC overview states directly that closed-loop methods outperformed fully open-loop ones, that diversity through sampling or ensembles is critical, and that collision-minimization strategies can reduce collisions but may over-trim the tails of real data (Montali et al., 2023). These observations shaped the architecture of subsequent top systems.

The Multiverse Transformer exemplifies the early closed-loop recipe. MVTA and MVTE use a transformer-based motion-prediction backbone, a receding-horizon mechanism that predicts $8$5 s ahead but executes only $8$6 s, variable-length history aggregation to mitigate drift, and periodic top-$8$7 sampling to balance diversity against unrealistic rollouts (Wang et al., 2023). The authors explicitly attribute gains in kinematic stability to variable-length history, gains in multi-modal diversity and TTC to receding horizon, and gains in interactive realism to periodic stochastic sampling (Wang et al., 2023).

TrafficBots V1.5 represents a different strand: a CVAE-based multi-agent policy conditioned on per-agent destination and a latent “personality,” combined with HPTR and relative-pose encoding for scalable, agent-centric interaction modeling (Zhang et al., 2024). Its training recipe adds free-nats KL clipping and scheduled teacher-forcing, while inference samples $8$8 full scenarios, ranks them by total collision count, and selects the $8$9 safest simulations (Zhang et al., 2024). The report attributes its relatively low interactive score to the absence of an explicit collision loss and its comparatively large minADE to CVAE-based regression on continuous trajectories (Zhang et al., 2024).

By 2025, tokenized next-token simulators had become dominant. TrajTok introduces a trajectory tokenizer with explicit symmetry and coverage construction, plus spatial-aware label smoothing for cross-entropy training, and reports that replacing SMART’s tokenizer with TrajTok improved the realism score from $10$0 to $10$1 and set a new state of the art on the map-based metric (Zhang et al., 23 Jun 2025). SMART-R1 then extends this line by applying “R1-style” reinforcement fine-tuning to a $10$2M-parameter next-token model, directly optimizing Realism Meta through metric-oriented policy optimization and an SFT–RFT–SFT schedule (Pei et al., 28 Sep 2025).

The benchmark has also been used beyond leaderboard ranking. Waymax provides a JAX/XLA, hardware-accelerated simulator built on WOMD with in-graph reset(state) and step(state, action) primitives, learned and hard-coded behavior models, and benchmarking of imitation-learning and reinforcement-learning agents (Gulino et al., 2023). RL fine-tuning work based on MotionLM showed that closed-loop policy-gradient updates with explicit collision penalties improved WOSAC metrics, for example raising the $10$3M-parameter model’s composite score from $10$4 to $10$5 while improving collision and offroad scores and reducing minADE (Peng et al., 2024). This suggests that the benchmark has become both an evaluation target and a driver of training methodology.

6. Limitations, controversies, and benchmark extensions

The original WOSAC paper already identified several unresolved problems: object birth and death, time-varying agent attributes, rare events and tail collision data, enforcing true closed-loop factorization, richer world models beyond mid-level boxes, and exploration of generative architectures such as flows, VAEs, GANs, and diffusion models (Montali et al., 2023). Many subsequent critiques and extensions can be read as direct responses to that list.

A central criticism is that the standard metametric may not be sufficiently sensitive to failures under partial replay, where the ego is frozen to its ground-truth trajectory while the simulator controls the remaining agents. Schofield et al. define scenario-wise deltas $10$6 and $10$7, together with confusion rates $10$8 and $10$9, to separate simulator sensitivity from policy effects (Schofield et al., 3 Aug 2025). In their evaluation on approximately $1$0 hold-out scenarios, GUMP had the best full-replay score but was the most sensitive to a frozen ego, SMART was almost unaffected, and TrafficBots V1.5 lay in between (Schofield et al., 3 Aug 2025). The same work argues for extending evaluation to a causal-agent domain using labels from CausalAgents (Schofield et al., 3 Aug 2025, Roelofs et al., 2022).

The CausalAgents benchmark provides a complementary robustness perspective. It annotates which agents in WOMD are causal to the self-driving car, finds that on average only $1$1 of agents in a scene are causal, and shows that deleting non-causal agents still causes large relative changes in minADE—between $1$2 and $1$3 across several forecasting architectures (Roelofs et al., 2022). This suggests that strong benchmark performance can coexist with substantial sensitivity to spurious context.

Another limitation is horizon length. The official WOSAC protocol is short-horizon—$1$4 s at $1$5 Hz from a short initialization window—whereas the SUMO comparison extends rollouts to $1$6 s and reports collision rate and off-road rate as stability metrics (Liang, 20 Dec 2025). In that study, SUMO maintained collision rate $1$7 and off-road rate $1$8, while TrafficBots V1.5 exhibited collision $1$9 and offroad $10$0 in long-horizon rollouts (Liang, 20 Dec 2025). A plausible implication is that short-horizon realism and long-horizon stability probe different simulator properties.

Taken together, these extensions show that the Waymo Sim Agents Benchmark is not a fixed scalar leaderboard. It is a layered evaluation ecosystem centered on closed-loop realism, but increasingly supplemented by causal robustness tests, partial-replay sensitivity analysis, long-horizon stability studies, and downstream planner-evaluation protocols (Schofield et al., 3 Aug 2025, Peng et al., 2024).