Waymo Open Sim-Agents Challenge (WOSAC)

• WOSAC is a benchmark for autonomous driving simulation that evaluates closed-loop, agent-centric traffic scenes using the WOMD.
• It uses multi-sample, autoregressive rollouts and composite realism metrics to assess mid-level object trajectories and safety indicators.
• The challenge drives advances in transformer-based motion simulators, next-token predictors, and reinforcement learning for realistic traffic modeling.

The Waymo Open Sim-Agents Challenge (WOSAC) is a public benchmark and evaluation framework for closed-loop simulation with realistic, interactive agents for autonomous driving, built on the Waymo Open Motion Dataset (WOMD). It evaluates sim agents at the mid-level object-trajectory representation rather than sensor simulation, ranks submissions by a realism-centered composite score derived from multi-sample rollouts, and treats minADE as a complementary reference rather than the primary objective. Since its 2023 introduction, WOSAC has become a focal benchmark for transformer-based motion simulators, tokenized next-token predictors, reinforcement-learning fine-tuning, controllable generative agents, and even model-based traffic simulators such as SUMO (Montali et al., 2023, Pei et al., 28 Sep 2025).

1. Original formulation and benchmark scope

In its original formulation, WOSAC defined simulation as closed-loop generative modeling of entire traffic scenes, with agents producing box centroids $(x,y,z)$ and heading at 10 Hz. The challenge was explicitly framed around a world model conditioned on context,

$$q^{world}(o_t \mid o_{<t}^c), \quad o_{<t}^c \doteq [o^{map}, o^{signals}, o_{-H-1}, \dots, o_{t-1}],$$

together with a required factorization between the autonomous vehicle policy and the environment dynamics,

$$q^{world}(o_t \mid o_{<t}^c) = \pi(o_t^{AV} \mid o_{<t}^c)\, q(o_t^{env} \mid o_{<t}^c).$$

A multi-agent factorization of the environment was also allowed,

$$q(o_t^{env} \mid o_{<t}^c) = \prod_{a=1}^{A} \pi_a(o_t^{env,a} \mid o_{<t}^c).$$

This distinguished valid closed-loop, agent-centric simulation from invalid one-shot generation of entire futures (Montali et al., 2023).

WOSAC inherits its scenario substrate from WOMD, a large-scale interactive motion dataset constructed around dense urban interactions among vehicles, pedestrians, and cyclists. WOMD provides 9.1-second windows with 10 history steps, 1 current step, and 80 future steps, derived from longer 20-second clips, together with high-definition maps, traffic signals, 3D boxes, velocities, and interaction-rich scenario mining. This dataset design is central to WOSAC’s emphasis on joint, scene-level realism rather than isolated single-agent prediction (Ettinger et al., 2021).

A common misconception is to treat WOSAC as a forecasting benchmark with a simulator wrapper. The benchmark is instead closed-loop and distributional by construction: realism is evaluated on the trajectories induced by autoregressive rollout, not merely on one-step or open-loop prediction accuracy. This is why the benchmark has repeatedly been used to study covariate shift, long-horizon stability, and planner-facing robustness rather than only predictive fit (Montali et al., 2023).

2. Scenario protocol, rollouts, and submission interface

Published descriptions consistently use 10 Hz closed-loop rollouts and 32 rollout samples per scenario, but they differ slightly in how the observed prefix is narrated. The original challenge paper describes WOMD scenarios as 1.1 seconds of history followed by 8.0 seconds of future, while later benchmark users describe “the standard WOSAC short-horizon setting” as generating 8-second simulations at 10 Hz after an initial 1-second history initialization. In both descriptions, the operational target is an 8-second closed-loop future from real logged context (Montali et al., 2023, Liang, 20 Dec 2025).

Later method papers make the same protocol concrete from the simulator side. TrafficBots V1.5 describes WOSAC 2024 as 10 Hz simulation over 91 timesteps per scenario, with multi-agent outputs generated autoregressively from vectorized agent histories, HD map polylines, and traffic-light tokens. SUMO benchmarking likewise states that “SUMO then generates 32 stochastic 8-second, 10Hz motion sequences for each scenario,” ready for WOSAC evaluation. CNeVA summarizes the task as 10 Hz rollout with 1.1 seconds of history, an 8-second rollout, and up to 128 agents per scene. These descriptions agree on the core operational structure even when the exposition of the history window differs (Zhang et al., 2024, Liang, 20 Dec 2025, Lu et al., 2 Jul 2026).

The original submission interface was scenario-centric and sample-based. Each scenario required 32 rollouts of 8-second simulations, serialized as ScenarioRollouts inside a SimAgentsChallengeSubmission protobuf and uploaded to the evaluation server. In the 2023 iteration, object insertion and deletion were intentionally simplified: newly spawned logged objects after history were discarded during evaluation, and deletion after logged disappearance was omitted. This historical design choice is important because later long-horizon work identifies dynamic cardinality as a central limitation of short-horizon WOSAC-style evaluation (Montali et al., 2023, Xiao et al., 30 Jun 2026).

3. Evaluation methodology and metric evolution

WOSAC’s realism metric is not a direct likelihood under an explicit generative density. Because submissions provide samples rather than tractable probabilities, the benchmark approximates negative log-likelihood by fitting histograms to the 32 submitted rollouts and evaluating logged measurements under the induced categorical distributions. The original paper defines per-component time-series likelihood as

$m = \exp\Big( - \frac{1}{\sum_t \mathbbm{1}\{v_t\}} \sum_{t} \mathbbm{1}\{v_t\} \, \text{NLL}_t \Big),$

with Laplace smoothing of histogram bins by pseudocount 0.1, and aggregates the component metrics into

$$\mathcal{M}^{K} = \frac{1}{NM} \sum_{i=1}^{N} \sum_{j=1}^{M} w_j \, m_{i,j}^{K}.$$

In the 2023 definition, collision and road departure likelihoods were given twice the weight of the other components to emphasize safety (Montali et al., 2023).

The original component suite comprised nine metrics. Kinematic realism covered linear speed, linear acceleration, angular speed, and angular acceleration. Interaction realism covered distance to nearest object, collisions, and time-to-collision. Map-based realism covered distance to road edge and road departures. This taxonomy persisted in later papers and became the canonical way WOSAC realism is discussed (Montali et al., 2023, Liang, 20 Dec 2025).

Later formulations made the bucket structure more explicit. The 2025 metric definition reported by SMART-R1 assigns weights of 0.05 each to linear speed, linear acceleration, angular speed, and angular acceleration; 0.25, 0.10, and 0.10 to collision indication, distance to nearest object, and time to collision; and 0.25, 0.05, and 0.05 to off-road indication, distance to road edge, and traffic light violation. AutoWorld summarizes the same weighting at the bucket level as

$RMM = 0.20 R_{kin} + 0.45 R_{int} + 0.35 R_{map}.$

This later metric stack therefore adds traffic-light violation and makes the 2025 Realism Meta computation effectively explicit at both submetric and bucket levels (Pei et al., 28 Sep 2025, Pourkeshavatz et al., 30 Mar 2026).

Another recurring misconception is that minADE determines WOSAC ranking. Multiple papers explicitly note that minADE is diagnostic and trajectory-prediction-centric, while leaderboard ranking is governed by the realism meta metric or composite realism score. This distinction matters because methods with strong minADE can still underperform on interaction, map adherence, or safety-centered likelihoods (Montali et al., 2023, Pei et al., 28 Sep 2025).

4. Methodological landscape

A first major family consists of motion-forecasting architectures adapted to closed-loop simulation. Early prominent examples include MVTA and MVTE, which used transformer encoders and decoders, receding-horizon prediction, variable-length history aggregation, and multimodal GMM heads to stabilize autoregressive rollout; CAD, which satisfied the 2023 factorization requirement by partitioning agents into ADV, world-tracks-to-predict, and world-others, then applying rejection-based collision-avoidance resampling; and MTR+++, which used a modified Motion Transformer with autoregressive execution and postprocessing for heading smoothing and feasibility (Wang et al., 2023, Chiu et al., 2023, Qian et al., 2023).

A second family centers on discrete next-token prediction. TrafficBots V1.5 combined a CVAE-based multi-agent policy with HPTR relative-pose encoding, then improved leaderboard results through scheduled teacher forcing and scenario filtering. SMART-R1 extended tokenized simulation with an R1-style SFT-RFT-SFT schedule and metric-oriented policy optimization, directly optimizing the WOSAC Realism Meta reward. TrajTok modified SMART’s tokenizer itself, replacing k-disks with a symmetric, rule- and data-driven trajectory vocabulary plus spatial-aware label smoothing, and reached 0.7852 on WOSAC 2025 (Zhang et al., 2024, Pei et al., 28 Sep 2025, Zhang et al., 23 Jun 2025).

A third family uses world models or explicit controllability. AutoWorld learns a LiDAR occupancy world model from unlabeled data, constructs a coarse-to-fine predictive scene context, and applies cascaded quality-aware DPP sampling at both world-model and motion levels. CNeVA introduces per-agent Gaussian behavior latents over safety, map compliance, speed, and acceleration, with rectified-flow generation and classifier-free guidance for controllable simulation. RosettaSim treats scene evolution as a structured autoregressive stream with dynamic token cardinality, adapting a mostly frozen Qwen2.5-0.5B-Instruct backbone to long-horizon traffic modeling and pairing it with Retrieval-based Traffic Evaluation for long-term assessment (Pourkeshavatz et al., 30 Mar 2026, Lu et al., 2 Jul 2026, Xiao et al., 30 Jun 2026).

WOSAC has also motivated hybrid and model-based baselines. The SUMO benchmarking study converts WOMD scenarios into SUMO via Waymo2SUMO, estimates traffic signals from incomplete WOMD signal heads, and evaluates a Krauß-plus-SL2015 rule-based microscopic simulator under WOSAC metrics. SPACeR instead combines decentralized self-play reinforcement learning with a centralized tokenized reference model that supplies likelihood rewards and KL anchoring to the human driving distribution. These systems are methodologically distant from large supervised simulators, yet both are explicitly positioned as WOSAC-relevant baselines or alternatives (Liang, 20 Dec 2025, Chang et al., 20 Oct 2025).

The surrounding software ecosystem is also notable. Waymax provides accelerator-native, JAX/XLA-based multi-agent simulation and evaluation on WOMD with sim-agent support, while GPUDrive enables high-throughput self-play RL on WOMD-scale scenario corpora. Both have become infrastructure papers for WOSAC-adjacent experimentation rather than leaderboard entries themselves (Gulino et al., 2023, Cornelisse et al., 20 Feb 2025).

5. Reported performance and leaderboard trajectory

Because WOSAC has changed across leaderboard versions, challenge years, and evaluation slices, published scores are not all numerically interchangeable. The 2023 technical report for MVTE reports a realism meta-metric of 0.5168, whereas the WOSAC challenge paper later reports MVTE at 0.645 and notes a Dec 2023 update to a V1 leaderboard with improved collision and off-road likelihood calculations. This discrepancy is structural, not merely experimental noise, and direct cross-year comparison without metric-version context is misleading (Wang et al., 2023, Montali et al., 2023).

Benchmark snapshot	Method	Reported realism/composite
2023 technical report	MVTE	0.5168
2023 V1 leaderboard summary	MVTE	0.645
WOSAC 2024 leaderboard report	TrafficBots V1.5	0.6988
WOSAC 2025 submission-time SOTA	SMART-R1	0.7858
WOSAC 2025 technical report	TrajTok	0.7852
WOSAC leaderboard report	AutoWorld	0.7865

The performance trend documented in the literature is nonetheless clear. By 2024, TrafficBots V1.5 reported a realism meta metric of 0.6988, with kinematic 0.4304, interactive 0.7114, map-based 0.8360, and minADE 1.8825, while also describing itself as a 3rd-place WOSAC 2024 baseline. In the same evaluation table, the SUMO benchmarking paper reported SUMO at 0.6532 on the WOSAC 2024 test set, with especially weak kinematics at 0.3294 but reasonable interactive and map-based scores of 0.7153 and 0.7585. The authors emphasized that SUMO used fewer than 100 tunable parameters and that, “Using only about 0.001% of the model size of the best-performing method, SUMO achieves 86.9% of its realism score” (Zhang et al., 2024, Liang, 20 Dec 2025).

By the 2025 metric formulation, performance had tightened into a narrow top cluster. SMART-R1 reported first-place performance at submission time with overall realism 0.7858, kinematic 0.4944, interactive 0.8110, map-based 0.9201, and minADE 1.2885. TrajTok reported 0.7852, with map-based realism 0.9207 and minADE 1.3179. RosettaSim reported 0.7833 on the WOSAC 2025 short-term test, with kinematic 0.4905, interactive 0.8105, map-based 0.9167, and minADE 1.3417. AutoWorld then reported an RMM of 0.7865, with kinematic 0.4931, interactive 0.8143, map-based 0.9185, and minADE 1.3051 (Pei et al., 28 Sep 2025, Zhang et al., 23 Jun 2025, Xiao et al., 30 Jun 2026, Pourkeshavatz et al., 30 Mar 2026).

These numbers support two broad conclusions. First, the strongest WOSAC systems increasingly combine strong short-horizon local realism with explicit mechanisms for diversity, distribution alignment, or post-training adaptation. Second, leaderboard compression near 0.786 suggests that metric gains at the top are now marginal, which in turn raises the importance of robustness, controllability, and long-horizon fidelity as secondary discriminators (Pei et al., 28 Sep 2025, Pourkeshavatz et al., 30 Mar 2026).

6. Robustness, long-horizon evaluation, and unresolved issues

A central issue in the WOSAC literature is that high short-horizon realism does not automatically imply robustness under perturbation or long rollouts. The SUMO benchmarking study extends evaluation to 60-second closed-loop rollouts and reports collision/off-road rates of 0.0047/0.0073 for SUMO, 0.0035/0.0206 for SMART 8M, and 0.2507/0.1526 for TrafficBots v1.5. The same paper presents long-horizon qualitative failures such as left-turn vehicles failing to yield in SMART and major off-road drift in TrafficBots. This suggests that the official 8-second benchmark window can conceal stability differences that only emerge after extended autoregressive rollout (Liang, 20 Dec 2025).

A second issue is partial-control robustness. The world-model analysis paper evaluates no-perturbation rollouts against ego replay and finds that top-ranking models can fail when the ego is forced to follow the original logged trajectory. On 4393 WOMD scenarios, TrafficBots V1.5 shows a simulation-only delta metametric of 0.025 on the standard WOSAC evaluation domain and 0.042 on a causal-agent domain, with corresponding confusion rates that become substantially larger once causal agents are included. The same paper proposes policy-aware delta metrics and control dropout, and shows that control dropout reduces TrafficBots V1.5’s sensitivity without materially improving the standard WOSAC metametric. The implication is that the benchmark’s aggregate score can miss precisely the kinds of simulator brittleness that matter for planner training (Schofield et al., 3 Aug 2025).

Long-horizon evaluation itself has become a research topic. RosettaSim argues that one-to-one agent correspondence breaks down once agents enter and exit the scene, and introduces Retrieval-based Traffic Evaluation (RTE) with context-aware retrieval anchors and an RMM-F1 defined as the harmonic mean of Behavior Realism and Traffic Flow Realism. On 31.1-second rollouts, RosettaSim reports RMM-F1 of 0.7624 and states that RTE correlates more strongly with standard short-term metrics than prior log-based long-horizon approaches, with Pearson correlation $r=0.83$ versus $r=0.74$ (Xiao et al., 30 Jun 2026).

Reliability and planner-facing use are equally unresolved. A self-play RL study on GPUDrive reports 99.81% goal achievement, 0.44% collision, and 0.31% off-road on 10,000 held-out WOMD scenarios under semi-realistic perception and control limits, but excludes pedestrians, cyclists, and traffic lights. Another study fine-tunes MotionLM with REINFORCE and shows improved WOSAC composite, collision, and off-road likelihoods together with better rank-correlation-based planner evaluation. These results indicate that WOSAC is increasingly being treated not just as a realism benchmark, but as a proxy for simulator reliability in planner assessment (Cornelisse et al., 20 Feb 2025, Peng et al., 2024).

Finally, controllability introduces its own complications. CNeVA shows that steering metrics for safety, map, speed, and acceleration can be made monotone, but also argues that such metrics must be interpreted together with guardrails such as stall fraction, mean speed ratio, and off-road rate to avoid reward-hacking confounds. This is a broader warning for WOSAC-style realism benchmarking: improvements along scalar realism axes can be genuine behavioral gains, or they can be artifacts of pathological operating regimes (Lu et al., 2 Jul 2026).

Open problems identified across the literature remain consistent. The original challenge paper highlights object insertion and deletion, time-varying attributes, richer tail scenarios, and stronger enforcement of closed-loop factorization. Later work adds causal-agent robustness, long-horizon evaluation without one-to-one correspondence, and explicit guardrails for controllability. WOSAC has therefore evolved from a leaderboard for short-horizon realism into a broader research program on how traffic simulators should be trained, stress-tested, and interpreted (Montali et al., 2023)