SMART-R1: Reinforcement Fine-Tuning for Traffic Simulation

• SMART-R1 is an R1-style fine-tuning paradigm that augments next-token prediction with a three-stage SFT–RFT–SFT pipeline to reduce covariate shift in multi-agent traffic simulation.
• It integrates Metric-oriented Policy Optimization to directly optimize the Realism Meta score, achieving a top leaderboard ranking with a score of 0.7858 on WOSAC.
• The method balances supervised fine-tuning and reinforcement signals to align agent trajectories with human preferences and safe driving metrics in autonomous driving simulations.

Searching arXiv for SMART-R1 and closely related traffic-simulation work to ground the article. SMART-R1 is an R1-style reinforcement fine-tuning paradigm for next-token prediction models in multi-agent traffic simulation. It is designed to reduce the distributional shift between supervised training and closed-loop simulation by aligning generated agent behavior with human preferences and non-differentiable evaluation metrics. In the formulation reported for the Waymo Open Motion Dataset and the Waymo Open Sim Agents Challenge, SMART-R1 augments the SMART autoregressive simulator with a three-stage SFT–RFT–SFT post-training pipeline and a Metric-oriented Policy Optimization objective, achieving an overall Realism Meta score of 0.7858 and ranking first on the leaderboard at the time of submission (Pei et al., 28 Sep 2025).

1. Problem setting and motivation

SMART-R1 is situated in the problem of scalable and realistic simulation of multi-agent traffic behavior for autonomous driving. The reported motivation is that many data-driven simulators rely on supervised learning—specifically behavior cloning or next-token prediction—which exhibits two persistent limitations in closed-loop use: covariate shift, where small rollout errors accumulate and degrade realism, and misalignment with non-differentiable evaluation metrics, where training targets do not directly optimize simulation objectives such as collision avoidance or off-road behavior (Pei et al., 28 Sep 2025).

The method models simulation as autoregressive prediction over discretized agent behavior tokens $m_k$. Agent trajectories are generated under a joint policy written as

$$P(S_{1:\tau} \mid S_0, \mathcal{C}) = \prod_{t=1}^{\tau}\pi_\theta(S_t \mid S_{<t}, \mathcal{C}),$$

with the per-step policy factorized across agents as

$$\pi_\theta(S_t \mid S_{<t}, \mathcal{C}) = \prod_{i=1}^{N}\pi_\theta(k_t^i \mid S_{<t}^*, \mathcal{C}^*).$$

Here $S_{<t}$ denotes history, $\mathcal{C}$ denotes context such as HD maps, and the architectural stack consists of temporal/agent/map attention layers that produce per-agent logits for motion tokens.

Within this formulation, the central claim is not merely that rollout quality depends on token prediction fidelity, but that policy optimization must target the challenge metrics used in closed-loop evaluation. A plausible implication is that SMART-R1 treats traffic simulation as an alignment problem rather than only as sequence modeling.

2. Training pipeline: SFT–RFT–SFT

SMART-R1 uses a three-stage post-training pipeline:

Stage	Procedure	Stated role
1	Supervised Fine-Tuning (SFT) with closed-loop rollouts using CAT-K	Refines the model to minimize covariate shift
2	Reinforcement Fine-Tuning (RFT) via Metric-oriented Policy Optimization (MPO)	Aligns the policy with target metrics on closed-loop samples
3	SFT again	Restores fidelity to logged data and prevents catastrophic forgetting

The first supervised stage starts from the SMART next-token prediction autoregressive model and applies closed-loop SFT rather than purely open-loop imitation. The reported training objective for the base next-token model is the cross-entropy loss

$$\mathcal{L}_\mathrm{base} = -\sum_{t=1}^{\tau}\sum_{i=1}^{N}\log\big(\pi_\theta(\tilde{k}_t^i \mid S_{<t}^*, \mathcal{C}^*)\big),$$

where $\tilde{k}$ denotes logged token indices. In closed-loop SFT, CAT-K rollout is used to choose tokens $\hat{k}$ closest to ground truth, and the same cross-entropy form is used with $\hat{k}$ replacing $\tilde{k}$ (Pei et al., 28 Sep 2025).

The second stage applies reinforcement fine-tuning to optimize simulation metrics directly. The third stage is a distinctive part of the pipeline: the paper states that a post-RFT SFT phase is necessary to restore the data distribution and mitigate catastrophic forgetting. This is not a generic post-processing step; it is presented as integral to the final performance gains.

The alternation of supervised and reinforcement objectives is the defining structural feature of SMART-R1. In the paper’s framing, the first SFT provides stable closed-loop behavior, RFT improves metric alignment, and the second SFT recovers distributional fidelity.

3. Metric-oriented Policy Optimization

The core reinforcement component is Metric-oriented Policy Optimization (MPO). SMART-R1 formulates simulation as a Markov Decision Process, where each action is a motion token and the reward is defined by the Realism Meta metric used in WOSAC. The stated objective is to maximize the expected realism score under model rollouts (Pei et al., 28 Sep 2025).

For each rollout, a reward $$P(S_{1:\tau} \mid S_0, \mathcal{C}) = \prod_{t=1}^{\tau}\pi_\theta(S_t \mid S_{<t}, \mathcal{C}),$$0 is computed from the evaluation metric, and the advantage is defined as

$$P(S_{1:\tau} \mid S_0, \mathcal{C}) = \prod_{t=1}^{\tau}\pi_\theta(S_t \mid S_{<t}, \mathcal{C}),$$1

where $$P(S_{1:\tau} \mid S_0, \mathcal{C}) = \prod_{t=1}^{\tau}\pi_\theta(S_t \mid S_{<t}, \mathcal{C}),$$2 is an empirical threshold used to normalize and stabilize the update. To regularize the policy against excessive drift from the reference model, SMART-R1 uses a per-token KL term: $$P(S_{1:\tau} \mid S_0, \mathcal{C}) = \prod_{t=1}^{\tau}\pi_\theta(S_t \mid S_{<t}, \mathcal{C}),$$3

The resulting MPO loss is

$$P(S_{1:\tau} \mid S_0, \mathcal{C}) = \prod_{t=1}^{\tau}\pi_\theta(S_t \mid S_{<t}, \mathcal{C}),$$4

where $$P(S_{1:\tau} \mid S_0, \mathcal{C}) = \prod_{t=1}^{\tau}\pi_\theta(S_t \mid S_{<t}, \mathcal{C}),$$5 is a no-gradient copy of the policy used for importance reweighting and $$P(S_{1:\tau} \mid S_0, \mathcal{C}) = \prod_{t=1}^{\tau}\pi_\theta(S_t \mid S_{<t}, \mathcal{C}),$$6 is the KL coefficient. The optimization procedure is summarized as: sample scenes, roll out trajectories, compute reward $$P(S_{1:\tau} \mid S_0, \mathcal{C}) = \prod_{t=1}^{\tau}\pi_\theta(S_t \mid S_{<t}, \mathcal{C}),$$7, compute $$P(S_{1:\tau} \mid S_0, \mathcal{C}) = \prod_{t=1}^{\tau}\pi_\theta(S_t \mid S_{<t}, \mathcal{C}),$$8 and per-token KL, backpropagate through $$P(S_{1:\tau} \mid S_0, \mathcal{C}) = \prod_{t=1}^{\tau}\pi_\theta(S_t \mid S_{<t}, \mathcal{C}),$$9, and iterate for multiple RFT epochs.

The paper explicitly contrasts MPO with several reinforcement-learning baselines. It reports that PPO (with value model) is unstable or refuses to converge, while DPO and GRPO have high sampling bias in this setting. MPO is presented as more suitable because it exploits predictable, deterministic rewards from the simulation metric rather than relying on auxiliary value estimation or grouped preference comparisons.

4. Datasets, benchmarks, and evaluation protocol

SMART-R1 is evaluated on the Waymo Open Motion Dataset (WOMD) and the Waymo Open Sim Agents Challenge (WOSAC). The reported WOMD scale is 487k train, 44k val, and 45k test scenarios, with up to 128 agents per scene (Pei et al., 28 Sep 2025).

WOSAC is described as the official distribution-matching simulation benchmark. Its principal evaluation target is the Realism Meta score, a composite metric built from kinematic, interactive, and map-based sub-metrics. The paper also reports minADE as an open-loop motion-prediction metric.

This evaluation design matters because SMART-R1 is not optimized for a proxy loss unrelated to challenge scoring. Instead, the benchmark metric itself supplies the reward signal used during RFT. That design choice links training and evaluation more tightly than in purely supervised traffic simulators.

A plausible implication is that SMART-R1 is intended less as a generic imitation learner and more as a simulator whose policy is explicitly tuned to the realism criteria used in downstream autonomous-driving assessment.

5. Empirical results and ablation findings

On WOSAC, SMART-R1 reports a Realism Meta score of 0.7858, ranked first on the leaderboard at the time of submission. The paper states that this surpasses prior methods including TrajTok (0.7852), CLSFT (0.7846), and RLFTSim (0.7844). It also reports the highest kinematic metrics (average) at 0.4944, and an open-loop minADE of 1.2885, described as the best among compared models (Pei et al., 28 Sep 2025).

The reported gains are especially strong on safety-critical metrics, including Collisions, Off-road Rate, and Traffic Light Violations, all characterized as sparse and non-differentiable. Qualitatively, SMART-R1 is reported to reproduce diverse, realistic multi-agent behaviors, including both conservative and aggressive driving styles, while negotiating intersections and producing plausible deviations from logged trajectories when warranted by the scenario.

The ablation studies are central to the interpretation of the method. The paper states that:

• SFT–RFT–SFT outperforms SFT alone and SFT–RFT alone.
• Omitting the post-RFT SFT stage causes degradation through catastrophic forgetting.
• MPO outperforms PPO, DPO, and GRPO for policy optimization in this setting.

These results support the paper’s main design claim: reinforcement fine-tuning is beneficial, but only when coupled to a training schedule that preserves distributional fidelity after metric-driven optimization.

6. Position within R1-style post-training

SMART-R1 is presented as the first application of R1-style post-training in multi-agent traffic simulation and as the first demonstration that direct, metric-oriented reinforcement fine-tuning can align token-level autoregressive traffic models with desired high-level simulation metrics without using auxiliary value models or complex ranking structures (Pei et al., 28 Sep 2025).

In broader context, contemporaneous R1-style systems in other domains also combine supervised learning with reinforcement objectives, but they differ in representation, reward structure, and task semantics. R1-Onevision develops generalized multimodal reasoning through supervised fine-tuning and rule-based reinforcement learning with GRPO over formalized visual-text representations (Yang et al., 13 Mar 2025). LongVideo-R1 applies a two-stage SFT followed by RL paradigm to long-video question answering, using a reward that balances answer correctness, localization, and repeated visits in hierarchical clip navigation (Qiu et al., 24 Feb 2026). SmartCoder-R1 combines continual pre-training, Long Chain-of-Thought SFT, and Security-Aware GRPO to optimize compilation success, security compliance, and format correctness in smart-contract generation (Yu et al., 12 Sep 2025).

Against that landscape, SMART-R1 is distinguished by three properties. First, its base model is a next-token traffic simulator rather than a language-only or multimodal reasoner. Second, its RL target is the Realism Meta evaluation metric rather than answer accuracy, navigation efficiency, or code security. Third, it adopts MPO rather than GRPO-style optimization. This suggests that the “R1-style” designation in SMART-R1 refers less to a single fixed RL algorithm than to a broader post-training pattern: supervised initialization, direct optimization against task-level metrics, and explicit control of policy drift.