RLFTSim: RL Fine-Tuning for Traffic Simulation

• RLFTSim is a reinforcement-learning framework that fine-tunes pre-trained multi-agent traffic simulators in a closed-loop for enhanced realism.
• It employs a REINFORCE-style policy gradient with MLOO rewards to reduce variance and directly optimize realism metrics such as RMM.
• The framework integrates goal-conditioned control, enabling explicit trajectory commands to simulate rare or safety-critical autonomous scenarios.

Searching arXiv for RLFTSim and the cited papers to ground the article in current arXiv records. arXiv search: RLFTSim (Ahmadi et al., 18 May 2026); VLA-RFT (Li et al., 1 Oct 2025). RLFTSim is a reinforcement-learning-based fine-tuning framework for multi-agent traffic simulators that post-trains a strong pre-trained simulator in a closed-loop RL setting so that its rollouts statistically match real-world traffic behavior and can be controlled via explicit goals. Instantiated on top of SMART-tiny and evaluated on the Waymo Open Motion Dataset, it targets microscopic, multi-agent traffic simulation for autonomous-vehicle development and testing, especially for rare or safety-critical cases, and combines realism alignment with goal-conditioned controllability (Ahmadi et al., 18 May 2026).

1. Motivation and problem setting

RLFTSim was proposed against the background of supervised open-loop training, which has been widely adopted for learned traffic simulators such as SMART, Trajeglish, and TrafficBots. In that regime, the model receives past trajectories and map context and is trained to predict next trajectory token(s) from human drivers by cross-entropy or regression, but deployment is closed-loop: the simulator conditions on its own previous predictions rather than on ground-truth histories. The framework is explicitly designed to address the resulting train–test mismatch, causal confusion, lack of direct optimization of realism metrics, and absence of controllability (Ahmadi et al., 18 May 2026).

The train–test mismatch arises because errors accumulate during rollout, pushing agents into states that are unseen in training and causing realism to degrade. The causal-confusion problem is that a model may learn spurious correlations from logs that do not generalize in interactive simulation. A further limitation is that realism is evaluated by metrics such as Waymo’s Realism Meta-Metric (RMM), which compare distributional properties of rollouts, whereas standard supervised training does not optimize these metrics directly. Finally, conventional open-loop simulators are black-box samplers over typical behavior and do not naturally support commands such as making a specific agent perform a U-turn while remaining realistic.

RLFTSim addresses these issues by running the simulator in closed loop, using rewards derived from the realism metric itself, and extending the simulator to accept explicit goals for selected agents. This yields a formulation in which realism alignment and controllability are optimized within the same post-training pipeline.

2. Base simulator and RL formulation

RLFTSim is instantiated on top of SMART-tiny, a transformer-based multi-agent simulator trained as a next-token predictor. Its inputs are agent histories, tokenized actions, and map context. Using the notation of the framework, the state and action are

$$S_t = \{ S_{t'}^j \mid j \le N_a,\, t' \le t\}, \qquad A_t = \{A_t^j \in \mathcal{V} \mid j \le N_a\},$$

and the context is

$C = \{M, L\},$

where $M \in \mathbb{R}^{N_m \times D_m}$ denotes static polylines and $L \in \mathbb{R}^{N_{tl} \times D_{tl}}$ denotes traffic-light states. The architecture is a large transformer over tokenized agents and map elements, with parameter sharing across agents, so that a single policy handles all agents jointly through attention (Ahmadi et al., 18 May 2026).

The RL formulation is a contextual Markov Decision Process. The context is the map and lights, optionally augmented with goals; the state is the full multi-agent history up to time $t$; and the action is the joint set of next motion tokens for all agents. The policy $\pi_\theta$ is the simulator itself, parameterized by transformer weights $\theta$. The multi-agent structure is handled centrally: one network outputs joint distributions over all agents’ actions at each step, conditioned on the joint state and context.

Fine-tuning uses a REINFORCE-style Monte Carlo policy gradient with per-rollout rewards and a KL penalty toward a frozen pre-trained reference model. Conceptually, the optimization objective is

$$\max_\theta \; \mathbb{E}_{\tau\sim\pi_\theta}\big[\mathrm{RMM}^{\text{MLOO}}(\tau)\big] \;-\; \beta\,\mathbb{E}_{s}\big[\mathrm{KL}\big(\pi_\theta(\cdot|s)\,\|\,\pi_{\text{ref}}(\cdot|s)\big)\big].$$

The KL term is controlled by an adaptive KL controller and is used to prevent instability and degenerate policies during closed-loop fine-tuning.

3. Realism alignment through RMM and MLOO

The realism target in RLFTSim is the Waymo Open Sim Agents Challenge Realism Meta-Metric. RMM compares distributions of rollout-derived features with real-world ground truth rather than comparing raw trajectories directly. For a given context and ground-truth trajectory, $N=32$ rollouts are generated; features are computed for each rollout, agent, and time step; each feature dimension is discretized into $K=20$ bins; empirical probabilities are formed; and the metric evaluates how much probability mass the simulated distribution assigns to the bins occupied by the ground-truth trajectory. The feature families include kinematic quantities such as speed and acceleration, interactive quantities such as distances to other agents, time-to-collision, and collisions, and map-based quantities such as distance to road boundary, off-road behavior, and traffic-light violations (Ahmadi et al., 18 May 2026).

RMM is difficult to use directly as an RL reward because it is sparse and population-based: one scalar is obtained from a group of 32 rollouts, and standard credit assignment to individual rollouts is unavailable. It also has high variance when estimated from limited rollout sets. RLFTSim’s central technical device is Meta-metric Leave-One-Out (MLOO), which transforms RMM into a dense, low-variance per-rollout reward. If $C = \{M, L\},$0 denotes the meta-metric computed over the set excluding rollout $C = \{M, L\},$1, then the MLOO reward for rollout $C = \{M, L\},$2 is

$C = \{M, L\},$3

This reward has a sum-to-zero property,

$C = \{M, L\},$4

so it measures each rollout’s relative contribution to group realism.

The REINFORCE estimator is then

$C = \{M, L\},$5

The appendix proves that this estimator is unbiased for the gradient of the expected meta-metric on $C = \{M, L\},$6 rollouts. Under the paper’s importance-sampling model of RMM, the variance of MLOO scales as

$C = \{M, L\},$7

whereas the standard REINFORCE Leave-One-Out estimator based on single-rollout meta-metrics satisfies

$C = \{M, L\},$8

The framework therefore attributes its sample efficiency relative to heuristic search-based fine-tuning methods to the dense reward and to variance that shrinks quadratically with the number of rollouts.

The authors characterize this setup as realism alignment “by design,” because the reward is the benchmark realism metric itself, transformed into a per-rollout signal rather than replaced by a heuristic proxy.

4. Goal-conditioned controllability

RLFTSim extends realism alignment to goal-conditioned controllability. In this setting, a subset of agents is assigned explicit spatial goals $C = \{M, L\},$9, with $M \in \mathbb{R}^{N_m \times D_m}$0, and the simulator is trained to steer those agents toward their goals while keeping RMM high and preserving realistic behavior in uncontrolled agents (Ahmadi et al., 18 May 2026).

Two goal-achievement criteria are defined. Under the hard goal criterion, agent $M \in \mathbb{R}^{N_m \times D_m}$1 reaches goal $M \in \mathbb{R}^{N_m \times D_m}$2 if its final position at the horizon is within 2.0 m of the goal. Under the soft goal, or passing criterion, the agent reaches the goal if it comes within 2.0 m of the goal at any time during the rollout. The framework also defines a goal polyline,

$M \in \mathbb{R}^{N_m \times D_m}$3

namely the map polyline whose origin is closest to the goal coordinate.

Two architectural variants are used. In the concatenation variant, each agent token embedding is augmented directly with goal information, either by raw coordinates $M \in \mathbb{R}^{N_m \times D_m}$4 or by relative coordinates in polar form. In the indication variant, coordinates are not concatenated; instead, the relative positional encoding between agent tokens and map polylines is extended with a binary flag indicating whether a given map polyline is that agent’s goal polyline. The indication variant has an additional property: an agent becomes goal-free simply by not marking any polyline as a goal, which permits hybrid simulations in which some agents are goal-conditioned and others are not.

Because goal-conditioned RL is sparse, RLFTSim uses a variant of Hindsight Experience Replay. For a given seed consisting of context, history, and goals, $M \in \mathbb{R}^{N_m \times D_m}$5 stochastic rollouts are generated; each rollout is evaluated with RMM; the highest-RMM rollout is selected; and its achieved final states are used as alternate goals for additional training tuples. During training, policy ratios for hindsight goals are recomputed following hindsight policy gradients:

$M \in \mathbb{R}^{N_m \times D_m}$6

The goal-conditioned fine-tuning reward combines realism and goal achievement:

$M \in \mathbb{R}^{N_m \times D_m}$7

where $M \in \mathbb{R}^{N_m \times D_m}$8 is the mean binary goal-reaching indicator over evaluated agents and $M \in \mathbb{R}^{N_m \times D_m}$9; the experiments use $L \in \mathbb{R}^{N_{tl} \times D_{tl}}$0. This formulation encodes a direct realism–controllability trade-off rather than optimizing either objective in isolation.

5. Training pipeline and evaluation protocol

The training pipeline has two stages. First, SMART-tiny is trained on the Waymo Open Motion Dataset for 32 epochs using cross-entropy for next-token prediction. The training set contains 486,995 scenarios, with 1.1 s history and 8 s future. Second, RL fine-tuning is performed in closed loop: goal-free RLFTSim uses $L \in \mathbb{R}^{N_{tl} \times D_{tl}}$1 as the realism reward, and GCFT further fine-tunes RLFTSim with the composite realism-plus-goal reward (Ahmadi et al., 18 May 2026).

The RL algorithm is Monte Carlo policy gradient. Stabilization is provided by a KL divergence penalty to a frozen reference model, controlled by an adaptive KL controller in the spirit of RLHF, with a KL target of approximately 0.01 nats. The reported hyperparameters are a learning rate of $L \in \mathbb{R}^{N_{tl} \times D_{tl}}$2, 4 rollouts per update for training, 32 rollouts for evaluation, batch size 8, gradient clipping with norm 1, controller horizon 5, minimum and maximum KL coefficients of $L \in \mathbb{R}^{N_{tl} \times D_{tl}}$3 and $L \in \mathbb{R}^{N_{tl} \times D_{tl}}$4, and goal reward weight $L \in \mathbb{R}^{N_{tl} \times D_{tl}}$5. Rollouts are generated on-policy with temperature sampling over top-32 trajectory tokens.

Evaluation follows the Waymo Open Sim Agents Challenge protocol. WOMD contains 486k train, 44k validation, and 44k test scenarios, each up to 128 agents including vehicles, cyclists, and pedestrians, over a total 9.1 s window. Realism is evaluated from 32 rollouts per scenario. Evaluation includes the ego agent and up to 8 tracks-to-predict agents, although the simulation itself includes all agents.

The realism metrics comprise overall RMM and its kinematic, interactive, and map-based sub-components. Auxiliary ablation metrics include collision rate, off-road rate, and ADE or minADE. Controllability metrics include Passing Miss Rate, reach rate under the hard criterion, and pass rate under the soft criterion. Baselines for realism include TrafficBots V1.5, VBD, MVTE, Trajeglish, KiGRAS, DRoPE-Traj, GUMP, BehaviorGPT, UniMM, TrajTok, the original SMART-tiny, and SMART-tiny with CAT-K closed-loop fine-tuning. Reward ablations include minADE(RLOO), RMM(RLOO), Collision + Offroad + ADE, and Collision + Offroad. Controllability baselines include SMART-tiny without goal conditioning, goal-free RLFTSim, and GCFT variants with concatenation or indication and hard or soft rewards.

6. Reported results and empirical interpretation

On the Waymo test leaderboard using RMM v2025 weighting, RLFTSim reports overall RMM 0.7867, kinematic 0.4927, interactive 0.8129, and map-based 0.9210. In the same comparison, SMART-tiny retrained by the authors reports RMM 0.7824, SMART-tiny with CAT-K reports 0.7856, and TrajTok reports 0.7861. The strongest gain is on interaction realism, where RLFTSim is reported as best overall (Ahmadi et al., 18 May 2026).

Validation-set ablations on a 20% subset report base SMART-tiny at RMM 0.7804, RLFT with $L \in \mathbb{R}^{N_{tl} \times D_{tl}}$6 at 0.7830, and RLFT with $L \in \mathbb{R}^{N_{tl} \times D_{tl}}$7 at 0.7821. RLFT with minADE(RLOO) does not improve RMM and worsens minADE. Heuristic rewards improve their targeted metrics, such as collisions or off-road behavior, but do not match MLOO-based optimization on RMM. These results are used to argue that directly optimizing an imitation metric does not improve realism at a near-optimal stage, and that heuristic safety rewards can plateau on RMM or damage other aspects of realism.

For controllability on the full WOMD validation set, goal-free RLFTSim reports a passing miss rate of approximately 16.6%. The best GCFT variant, indication with soft goal reward, reports passing miss rate approximately 9.2% and RMM about 0.782–0.7827, described as only slightly reduced relative to realism-only RLFTSim. The concatenation variant with soft reward reports passing rate 10.47% and RMM 0.7776, while indication with soft reward reports passing 9.18% and RMM 0.7819. Hard rewards improve exact reaching more strongly, whereas soft rewards improve passing near the goal.

On the maneuver controllability benchmark restricted to alternative maneuvers, baseline SMART-tiny reports reach rate 35.4% and pass rate 42.9%. The best GCFT soft variant, concatenation with soft reward, reports reach 37.6% and pass 75.3%; the best hard variant, concatenation with hard reward, reports reach 50.0% and pass 68.4%. On the kinematic perturbation benchmark, with goals displaced in time from $L \in \mathbb{R}^{N_{tl} \times D_{tl}}$8 to $L \in \mathbb{R}^{N_{tl} \times D_{tl}}$9 seconds, GCFT variants consistently outperform the baseline across perturbations.

Qualitative examples reported in the paper show reductions in off-road driving, collisions at intersections, failures to yield to pedestrians, and dangerous departures of parked cars into traffic. They also show that distinct goals in the same scene can induce U-turn, left-turn, straight, stop, or right-turn behaviors while other agents continue to behave realistically. A plausible implication is that the framework’s controllability mechanism is not merely endpoint matching but conditional reshaping of the joint scene dynamics.

7. Broader context, terminological scope, and limitations

Within the broader traffic-simulation literature, RLFTSim is positioned against rule-based simulators such as IDM and Waymax, which provide physics-consistent behavior but lack realism in human interactions and diversity, and against learning-based simulators such as TrafficSim, SMART, BehaviorGPT, and TrafficBots, which typically rely on open-loop imitation learning. Prior alignment attempts discussed in the paper include CAT-K, imitation-plus-RL for ego-centric planning, RLHF-style methods for traffic, and offline preference-based alignment such as DPA-OMF. RLFTSim is distinguished as an on-policy RL method that uses the official realism metric as reward, transforms it into a dense low-variance signal via MLOO, provides formal variance analysis, and adds goal-conditioned controllability through architectural conditioning and hindsight relabeling. The paper further states that the framework is model-agnostic and is demonstrated on SMART and TrafficBots V1.5, improving RMM in both (Ahmadi et al., 18 May 2026).

The paper also records several limitations. First, trajectories are tokenized into approximately 0.5 s chunks, which may reduce responsiveness in highly dynamic or emergency situations. Second, controllability improves substantially but does not guarantee goal completion. Third, RMM is an imperfect proxy for realism, so saturation in RMM may reflect metric limitations rather than simulator limitations. Fourth, the reported evaluation horizon is 8 seconds, and longer horizons or more complex urban scenarios may require additional modeling and reward shaping. The proposed future directions are to apply MLOO-style RL fine-tuning to other population-based metrics, scale to larger or different datasets such as nuPlan or Argoverse2, design richer goal specifications including natural language or semantic goals, and integrate aligned simulators more tightly with planning and control stacks.

A separate terminological usage appears in the embodied-policy paper "VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators" (Li et al., 1 Oct 2025). There, “RLFTSim” denotes reinforcement-learning fine-tuning conducted almost entirely inside a learned, data-driven simulator, with the simulator used both for rollouts and for dense, trajectory-level, verified rewards. That usage belongs to world-model-based reinforcement fine-tuning for vision-language-action systems rather than to the multi-agent traffic-simulation framework introduced in RLFTSim proper.