Controllable Sim Agents with Behavior Latents

Abstract: Realistic traffic simulation requires agents that imitate logged behavior and can also be steered along interpretable axes. Such controllability enables engineers to isolate variables, reproduce specific edge cases, and test autonomous systems without real-world risk. We introduce Controllable Neural Variational Agents (CNeVA), a controllable simulated-agent framework that learns to infer a per-agent Gaussian behavior latent from per-channel discounted returns via a closed-form conjugate variational update, conditioning a rectified-flow trajectory generator trained on a mixed channel-mask curriculum for classifier-free guidance. To tackle scarcity in reward signals, we propose soft eligibility gates that replace hard binary thresholds with smooth exponential decay, preserving the gradient signal for near-threshold agents. On the Waymo Open Motion Dataset, CNeVA attains competitive realism on the benchmark while exposing per-channel controllability that the higher-ranked imitation models lack. Speed- and acceleration-based steering produces monotone responses without stall-induced reward hacking. Safety controllability is monotone and substantial with the introduction of soft eligibility. We manage to achieve steerable map compliance under a context-residual return measure. Furthermore, our experiment demonstrates that steering metrics must be read alongside physical-plausibility guardrails to avoid reward-hacking confounds.

• The paper introduces a closed-form variational inference method that learns per-agent behavior latents from traffic data for explicit control.
• It employs a conditional neural generator with multi-branch channel masking to achieve fine-grained steering over safety, speed, and map adherence.
• Experimental results on the Waymo dataset confirm competitive realism and effective controllability with robust physical guardrails.

Controllable Neural Variational Agents: A Framework for Behavior-Controllable Traffic Simulation

Overview and Motivation

The paper "Controllable Sim Agents with Behavior Latents" (2607.02496) introduces Controllable Neural Variational Agents (CNeVA), a novel framework for simulating traffic agents whose behavior is both realistic and controllable along interpretable axes such as safety, speed, acceleration, and map compliance. The need for such controllability is motivated by the necessity to isolate behavioral variables, reproduce edge cases, and evaluate autonomous vehicle (AV) systems under a wide spectrum of agent responses within high-fidelity simulations.

Compared to existing VAE, diffusion, and RL-based simulation approaches, CNeVA provides direct, per-channel behavioral steering without resorting to expensive retraining or the complexity and instability of gradient-based guidance. The framework grounds behavioral control in a compact per-agent latent learned from log-replay data via a closed-form variational update, and uses a highly scalable neural conditional generator allowing flexible downstream control.

Probabilistic Modeling and Behavior Latents

CNeVA models each simulated agent's behavior using a $K$-dimensional Gaussian latent vector $\bm{\lambda}_n$, where each dimension corresponds to a reward "channel" (e.g., safety, speed, map adherence, acceleration). The latent acts as a behavioral profile that encodes the agent's weighting of these semantic dimensions. Key architectural advances include:

• Closed-form conjugate variational inference: For each agent, CNeVA infers $\bm{\lambda}_n$ as a conjugate Gaussian posterior conditioned on empirical, per-channel discounted returns computed from historical trajectory data. This inference is analytic and leverages the exponential family structure of the reward decomposition.
• Equivalence to tilted regression: The update can be interpreted as Bayesian regression, where dense (frequent) return channels yield strong identifiability (steering power), while sparse channels (such as collisions) remain prior-dominated, anticipating the controllability hierarchy observed in experiments.

  Figure 2: CNeVA infers a per-agent behavior latent $\bm{\lambda}_n$ from per-channel discounted returns, enabling fine-grained, interpretable control during scenario generation.

Conditional Generation and Mixed Channel Guidance

CNeVA incorporates its per-agent latent into a conditional rectified-flow trajectory generator trained using a mixed channel-mask curriculum. Notable components include:

• Classifier-free guidance via multi-branch masking: The model sees null, single-channel, partial, and full conditioning branches during training, with channel masks allowing precise activation or deactivation of behavior axes. This curriculum enables robust guidance even outside the support of the standard population posterior.
• Operator control at inference: At deployment, users provide $\bm{\lambda}_n^{\mathrm{op}}$ directly in standardized coordinates, supporting fine-grained population-marginal, user-controlled, or prior-randomized rollouts. Classifier-free guidance with continuous scaling $w$ offers smooth control over the influence of each axis.

Reward Labeling, Eligibility, and Plausibility

To address the practical challenges of reward sparsity and metric validity, the paper introduces:

• Context-residual returns: Per-channel returns are residualized relative to scenarios, isolating behavioral signals from scene-specific difficulty and providing more meaningful labels.
• Soft eligibility gates: Unlike standard hard gating (where only agents near a threshold are supervised), CNeVA employs smooth exponential decay on risk metrics (clearance, TTC, margin), ensuring a continuous and informative gradient for agents in ambiguous or near-hazard states.
• Contrastive conditioning: Training is further stabilized by explicit contrastive loss between "steered" and null paths, improving the model's robustness to closed-loop drift and covariate shifts.

  Figure 1: Empirical histogram of per-channel discounted returns $G_{n,k}$, motivating the necessity of standardized, channel-balanced latent inference.

Experimental Results: Realism and Controllability

CNeVA is evaluated on the Waymo Open Motion Dataset using the stringent WOSAC protocol. Results include:

• Realism (WOSAC meta-metric 0.7145, minADE 1.80m): CNeVA's unconditional path attains competitive realism relative to state-of-the-art imitation and generative baselines, though some gap remains versus token-based, non-controllable models, mainly due to open-loop error accumulation.
• Per-channel steering: Through drift-paired channel-steering matrices (CSM), the model demonstrates strong, monotonic, and physically plausible channel controllability:
  • Speed/accel steering:
  • Large response in corresponding channels ($\Delta R_{\mathrm{speed}} = +8.15$, $\Delta R_{\mathrm{accel}} = +8.76$ at calibrated settings), without stalling or unrealistic reward hacking.
  • Safety/map compliance:
  • Significant controllability under soft eligibility ($\Delta R_{\mathrm{safety}} = +0.66$, $\bm{\lambda}_n$0), outperforming hard eligibility ablations.
  • Physical guardrails: Guardrails such as stall fraction and retained speed confirm that increased returns result from genuine agent behavior changes, not reward-channel exploitation.
  • 

  • Figure 5: Reward-hacking contrast: early-stage models can inflate channel returns (e.g., speed) via pathological behavior (stalling), while the full CNeVA model achieves high returns with minimal stalling and realistic motion.

  • 

  • Figure 4: Channel steering matrices (CSM) demonstrate strong controllability for kinematic channels and substantial, positive responses for safety and map channels under soft eligibility.

Limitations and Interpretability

Despite its strengths, CNeVA exhibits channel-specific limitations:

• Map compliance controllability is sensitive to reward definitions: Steering in the "map" channel is only effective when context-residual returns are used; coordinate-level (e.g., precise lane-keeping) control is not directly achieved under aggregate or geometric map returns.
• Controller mismatch at extreme settings: For high-magnitude steering ($\bm{\lambda}_n$1), physical plausibility guardrails degrade, highlighting the importance of calibrated operator settings and possible need for tighter integration of physics constraints.

Implications and Future Directions

CNeVA offers a tractable and highly interpretable paradigm for simulating diverse agent behaviors relevant to AV evaluation, scenario generation, and robustness analysis. Its approach suggests a broader framework for behavior-conditioning in generative models, wherein reward decomposition and analytic inference provide a thin and powerful interface between log data and scenario control.

Theoretically, CNeVA's reliance on linear exponential-family conjugacy points to the potential for more expressive behavior spaces, structured priors, or non-Gaussian extensions, especially in settings with rich, high-dimensional reward bases. Practically, further advances may include:

• Enhanced, coordinate-level reward decompositions for spatially localizable control (e.g., lane selection).
• Closed-loop fine-tuning or contrastive objectives leveraging both simulation and real-world interactions.
• Extension of the control interface beyond one-hot per-channel steering for richer, multi-objective scenario manipulation.

Conclusion

CNeVA delivers a technically rigorous, data-efficient framework for generating traffic agents with explicit, interpretable, and physically-plausible behavior control. Through a tight integration of closed-form latent inference, conditional neural generation, and novel eligibility mechanics, it establishes a strong baseline for future work at the intersection of generative modeling and simulation control for autonomy research.

Figure 7: Ground truth (left) and reward-hacking analysis: soft eligibility prevents unrealistic return inflation, confirming genuine behavioral adaptation rather than spurious metric exploitation.

Figure 3: Map CSM diagonal ($\bm{\lambda}_n$2) shows positive controllability only under context-residual returns, revealing the metric's dependence on reward decomposition choices.