ECoSim: Data Efficient Fine-Tuning for Controllable Traffic Simulation

Abstract: Controllable traffic simulation is critical for testing autonomous driving systems, yet existing approaches often require retraining large generative models with extensive annotated data. We introduce a lightweight control adaptation framework that enables multi-modal controllability (sketch, latent behavior codes, and text) for pretrained state-of-the-art diffusion and autoregressive traffic models. By modulating intermediate features through identity-initialized FiLM layers, our method efficiently adds new control modalities while preserving the base model's generative prior. Evaluated on Waymo Open Sim Agents Challenge, our approach demonstrates strong controllability with less than 1% of the paired control data. Through context-aware condition transfer, our framework enables counterfactual scenario generation and long-tail synthesis while maintaining stable closed-loop driving realism and safety. Our framework unlocks new possibilities for controllable traffic simulation, enabling targeted scenario generation through lightweight adaptation of pretrained generative models. Project page: https://ecosim-web.github.io/

• The paper demonstrates a data-efficient framework that integrates lightweight FiLM adapters into frozen traffic models, achieving up to 83% mADE reduction with only 0.01%-1% paired data.
• It employs multi-modal control via sketch inputs, latent behavior codes, and textual descriptions to generate targeted, counterfactual traffic scenarios.
• The approach leverages context-aware retrieval and identity initialization to ensure realistic, diverse, and safe autonomous driving simulation outcomes.

Data-Efficient Control Adaptation for Multi-Modal Controllable Traffic Simulation

Introduction

Controllable traffic simulation is a critical requirement for scalable and systematic evaluation of autonomous driving systems. The core challenge is inducing user-specified behaviors—including rare and safety-critical interactions—while preserving distributional realism under closed-loop, multi-agent dynamics. The "ECoSim: Data Efficient Fine-Tuning for Controllable Traffic Simulation" paper (2607.00545) presents a comprehensive framework that augments state-of-the-art generative traffic models with lightweight, multi-modal control adapters, achieving effective scenario targeting and counterfactual generation with minimal paired supervision.

Model-Agnostic Control Adaptation via FiLM

ECoSim proposes a control adaptation architecture that embeds a lightweight parameterization—Feature-wise Linear Modulation (FiLM)—into frozen pretrained generative traffic models. Control signals, which may be trajectory sketches, learned latent behavior codes, or text descriptions, are encoded into a shared low-dimensional space. For agent $i$ at decoder layer $l$, feature vectors are modulated via affine FiLM transformations parameterized by control encodings:

$$\mathrm{FiLM}(\mathbf{h}^{(l)}; z_i) = \gamma^{(l)}(z_i) \odot \mathbf{h}^{(l)} + \beta^{(l)}(z_i)$$

where $\gamma^{(l)}$, $\beta^{(l)}$ are layer-specific MLPs initialized to approximate identity transformations, thus preserving the pretrained model prior at initialization. Only these adapters and control encoders are trained, keeping the core model static, which is crucial to prevent error compounding in closed-loop rollouts and enables robust, architecture-agnostic adaptation.

Figure 1: A lightweight, model-agnostic adapter injects multi-modal control signals via FiLM into a frozen pretrained traffic model, supporting both diffusion and autoregressive backbones.

Multi-Modal Control: Sketch, Latent, and Language Interfaces

The framework introduces three primary conditioning modalities:

1. Sketch Control: Sparse future waypoints specifying intended spatial maneuvers, encoded temporally and injected as control tokens.
2. Latent Behavior Codes: Compact, learned representations from a behavior-level conditional VAE (BehaviorVAE) trained on agent trajectories and scene context, supporting high-level intent transfer.
3. Textual Descriptions: Natural language processed via DistilBERT+LoRA adapters, allowing intent-driven scenario generation from abstract instructions.

The BehaviorVAE is a Gaussian latent scene-level autoencoder that infers and reconstructs multi-agent behavior through per-agent posteriors, with KL regularization to encourage smooth, semantically structured latent spaces.

Figure 2: BehaviorVAE encodes all agents' trajectories and contextual features into a Gaussian posterior, enabling structured, interaction-aware latent behavior control.

Context-Aware Scenario Transfer

Beyond direct conditioning, ECoSim incorporates a retrieval-augmented transfer mechanism. Agents in a large dataset are indexed by context embeddings; for a target agent in a scene, context-matched donor agents are retrieved and their behavioral encodings are injected, supporting domain-consistent counterfactual and long-tail scenario synthesis. This yields realistic context-compatible variations, overcoming limitations of direct, context-agnostic conditioning.

Figure 3: In the retrieval pipeline, agents are encoded and matched by context; behavior signals from selected donors are transferred to generate contextually compatible scenario variants.

Empirical Results and Sample Efficiency

Extensive evaluation is conducted on the Waymo Open Sim Agents Challenge (WOSAC), encompassing both diffusion-based (VBD) and autoregressive (SMART) backbone models. The key metrics are:

• WOSAC Meta Score: holistic closed-loop realism.
• mADE: control signal alignment.
• Collision/Offroad rates and PDMScore: safety and driving quality.
• Coverage: diversity across generated rollouts.

With only $0.01\%-1\%$ paired control data, the adapters yield strong controllability (up to $83\%$ mADE reduction), high realism (improvements of $0.02$–$0.03$ in Meta Score), and sample efficiency competitive to full-data LoRA finetuning.

Figure 4: Controllability (mADE, top) and realism (Meta Score, bottom) rapidly approach full-data reference performance with only $0.01\%$–$l$0 of paired data for latent and sketch control; saturation occurs near $l$1.

For both counterfactual and long-tail synthesis, context-aware transfer substantially enhances behavioral coverage and intent fidelity while maintaining safety.

Figure 5: Long-tail scenario generation—behaviors from rare cases (top) are transferred and meaningfully realized in new contexts, contrasting with conservative base model predictions (middle).

Qualitative results underline the framework's capacity to inject semantically diverse control signals that adapt to context constraints, such as maneuvers at intersections, dynamic acceleration, or complex lane changes.

Figure 6: Diverse control queries, including accelerating and lane change behaviors, are successfully injected and adaptively synthesized to match road topologies and multi-agent interactions.

Ablation and Design Analysis

Ablation studies confirm the criticality of multiplicative FiLM and identity initialization for realism preservation and rapid convergence relative to additive modulation or random initialization.

Figure 7: Multiplicative FiLM with identity initialization maintains realism and achieves strong control with fewer steps, unlike additive or randomly initialized variants.

For the BehaviorVAE, context concatenation—in addition to raw trajectory encoding—yields larger intra-cluster latent diversity, enabling robust generalization for control transfer.

Figure 8: t-SNE of the BehaviorVAE latent space—contextual information increases intra-cluster diversity, preventing mode collapse and enhancing topology-awareness.

Limitations and Implications

While highly data-efficient, the framework may prioritize user control signals to the detriment of reactive safety under strong context conflicts (e.g., forcing a stop amidst trailing vehicles or inducing aggressive maneuvers in dense traffic).

Figure 9: Failure cases—strict intent enforcement leads to unsafe maneuvers under severe context mismatch.

This highlights the need for future work in RL-driven or constraint-guided adaptation that can explicitly balance control fidelity and interactive safety, as well as extension to coordinated multi-agent control.

Implications and Outlook

ECoSim demonstrates that data-efficient, plug-in control adapters enable targeted, realistic, and diverse scenario generation across generative traffic simulation models and modalities. The adoption of context-aware transfer mechanisms addresses the open challenge of synthesizing rare behaviors for exhaustive evaluation. This architecture-agnostic, closed-loop framework paves the way for broad deployment in AV testing, data augmentation, and "what-if" analysis.

Anticipated future directions include:

• RL-based fine-tuning for balancing goal-seeking and reactive behaviors.
• Scalable multi-agent coordinated control.
• Generalization to OOD contexts and richer, human-in-the-loop scenario design.
• Integration with high-fidelity perception and planning stacks for end-to-end AV validation.

Conclusion

ECoSim establishes a highly practical and technical approach to controllable, multi-modal traffic simulation. The method achieves robust, targetable scenario generation with minimal paired data, leveraging feature modulation and context-matched transfer. It sets a new baseline for efficient, expressive, and safe simulation in data-driven autonomous driving research (2607.00545).