Can Users Specify Driving Speed? Bench2Drive-Speed: Benchmark and Baselines for Desired-Speed Conditioned Autonomous Driving

Abstract: End-to-end autonomous driving (E2E-AD) has achieved remarkable progress. However, one practical and useful function has been long overlooked: users may wish to customize the desired speed of the policy or specify whether to allow the autonomous vehicle to overtake. To bridge this gap, we present Bench2Drive-Speed, a benchmark with metrics, dataset, and baselines for desired-speed conditioned autonomous driving. We introduce explicit inputs of users' desired target-speed and overtake/follow instructions to driving policy models. We design quantitative metrics, including Speed-Adherence Score and Overtake Score, to measure how faithfully policies follow user specifications, while remaining compatible with standard autonomous driving metrics. To enable training of speed-conditioned policies, one approach is to collect expert demonstrations that strictly follow speed requirements, an expensive and unscalable process in the real world. An alternative is to adapt existing regular driving data by treating the speed observed in future frames as the target speed for training. To investigate this, we construct CustomizedSpeedDataset, composed of 2,100 clips annotated with experts demonstrations, enabling systematic investigation of supervision strategies. Our experiments show that, under proper re-annotation, models trained on regular driving data perform comparably to on expert demonstrations, suggesting that speed supervision can be introduced without additional complex real-world data collection. Furthermore, we find that while target-speed following can be achieved without degrading regular driving performance, executing overtaking commands remains challenging due to the inherent difficulty of interactive behaviors. All code, datasets and baselines are available at https://github.com/Thinklab-SJTU/Bench2Drive-Speed

• The paper presents Bench2Drive-Speed, the first framework offering a benchmark and dataset with explicit speed and overtaking commands for E2E autonomous driving.
• It utilizes a CARLA-based closed-loop evaluation with three difficulty levels, measuring performance through Speed-Adherence and Overtake Scores alongside traditional metrics.
• Results show that models with explicit speed conditioning, including those using virtual annotations, achieve strong speed tracking but face challenges in complex overtaking scenarios.

Explicit Speed Conditioning in End-to-End Autonomous Driving: Bench2Drive-Speed

Motivation and Contributions

Bench2Drive-Speed introduces the first benchmark, dataset, and baseline methodology for explicit user-specified speed and overtaking commands in end-to-end autonomous driving (E2E-AD), addressing a significant deficiency in prior work: the absence of controllable speed adherence and overtaking behavior as first-class evaluation targets. Unlike prior style-aware AD frameworks that embed speed within abstract behavioral categories, Bench2Drive-Speed makes speed and overtaking instructions direct input modalities, enabling quantitative assessment of controllability in realistic closed-loop environments.

Figure 1: Bench2Drive-Speed enables specification of target-speed and overtake/follow commands, introducing explicit controllability metrics for autonomous vehicles.

Benchmark Design and Scenario Structure

Bench2Drive-Speed defines a closed-loop evaluation suite built on the CARLA simulator, extending route configuration to support segment-wise speed commands and overtaking/follow instructions. Three difficulty strata (Easy, Medium, Hard) systematically stress control compliance under progressively challenging traffic configurations, ranging from isolated routes to dynamic urban environments and complex accident avoidance.

Figure 2: Drive includes speed-conditioned tasks and explicit metrics, with expert and virtual speed annotations across 2,100 annotated scenarios.

Command adherence is evaluated along two axes: speed tracking (Speed-Adherence Score) and overtaking execution (Overtake Score), both jointly reported with traditional AD metrics (safety, comfort, traffic compliance). This allows rigorous analysis of efficiency-safety trade-offs and passenger comfort impacts attributable to speed control modeling.

Figure 3: Scenario difficulty increases from easy to hard, with overtaking and following adherence evaluated in challenging environments.

Dataset and Annotation Methods

The “Customized” dataset comprises 2,100 scenarios systematically annotated with target-speed and overtaking/follow commands, collected via the PDM expert-driven controller in CARLA. Customized incorporates both expert target-speed signals (derived from internal model hyperparameters) and re-annotated "virtual target speed" labels, constructed via monotonic trend extraction and controlled extrapolation from future speed trajectories, reflecting feasible annotation strategies for real-world data devoid of privileged access.

Figure 4: Customized dataset structure, encompassing visual sensory inputs, ego-state, explicit bounding and command annotations.

Virtual speed annotation is formulate parameterizable as either “short” (fine, conservative) or “long” (aggressive, broader extrapolation), enabling empirical analysis of supervision quality and transferability.

Figure 5: Expert-derived speed annotations require controller access; re-annotation (virtual speed) is tractable for large real-world datasets.

Customized features diverse scenarios—complex urban, dense traffic, obstacle avoidance—and balanced distributions across speed targets, environmental factors, and route configurations, preventing shortcut learning and ensuring within-route command variability.

Figure 6: Customized covers complex safety-critical driving scenarios, enforcing target-speed and overtaking/follow commands.

Figure 7: Customized exhibits a diverse, balanced target-speed command distribution and scenario difficulty stratification.

Baseline Architecture and Model Evaluation

The baseline implementation adapts the TCP architecture, concatenating visual features, ego state, desired-speed, and overtaking commands to guide trajectory prediction. The model is agnostic to input modality, permitting generic adoption across architectures.

Figure 8: TCP model accepts fused input (visual + state + speed/overtake command) for trajectory generation.

Evaluations reveal TCP trained with explicit speed-conditioned supervision (both expert and virtual) achieves strong Speed-Adherence Score—consistently outperforming unconditioned models by substantial margins. Models trained with virtual annotations reach parity with expert-labeled counterparts, establishing a proof-of-concept for scalable, non-privileged speed conditioning in practical datasets.

Figure 9: TCP exhibits robust adherence to dynamically specified target speeds across varying route fragments.

Figure 10: Heatmap of speed-adherence: virtual-short supervision attains maximal consistency and stability across evaluation routes.

Overtaking command execution remains challenging, particularly in hard scenarios—baseline policies succeed in adhering to longitudinal speed commands without notable safety degradation but underperform on lateral overtaking-risk trade-offs, frequently incurring safety failures in aggressive maneuvers.

Figure 11: TCP successfully executes both overtaking (upper) and following (lower) cases, but overall overtaking reliability remains low in complex scenarios.

Practical and Theoretical Implications

Bench2Drive-Speed demonstrates that explicit speed-conditioning is technically feasible within E2E-AD frameworks and can be reliably trained from regular datasets via virtual annotation. This versatility opens pathways for user-tailored AD policies, addressing practical needs such as comfort-speed trade-offs, aggressive/conservative cruising, and customized behavior profiles.

The framework establishes quantitative controllability metrics for closed-loop evaluation, formally linking user command inputs to measurable policy outcomes in speed and overtaking, advancing the science of interpretable and customizable E2E-AD. The challenges in overtaking execution highlight intrinsic difficulties in interactive planning not yet fully addressed by current architectures, suggesting promising future research directions in long-horizon planning and risk-aware command compliance.

Scalability of virtual annotation mechanisms enables practical deployment in large real-world AD datasets. Adoption of explicit command modality marks a shift from implicit style embedding toward transparent policy parameterization, fostering tangible end-user control.

Conclusion

Bench2Drive-Speed provides the first systematic benchmark, dataset, and methodology for user-specified speed-conditioned autonomous driving, establishing explicit controllability evaluation for E2E-AD policies. Models trained with speed-conditioning achieve strong speed adherence and preserve safety and comfort, but overtaking reliability remains limited. These results motivate further development of interactive control architectures and scalable annotation mechanisms. The work lays foundational infrastructure for customizable AD policy research and practical deployment of end-user controllable autonomous vehicles (2603.25672).