AutoDriver: Design Space of Autonomous Driving

• AutoDriver is a family of systems that translates sensory observations into driving behavior using both direct end-to-end and modular hierarchical approaches.
• It integrates diverse methodologies such as vision-only controllers, multimodal world models, and language-mediated planning to optimize control and interpretability.
• Recent studies report robust closed-loop performance with clear trade-offs in safety, occlusion handling, and modularity, guiding next-generation autonomous systems.

AutoDriver denotes a family of systems that translate sensory observations, contextual instructions, or user constraints into driving behavior. In the literature considered here, the term spans practical vision-only end-to-end controllers deployed on real vehicles, multimodal world models that interleave perception and action, hierarchical language agents that emit mid-level commands, diffusion planners that synthesize a unified bird’s-eye-view representation, user-controlled planning interfaces, and human-centered vehicle concepts for specific passenger populations (Aoki et al., 2023, Jia et al., 2023, Zhang et al., 2024, Wang et al., 15 Mar 2025, Wang et al., 2024, Padmanaban et al., 2021). Across these variants, the central question is how much of the classical perception–prediction–planning–control stack should remain modular, and how much should be absorbed into a single learned policy, a generative model, or a language-mediated decision process.

1. Architectural scope and system families

The recent literature does not present a single canonical AutoDriver architecture. Instead, it presents a set of competing formulations that differ in supervision, temporal abstraction, interpretability, and control interface.

System	Primary formulation	Representative reported outcome
SuperDriverAI (Aoki et al., 2023)	Vision-only imitation learning from camera frames to multi-step steering	Real-world steering actuation with 150 runs in one scenario
ADriver-I (Jia et al., 2023)	Interleaved vision-action world model with MLLM and video diffusion	Speed L1 0.072, steering L1 0.091 on nuScenes; FID 5.5, FVD 97.0
AD-H (Zhang et al., 2024)	Hierarchical planner-controller system with mid-level language commands	LangAuto DS 44.0, RC 53.2, IS 0.83
UAD (Guo et al., 2024)	Vision-based E2EAD with unsupervised angular perception pretext	Avg collision 0.19% on nuScenes; 7.2 FPS
DiffAD (Wang et al., 15 Mar 2025)	Conditional latent diffusion over unified BEV perception/prediction/planning canvases	Bench2Drive DS 67.92, SR 38.64

These systems share the ambition of reducing redundancy in the conventional stack, but they do so in different ways. SuperDriverAI replaces the modular stack with a direct perception-to-control pipeline. ADriver-I couples action prediction with future-frame generation, so that control and environment evolution are modeled jointly. AD-H explicitly preserves a hierarchy, but shifts the intermediate representation from trajectories or object lists to language-driven commands. UAD removes manual 3D supervision through an unsupervised proxy, and DiffAD unifies perception, prediction, and planning as a single conditional image-generation problem. This suggests that “AutoDriver” is best understood not as a single architecture, but as a design space organized around how end-to-end the system is, what latent interface it uses, and how it enforces safety and interpretability.

2. End-to-end perception-to-control formulations

In its most direct form, AutoDriver is a learned policy that maps sensor observations to low-level actuation. SuperDriverAI is a practical example of this approach. Its pipeline has two phases: in Phase A, experienced human drivers operate a real vehicle equipped with an embedded monocular forward-facing camera and a data logging system that records GPS, steering wheel angle, throttle, and brake states; in Phase B, trained models are loaded back to the vehicle for deployment under real-world conditions. During inference, the main input is the image stream, while GPS is used primarily for time-alignment and metadata rather than as a model input. The model ingests a short sequence of consecutive frames corresponding to approximately three seconds, uses a ResNet18 backbone with multi-frame input, and regresses a vector of steering angles over multiple future steps. Training follows supervised imitation learning with the objective

$$L(\theta)=\sum_t \|a_t-a_t^*\|_2^2+\lambda\|\theta\|_2^2,$$

and deployment in the reported demonstration actuates only steering while speed is held constant for safety. The same hardware set is used across training and testing, and the abstract reports 150 runs for one driving scenario in Tokyo, while the detailed deployment description specifies a practical test field in Kawasaki City (Aoki et al., 2023).

A distinctive element of SuperDriverAI is the “slit model,” which is not a multi-branch network but a training strategy that crops the camera image to simulate pseudo-displacement induced by sensor misalignment or delays. By training cropped views with the same ground-truth actions, the policy learns corrective steering and path recovery. The result is an end-to-end controller that remains comparatively simple at inference time, yet is explicitly regularized for robustness to installation drift and physical-world perturbations (Aoki et al., 2023).

UAD and DiffAD push the end-to-end idea further by attacking the cost of supervision rather than only the model boundary. UAD argues that even recent end-to-end methods still mimic modular stacks through supervised 3D detection, tracking, mapping, and occupancy heads. Its alternative is an Angular Perception Pretext that partitions the BEV into angular sectors, predicts angular-wise spatial objectness and temporal dynamics, and couples this with a self-supervised directional consistency objective. Its overall objective is

$$\mathcal{L}=\omega_1\mathcal{L}_{\rm spat}+\omega_2\mathcal{L}_{\rm drm}+\omega_3\mathcal{L}_{\rm imi}+\omega_4\mathcal{L}_{\rm dir}+\omega_5\mathcal{L}_{\rm cons},$$

thereby removing the requirement for manual 3D labels in the target domain while still planning directly from multi-view camera input (Guo et al., 2024).

DiffAD redefines autonomous driving as conditional BEV image generation. It rasterizes three RGB canvases—perception, prediction, and planning—into a 9-channel pixel representation, compresses them with a Stable Diffusion VAE into a latent BEV tensor, and denoises that latent with a DiT backbone. A Trajectory Extraction Network then regresses the ego trajectory directly from the latent representation, avoiding a proliferation of task-specific heads. The training objective is

$L = L_{\rm denoising}+L_{\rm extraction},$

with the denoising term learning the latent BEV distribution and the extraction term supervising the ego trajectory (Wang et al., 15 Mar 2025). Relative to direct steering regression, this formulation preserves a structured intermediate state while still training the entire system as a unified generative model.

3. World models, language, and hierarchical agents

A second line of AutoDriver research retains end-to-end ambition but changes the interface from direct control to language or token sequences. ADriver-I is the most explicit world-model formulation in this set. It introduces interleaved vision-action pairs

$S=\{(x_1,a_1),(x_2,a_2),\ldots,(x_T,a_T)\},$

where visual input is encoded by CLIP-ViT-Large plus a learned two-layer visual adapter into the Vicuna-7B-1.5 token space, and control signals are encoded as “absolute number” text tokens. The model factorizes joint action and future-frame prediction as

$$p(a_{1:T},x_{1:T})=\prod_{t=1}^{T} p_\theta(a_t\mid h_t)\,p_\phi(x_{t+1}\mid h_t,a_t),$$

with the MLLM predicting the current control and a video diffusion model, built on Stable Diffusion 2.1 with temporal modules, generating future frames conditioned on the chosen action. Because predicted frames are fed back into subsequent control prediction, the system performs autonomous driving “in the world created by itself” (Jia et al., 2023).

Empirically, ADriver-I reports on nuScenes a speed prediction error of 0.072 L1 and a steering prediction error of 0.091 L1, with threshold accuracies reported at $\theta \in \{0.01,0.03,0.05,0.07\}$. For future generation, with four history frames and four predicted future frames, it reports FID $=5.5$ and FVD $=97.0$, substantially stronger than the baselines included in the study. The system also reports that “absolute number” action encoding is better than English phrases or relative differences, and that multi-round supervision improves final action L1 relative to single-round training (Jia et al., 2023).

AD-H addresses a different failure mode: the mismatch between MLLM pretraining and continuous low-level control. Its solution is hierarchical decomposition. A multimodal planner, instantiated with LLaVA-7B-V1.5 or Mipha-3B, ingests four surround-view RGB images concatenated vertically and a high-level instruction, and emits a mid-level language command. A lightweight controller, based on ResNet-50, Q-Former, and OPT-350M, then converts the command into five future waypoints, which are tracked by PID or MPC-like control. The action hierarchy contains 26 sub-commands that compose into more than 160 mid-level commands, such as “Approaching a junction, prepare to follow traffic rules,” “Slow down to ensure safety,” and “Make a slight left turn” (Zhang et al., 2024).

This decomposition materially affects closed-loop performance. On LangAuto, AD-H with LLaVA-7B and OPT-350M reports DS 44.0, RC 53.2, and IS 0.83, compared with LMDrive’s DS 36.2, RC 46.5, and IS 0.81. On LangAuto-Long-Horizon it reports DS 62.1, IS 0.875, and RC 68.3, compared with LMDrive’s DS 49.1, IS 0.871, and RC 56.4. The paper further attributes a self-correction capability to language re-planning at each decision frame, including oversteering scenarios absent from the training dataset (Zhang et al., 2024).

Agent-Driver takes yet another route. Rather than replacing the stack with a single network, it places an LLM agent above a tool library, a cognitive memory, and a self-reflective planning loop. Tool calls query detection, prediction, occupancy, and map modules only as needed; memory retrieval combines vector K-NN with LLM fuzzy search over commonsense rules and episodic experiences; the reasoning engine produces chain-of-thought summaries, high-level task plans, motion plans, and a self-reflection step that checks and optimizes the candidate trajectory. On nuScenes, Agent-Driver reports under the ST-P3 metric convention an average L2 of 0.37 and an average collision rate of 0.09%, compared with GPT-Driver’s 0.44 and 0.17% and VAD’s 0.37 and 0.14%. Under the UniAD convention it reports average L2 0.74 and average collision 0.21%, outperforming UniAD’s 1.03 and 0.31% (Mao et al., 2023).

4. Interpretability, safety layers, and human-facing control

A persistent criticism of end-to-end driving is that it obscures the basis of decisions. Several AutoDriver variants therefore add interpretability or human-facing control as first-class components. In SuperDriverAI, Grad-CAM is used to identify image regions most influential for the predicted steering command, and the reported examples show attention concentrated on lane markers and roadside objects such as garden trees. A separate monocular depth estimation network provides qualitative scene geometry, including distances to road blocks. These modules are not used directly for control in the reported deployment, but they expose “what the model sees” and help diagnose whether the policy is using relevant cues (Aoki et al., 2023).

Safety in the narrow control sense is usually enforced outside the learned planner. SuperDriverAI’s real vehicle uses a PID steering controller, and the desired and actual steering track closely in the reported plots. The system does not report barrier functions or MPC safety constraints in field deployment, and its demonstration is restricted to steering-only control at constant speed under supervision. DiffAD similarly relies on a standard trajectory-tracking controller, specifically the official PID controller provided by Bench2Drive, while AD-H explicitly proposes PID or MPC downstream of waypoint generation (Aoki et al., 2023, Wang et al., 15 Mar 2025, Zhang et al., 2024).

A more explicit human-facing intervention layer appears in $\mu$Drive. Rather than learning preferences implicitly, $\mu$Drive provides an event-driven DSL that lets users write persistent IF-THIS-THEN rules over triggers, conditions, and actions that directly modify planner parameters at runtime. The action space includes speed controls such as max_plan_speed(n), distance parameters such as follow_dist(n) and yield_dist(n), maneuver directives such as change_lane(e,n) and stop, and policy toggles such as r_turn_red(b) or check_traj(b). Integrated with Apollo 9.0, the parser exhibits 2.2 ms to 6.08 ms latency for a single rule with 1–10 actions or conditions, and 3.21 ms to 25.02 ms for 1–20 rules. Time-to-effect averages are 108.52 ms for speed actions, 105.36 ms for distance actions, 954.86 ms for re-planning, and 973.20 ms for lane change. In the reported scenarios, $$\mathcal{L}=\omega_1\mathcal{L}_{\rm spat}+\omega_2\mathcal{L}_{\rm drm}+\omega_3\mathcal{L}_{\rm imi}+\omega_4\mathcal{L}_{\rm dir}+\omega_5\mathcal{L}_{\rm cons},$$0Drive interventions improve compliance for Law44 by 90%, and for Law46, Law52, and Law53 by 100% (Wang et al., 2024).

Human-centered AutoDriver research also extends beyond planning logic. The ADS-DV concept for passengers with Autism Spectrum Disorder targets SAE Level 4–5 autonomy in geo-fenced operation and pairs a caregiver-facing mobile application with an interior adapted for sensory and behavioral needs. The design process identifies safety, monitoring and updates, comfort, trust, reliability, anxiety reduction, and individual preferences as central requirements. The resulting concept includes live video communication, profile-based customization, safer window treatments, and virtual rendering on windows to mask social distractors and display preferred scenery. Survey data in that study report baseline trust in AVs of $$\mathcal{L}=\omega_1\mathcal{L}_{\rm spat}+\omega_2\mathcal{L}_{\rm drm}+\omega_3\mathcal{L}_{\rm imi}+\omega_4\mathcal{L}_{\rm dir}+\omega_5\mathcal{L}_{\rm cons},$$1 ($$\mathcal{L}=\omega_1\mathcal{L}_{\rm spat}+\omega_2\mathcal{L}_{\rm drm}+\omega_3\mathcal{L}_{\rm imi}+\omega_4\mathcal{L}_{\rm dir}+\omega_5\mathcal{L}_{\rm cons},$$2) and ride frequency of $$\mathcal{L}=\omega_1\mathcal{L}_{\rm spat}+\omega_2\mathcal{L}_{\rm drm}+\omega_3\mathcal{L}_{\rm imi}+\omega_4\mathcal{L}_{\rm dir}+\omega_5\mathcal{L}_{\rm cons},$$3 rides/week ($$\mathcal{L}=\omega_1\mathcal{L}_{\rm spat}+\omega_2\mathcal{L}_{\rm drm}+\omega_3\mathcal{L}_{\rm imi}+\omega_4\mathcal{L}_{\rm dir}+\omega_5\mathcal{L}_{\rm cons},$$4) (Padmanaban et al., 2021).

For Level 3 automation, driver monitoring becomes part of the AutoDriver envelope because takeover readiness depends on anticipating secondary tasks. The AVDM dataset contributes a real-world, single-camera, 640 × 480 cabin video corpus with 17 participants, 200 minutes of RGB video, and 335,000 images across eight annotated activities. Using the 6-class label set, the reported I3D baseline achieves 100% per-class accuracy for driving, sitting_still, and reading, 83% for talking_phone, and 50% for both drinking_bottle and using_phone. The reported failure modes—phone or bottle occlusion near the face, and hands near the wheel causing confusion with driving—are precisely the sort of handover-related ambiguities that a deployment-grade AutoDriver would need to resolve (Sabry et al., 2024).

5. Platforms, datasets, and evaluation regimes

AutoDriver research is unusually dependent on the quality of its evaluation environment. DriverGym addresses this by formalizing autonomous driving as an OpenAI Gym–compatible MDP over BEV raster observations derived from more than 1000 hours of expert logged data from the Level 5 Prediction Dataset. In the reported experiments, the observation tensor is 112 × 112 × 11, combining semantic HD map channels with current and historical agent boxes. DriverGym supports both log replay and reactive learned agents, and evaluates policies with a closed-loop evaluation protocol that includes ADE, FDE, distance-to-reference, and front/side/rear collisions. Its baseline results show the brittleness of plain supervised imitation: SL reports ADE 32.4 ± 2.7 m and FDE 74.5 ± 5.5 m, while SL + perturbations improves to ADE 13.4 ± 1.4 m and FDE 25.5 ± 3.5 m. PPO reduces front and side collisions but raises rear collisions to 27 ± 5.0, indicating passive or hesitant behavior (Kothari et al., 2021).

On the hardware side, the JKU-ITS Automobile is a Toyota RAV4 hybrid SUV configured as a low-cost, low-invasiveness SAE Level 3 research platform “under certain conditions.” It integrates three Basler roof-mounted cameras, Ouster OS0/OS2 LiDARs, GNSS/IMU, wheel encoders, radar as part of the exteroceptive suite, ROS1 on Linux 20.04, and Openpilot on Comma Two for drive-by-wire through OBD-II. The paper computes aggregate sensor bandwidth as 240 MB/s, or 1.92 Gbps, motivating a 10 Gbps uplink to the processing unit. The platform is used for autonomous-vehicle research, human–AV interaction, and energy consumption studies rather than as a fully custom autonomy stack (Certad et al., 2023).

At a different scale, AutoDRIVE provides a 1:14-scale research and education ecosystem comprising Devkit, Simulator, and Testbed. The vehicle platform, “Nigel,” uses rear-wheel drive, front-wheel steering with Ackermann geometry, a Jetson Nano, wheel encoders, IMU, RPLIDAR A1, and an Indoor Positioning System based on AprilTags. The platform demonstrates autonomous parking, behavioral cloning, intersection traversal, and smart city management, thereby functioning less as a single AutoDriver architecture than as an integrated experimentation environment for SLAM, RL, imitation learning, and V2I coordination (Samak et al., 2022).

Recent benchmark papers have also shown that open-loop and closed-loop evaluation can diverge sharply. UAD reports on nuScenes under the NoAvg protocol an average L2 of 0.90 and an average collision rate of 0.19%, outperforming UniAD’s 1.03 and 0.31%, while also running at 7.2 FPS and consuming only 44.3% of UniAD’s training resources. In closed-loop CARLA Town05 Long, it reports Driving Score 71.63 and Route Completion 94.44, surpassing VAD by 41.32 points on driving score. DiffAD makes the open-/closed-loop mismatch even more explicit: it reports Bench2Drive DS 67.92 and SR 38.64, and states that its closed-loop DS/SR are best even with an average L2 trajectory error higher than UniAD and DriveAdapter, underscoring that open-loop L2 does not capture interactive decision quality (Guo et al., 2024, Wang et al., 15 Mar 2025).

6. Limitations, contested assumptions, and broader uses of the term

Several limitations recur across these systems. Vision-only policies are repeatedly acknowledged as vulnerable to occlusions, poor lighting, adverse weather, and rare events. SuperDriverAI operates with a monocular camera and steering-only actuation at constant speed in supervised tests; ADriver-I uses only front-view camera input and reports that low-quality generated frames can degrade subsequent action prediction in closed loop; AD-H remains simulator-heavy; UAD’s unsupervised proxy depends on the quality of 2D ROIs from GroundingDINO; DiffAD’s dominant failure mode in CARLA v2 is collision, at 44.54% of failures. Formal safety guarantees are generally proposed as future extensions rather than reported deployment features (Aoki et al., 2023, Jia et al., 2023, Zhang et al., 2024, Guo et al., 2024, Wang et al., 15 Mar 2025).

The term “AutoDriver” also has adjacent meanings outside the primary autonomous-driving stack. In driver evaluation, the behavioral-advantage framework models performance as $$\mathcal{L}=\omega_1\mathcal{L}_{\rm spat}+\omega_2\mathcal{L}_{\rm drm}+\omega_3\mathcal{L}_{\rm imi}+\omega_4\mathcal{L}_{\rm dir}+\omega_5\mathcal{L}_{\rm cons},$$5 and demonstrates environment-normalized ranking on 92,273 freight trips from 208 drivers (Qiu et al., 2018). In driver identification, raw CAN logs from 33 drivers enable re-identification with mean 1-vs-all accuracy rising from 0.758 at 20 s windows to 0.847 at 120 s windows without reverse-engineering CAN semantics (Graur et al., 2019), while single-turn classification on 12 turns reports average accuracies of 76.9% for two drivers and 50.1% for five drivers (Hallac et al., 2017). In motorsport simulation, an autonomous test driver based on distributional SAC beats a professional human driver at Fuji Speedway across all four reported setups, including 96.63 ± 0.02 s versus 98.38 s on the baseline setup, and predicts lap-time trends across 100 interpolated setups with MAE 0.020 s (Subosits et al., 2024). Outside road autonomy altogether, “AUTODRIVER” denotes a closed-loop LLM system for Linux kernel driver maintenance, achieving 56.4% compilation success across 55 cases (Kharlamova et al., 24 Nov 2025).

This breadth of usage suggests that AutoDriver is less a single technical object than a recurring research motif: automation of a driving-related role, whether that role is vehicle control, driver monitoring, driver identification, evaluation, simulation, or software maintenance. Within autonomous driving proper, however, the dominant trajectory is clear. The field is moving from narrowly defined end-to-end steering regressors toward systems that combine learned world modeling, language-mediated abstraction, explicit memory, user-configurable constraints, and richer evaluation under real or simulated closed-loop conditions.