---
title: On-Board Driving Stack for Autonomous Vehicles
url: https://www.emergentmind.com/topics/on-board-driving-stack
type: topic
---

# On-Board Driving Stack for Autonomous Vehicles

An on-board driving stack refers to the complete integrated suite of sensing, perception, localization, prediction, planning, and control software and hardware deployed within an autonomous vehicle to enable real-time, closed-loop driving without reliance on off-board computation. Such stacks are responsible for real-time acquisition and fusion of multi-modal sensor data, semantic scene understanding, local state estimation, trajectory planning with hard safety guarantees, low-latency actuation, and increasingly, high-level reasoning via vision-language or language models—all operating under the compute, bandwidth, and safety constraints of deployment on production or research vehicles.

## 1. System Architecture and Module Composition

On-board stacks are universally architected as a real-time, modular sense–plan–act pipeline integrating the following primary subsystems:

- **Sensing and Time-Synchronization:** Sensor suites typically comprise multi-camera arrays, 3D LiDAR, radar, GNSS-RTK, IMU, wheel encoders, and proprietary vehicle state sensors. Hardware platforms commonly feature dedicated real-time controllers (e.g., dSPACE MicroAutoBox II), GPU-accelerated compute units (e.g., NVIDIA Jetson Orin AGX), or COTS industrial PCs [2408.06113][1905.02980][2308.02176].
- **Perception and Scene Understanding:** Fusion of vision (YOLO, Faster R-CNN, ViT, etc.), LiDAR (point cloud ground-plane removal, clustering), and radar is implemented to localize static/dynamic obstacles and track semantics (cones, vehicles, pedestrians). Recent architectures leverage deep learning models (YOLOv5s/TensorRT, RekTNet, ByteTrack) and multiple depth-estimation pipelines, achieving mean depth errors as low as 0.85% via LiDAR–camera fusion [2408.06113][2210.10933][2601.12373][2411.11913].
- **Localization and State Estimation:** Techniques range from GNSS/IMU-based EKF or ESKF, graph-based SLAM (g2o, MRPT, Cartographer, iSAM2), to pose-graph optimizers, frequently operating in parallel with robust outlier rejection. State representations often couple ego pose/velocity and mapped landmarks: $x = [x_v, y_v, \psi_v, v, m_1, ..., m_N]$ [2408.06113][2210.10933][2509.19636][2404.02645].
- **Prediction:** Leading stacks employ interpretable, goal- and intent-based predictors (IGP2, GOFI), tree-ensembles (GRIT), and GNNs for multi-modal distributional forecasting. Bayesian posteriors over agent goals and scene occlusion probabilities inform the planning module, supporting safety by design [2208.00096][2404.02645].
- **Planning:** Trajectory or path planners leverage curvature or comfort cost minimization, typically as finite-horizon nonlinear optimal control, hybrid QP/NLP stages (2s-OPT), relaxation-based QP solvers, or particle-based optimizers (PSO). Safety constraints are encoded as hard boundaries over reachable state-sets and STL logic [2408.06113][2208.00096][2404.02645][2210.10933][2509.19636].
- **Control:** Lateral and longitudinal controls range from kinematic bicycle models, Stanley controller, Pure Pursuit, LTV-MPC, optimal control allocation (torque vectoring), to model-based PID/PI/PID+MPC cascades. Delay compensation and distributed over actuation channels (CAN, dSPACE CAN, Raptor DBW) are required for robust performance [2408.06113][2210.10933][2509.19636][1905.02980].
- **Safety and Diagnostics:** Supervisory modules explicitly monitor health/faults, maintain time budgets, enforce fallbacks, and publish diagnostic status. These routines intercept violations of latency, staleness, actuation limits, or anomalous localization [2404.02645][2308.02176][2509.19636].

Typical inter-module communication is implemented in ROS 1/2 (DDS or Fast-RTPS middleware) with type-safe topics and rigorous time-stamping. Cross-domain CAN/Ethernet bridges and UDP/MQTT relays support hardware interface and V2X scenarios [2408.06113][2308.02176][2601.12373][1905.02980].

## 2. Sensing, Perception, and Localization Pipelines

Modern stacks implement multi-tiered fusion and parallelized perception, combining:

- **Deep Learning Detectors:** YOLOv5s on NVIDIA TensorRT achieves mAP≈0.985 for cone detection. ByteTrack and YOLOv8-n are used for real-time object/subclass detection with <20ms latencies per frame at up to 50Hz [2408.06113][2601.12373].
- **Depth Estimation:** Pipelines include LiDAR–camera fusion (ground-plane removal by RANSAC, DBSCAN clustering, projective fusion for 0.85% avg. error), monocular bounding-box height power-law fits (4.49% error), and stereo keypoint-based triangulation by RekTNet and SIFT (6.39%) [2408.06113]. RAFT-Stereo, DeepLabV3, and classical triangulation are also employed [2601.12373].
- **SLAM/State Estimation:** Feature-based EKF-SLAM or factor-graph SLAM enables vehicle pose and map estimation $x = [x_v, y_v, \psi_v, v, m_1..., m_N]$, with motion/measurement updates at 100Hz/30Hz, and $O(n^2)$ data association in parallelized background threads, yielding <0.2m RMS error [2408.06113][2210.10933]. RMS error of <0.03m (GNSS RTK) or <0.2m (pose-graph) is attained for urban deployments [2404.02645].
- **Real-Time Synchronization:** All sensors disciplined to GPS-PPS, with ROS message_filters ApproximateTime policy (±5ms thresholds), and dedicated buffer nodes to prevent sensor backlog [2408.06113].

Depth, object class, velocity, yaw, and safety metrics such as time-to-collision (TTC) and time-headway (THW) are computed per detection and serialized using custom ROS2 messages for distributed multi-agent or infrastructure interaction [2601.12373].

## 3. Planning and Control Methodologies

The planning subsystem is typically formalized as:

\[
\min_{\text{path } \Gamma} \int_{s=0}^{L} \kappa(s)^2 ds
\]
where $\kappa(s)$ is path curvature, $L$ is path length, subject to static/dynamic safety constraints [2408.06113][2210.10933][2509.19636]. For urban/complex domains, complete nonlinear finite-horizon optimal control problems are posed, with MILP warm-start (2s-OPT) and NLP refinement [2208.00096]. For racing, Delaunay triangulation and minimum-curvature cubic splines define racelines, velocity profiles are shaped by G–G or GGS diagrams and aerodynamic models [2210.10933][2408.06113][2509.19636]. Adaptive sector scaling and overtake planners (Frenet or local spline-based) are applied in head-to-head racing [2403.11784].

Control architecture:

- **Lateral Control:** Stanley/Pure Pursuit (with $\delta = \psi_{\rm err} + \arctan(k_e e / v)$ or $\delta = \arctan(2L\sin\alpha/d_\mathrm{LA})$), LTV-MPC on bicycle model ($x=[y,v_y,\psi,\dot\psi]$, input $u=\delta$), MAP/L1 pursuit for scaled platforms [2408.06113][2210.10933][2403.11784].
- **Longitudinal Control:** 2-DOF PI/PID (e.g., $u_\mathrm{throttle}=K_p e_v+K_i\int e_v\,dt$), with optional feed-forward term from trajectory [2408.06113][2210.10933][2509.19636].
- **Low-Level Control Allocation:** Quadratic programs for four-wheel torque vectoring, active yaw stabilization [2210.10933].
- **Real-Time Budgeting:** Module latencies: YOLOv5s ≈12 ms (GPU), SLAM prop. 0.8 ms, measurement upd. ≈15 ms for 30 landmarks, control loops ≤1 ms, CAN round-trip ≈3 ms [2408.06113].

Switching between controllers (Stanley/Pure Pursuit) can reduce cross-track error (from 0.33m to 0.27m), with performance monitored during simulation and full-scale robot operation [2408.06113]. Competitive stacks maintain ≥1.5× emergency stopping distance at all times [2404.02645].

## 4. Integration of High-Level Reasoning and Personalization via Deep Language/Vision Models

Recent architectures extend the stack through on-board deployment of compressed large language models (LLMs) and vision-language models (VLMs):

- **Motion Control Personalization:** VLMs (e.g., Qwen-VL, ~9B parameters, INT4 quantization) receive image, natural language instructions, and system/user context plus retrieved RAG memory, producing a $2 \times 3$ action matrix that parametrizes longitudinal PID and lateral MPC controllers. The system learns from human feedback, updating memory via Chroma DB to fuse previous scenarios and user ratings. End-to-end stack achieves $\approx$1.6 s VLM inference latency on RTX-A4000, with customization lowering takeover rates by up to 76.9% in human-in-the-loop Level 3 trials [2411.11913].
- **Knowledge-Driven Adaptive Control:** LLM-based DecisionxLLM and MPCxLLM modules operate as slow periodic (0.2–0.3Hz) or on-demand controllers, translating high-level behavioral intent and state summaries into dynamic reparametrization of MPC weights/bounds via prompt-based interaction (LoRA + RAG + Q5 quantization; up to 52.2% improvement in control adaptability, 10× computational speedup compared to FP16). The MPC always enforces hard constraints; the LLM never overrides feasibility [2504.11514].
- **HMI and Safety:** Human-comprehensible language prompts, continual learning from interaction, and integration of safety fallbacks (e.g., revert to conservative PID if predicted TTC < 1.5s) characterize the state of the art [2411.11913][2504.11514].

## 5. Communication, Modularity, and Real-Time Guarantees

Stacks universally rely on modular software infrastructure:

- **Middleware:** ROS/ROS2 (DDS/Fast-RTPS) for intra-vehicle, CAN bus (up to CAN FD), high-speed Ethernet (VLAN segmented for criticality separation), and V2I via UDP/MQTT/ASN.1 [2308.02176][2408.06113][2601.12373].
- **Time Synchronization:** GPS-PPS, broadcasting time-stamped messages. Buffering camera/point-cloud data to match real-time pipeline clearance [2408.06113].
- **Node Modularity:** Each functional subsystem operates as a stateless ROS node or lifecycle container with update/activate/deactivate semantics, facilitating hot-swapping, rapid edge-case testing, and real-to-sim/sim-to-real transitions [2404.02645][1905.02980].
- **System Integration:** Full system latencies as low as 49.7ms (end-to-end control), control loops at 10–100Hz, node-level robustness enforced by heartbeat timeouts, hardware modularity (plug-and-play sensor frames), and resilience to node failures [2308.02176][2404.02645][2509.19636].

Most stacks adopt rigorous performance instrumentation and diagnostic logging, with formal diagnostic rules triggering safety stops or failover in the event of fault [2404.02645][2308.02176][2601.12373]. This ensures real-world safety and makes them suitable for Level 4–proof urban and racing deployments spanning >3,000 km of autonomy [2404.02645].

## 6. Experimental Performance and Lessons Learned

Quantitative results for competitive stacks:

| Task                | Depth Error (%) | Localization RMS (m) | Cross-Track Error (m) | Takeover Rate Reduction (%) | End-to-End Latency (ms) |
|---------------------|:--------------:|:--------------------:|:---------------------:|:--------------------------:|:-----------------------:|
| Perception (fusion) |    0.85–6.4    |       0.03–0.2       |      0.10–0.33        |           76.9             |        15–70            |
| Planning (PSO/NLP)  |      N/A       |         N/A          |        N/A            |           N/A              |       50–180            |
| Control (PID/MPC)   |      N/A       |         N/A          |        0.10           |           N/A              |        1–50             |
| LLM/VLM inference   |      N/A       |         N/A          |        N/A            |       +52.2 adaptability   |        900–1980         |

- Formula Student-AI, Indy Autonomous Challenge, and F1TENTH race teams achieve state-of-the-art by focusing on highly modular, minimal-latency, safety-verified subsystems and rapid iteration. Real-time reporting, code modularity, and system introspection tools are universally recommended for reproducible and safe deployment [2408.06113][2210.10933][2509.19636][2403.11784].
- Integration of knowledge-driven and learning-based modules facilitates robust handling of rare edge cases and individualized user preferences while maintaining deterministic control guarantees [2411.11913][2504.11514].

Key guidelines arising from empirical experience include early end-to-end instrumentation, mixed-criticality partitioning (e.g., safety-critical controllers physically separated from soft-real-time pipeline), and progressive hardware/software-in-the-loop testing [1905.02980][2404.02645].

## 7. Trends and Research Directions

- **Edge Personalization:** Wide adoption of vision-language and language models (VLM, LLM) on the vehicle compute edge, with quantization (INT4/INT5), LoRA, and RAG techniques to enable real-time, memory-efficient customization and continual adaptation [2411.11913][2504.11514].
- **Safety-By-Design and Verification:** Formal verification of tree-based predictors (e.g., GRIT/SMT invariants), scenario-based ODD parameterization, and dual high/low-fidelity simulation loops characterize modern validation methods [2208.00096][2404.02645].
- **Open, Modular Toolchains:** Standardization in ROS 2 message contracts, hardware abstraction, and modularization accelerates cross-platform support, facilitating adaptation from high-speed racing to urban shuttles and scaled platforms [2404.02645][2403.11784].
- **Human-in-the-Loop and Edge Reasoning:** RAG-based interaction histories and prompt-driven dynamic control allow true human–machine co-adaptation and explainability, reducing intervention rates and enabling new interaction paradigms [2411.11913][2504.11514].

A plausible implication is that future on-board stacks will increasingly integrate formally verified learning-based reasoning modules, dynamic reconfiguration, and modular hardware/software to maximize both safety and adaptability across a growing diversity of vehicles and operational domains.

Source: https://www.emergentmind.com/topics/on-board-driving-stack