Beyond Safety Filtering: Control Barrier Function-Informed Reinforcement Learning for Connected and Automated Vehicles

Abstract: Reinforcement Learning (RL) uses rewards to guide learning, yet reward design is typically hand-crafted using heuristics that can be difficult to tune. We propose a Control Barrier Function (CBF)-informed reward design for Multi-Agent RL (MARL) that converts CBF constraint values under joint MARL actions into a reward signal that explicitly guides safe learning. We compare against two heuristic reward baselines in a four-way multi-lane intersection with connected and automated vehicles. Results show that our method achieves the highest task performance and is less sensitive to reward hyperparameters, yielding consistently strong performance across the tested hyperparameter range. Code for reproducing the experimental results and a video demonstration are available at https://github.com/bassamlab/SigmaRL.

• The paper introduces a CBF-informed reward framework that formally integrates safety constraints into multi-agent RL for intersection navigation.
• It demonstrates superior performance with higher mean rewards, reduced intervention rates, and enhanced robustness to hyperparameter variations compared to heuristic baselines.
• The approach yields coordinated and adaptive vehicle behaviors, offering a scalable path toward safe and efficient autonomous driving in complex urban settings.

Control Barrier Function-Informed Reinforcement Learning for Connected and Automated Vehicles

Introduction

This work addresses a fundamental challenge in Reinforcement Learning (RL) for Connected and Automated Vehicles (CAVs): effective, robust, and safe reward specification in complex multi-agent urban scenarios. Traditional RL reward engineering relies heavily on hand-crafted heuristics, which are error-prone, labor-intensive, and often misalign with formal safety requirements. The paper introduces a Control Barrier Function (CBF)-informed reward framework that leverages explicit, formal safety constraints to guide Multi-Agent RL (MARL), particularly in intersection navigation tasks.

The method constructs reward signals directly from CBF constraint satisfaction under joint actions, shifting the paradigm from safety filtering via online constraint modification to implicit policy-level safety guidance. The evaluation is performed in a multi-lane, four-way intersection, comparing against two standard heuristic baselines: distance-based and time-to-collision (TTC)-based reward shaping. The results demonstrate superior task performance, enhanced robustness to hyperparameter choices, and reduced reliance on online safety filtering.

Figure 1: Four-way multi-lane intersection workspace showing entry/exit regions, reference paths, drivable corridors, and road boundaries.

CBF-Integrated Reward Shaping in MARL

Problem Formulation

The study formulates intersection navigation as a multi-agent control problem:

• Agents: $N$ CAVs instantiate their actions in a shared planar intersection.
• Observations/Actions: Each vehicle follows a reference path and selects control inputs subject to kinematic bicycle model dynamics.
• Objective: The joint policy maximizes a sum of event-driven rewards (successful exits, collision penalties) and minimizes discomfort (acceleration and jerk), with explicit formalization for safety-valued events.

Control Barrier Function Construction

CBFs define forward-invariant “safe sets” in continuous state space. For each CAV, the method approximates rectangular vehicle envelopes with unions of circles and constructs:

• Road Boundary CBFs: Enforce safe clearance from polyline-based road boundaries.
• Inter-vehicle CBFs: Ensure pairwise safety between every vehicle pair using minimum distances between occupancy-approximating circles.

The paper applies a Truncated Taylor CBF (TTCBF) construction, with CBF constraints parameterized for use in sampled-data MARL settings. Each CBF condition yields a scalar constraint, evaluated on proposed agent actions.

CBF-Informed Reward Signal

CBF constraint values under joint agent actions are clipped and mapped to penalties via a piecewise-linear function. The aggregate per-agent CBF reward consists of:

$$r^{i}_{\mathrm{cbf},k} = \frac{1}{3} (r^{i}_{\mathrm{veh},k} + r^{i,\mathrm{L}}_{\mathrm{road},k} + r^{i,\mathrm{R}}_{\mathrm{road},k})$$

Combined with a forward progress reward, the overall MARL reward directly incorporates safety constraint satisfaction at the policy level. This is in contrast to prior approaches, which (i) shape rewards heuristically without formal safety linkage, or (ii) filter actions via CBFs post-hoc (which is computationally intractable at intersection scale).

Figure 2: Training reward curve for the CBF-informed method demonstrates stable and monotonic learning compared to heuristic baselines.

Experimental Evaluation

Baselines and Environment

Two reward shaping baselines are chosen:

• Baseline Distance: Penalizes proximity to road boundaries and other vehicles linearly, based on hand-chosen safety thresholds.
• Baseline TTC: Penalizes small time-to-collision estimates under constant velocity assumptions.

Policies are trained using multi-agent Proximal Policy Optimization (MAPPO) with identical architectures; only the reward signal varies.

Numerical Results

Key performance metrics are:

• Mean Total Reward: CBF-rewarded MARL attains $+7.2$ on average, compared to $-2.8$ and $-2.3$ for Distance and TTC, respectively. Under optimal hyperparameters, the CBF achieves $+8.5$ versus $+4.5$ (Distance) and $+7.7$ (TTC).
• Reward Sensitivity: Standard deviation of total reward across the tested hyperparameter range is $1.1$, a $66\%$–$$r^{i}_{\mathrm{cbf},k} = \frac{1}{3} (r^{i}_{\mathrm{veh},k} + r^{i,\mathrm{L}}_{\mathrm{road},k} + r^{i,\mathrm{R}}_{\mathrm{road},k})$$0 sensitivity reduction compared to baselines.
• CBF Activation Degree: The fraction of time a posteriori CBF-based safety filtering would need to be invoked. The CBF-informed agent requires $$r^{i}_{\mathrm{cbf},k} = \frac{1}{3} (r^{i}_{\mathrm{veh},k} + r^{i,\mathrm{L}}_{\mathrm{road},k} + r^{i,\mathrm{R}}_{\mathrm{road},k})$$1 intervention, versus $$r^{i}_{\mathrm{cbf},k} = \frac{1}{3} (r^{i}_{\mathrm{veh},k} + r^{i,\mathrm{L}}_{\mathrm{road},k} + r^{i,\mathrm{R}}_{\mathrm{road},k})$$2 (Distance) and $$r^{i}_{\mathrm{cbf},k} = \frac{1}{3} (r^{i}_{\mathrm{veh},k} + r^{i,\mathrm{L}}_{\mathrm{road},k} + r^{i,\mathrm{R}}_{\mathrm{road},k})$$3 (TTC), with the best policy reducing intervention by up to $$r^{i}_{\mathrm{cbf},k} = \frac{1}{3} (r^{i}_{\mathrm{veh},k} + r^{i,\mathrm{L}}_{\mathrm{road},k} + r^{i,\mathrm{R}}_{\mathrm{road},k})$$4 compared to hand-crafted shaping.

Figure 3: Total reward profiles for CBF, Distance, and TTC reward methods over their respective hyperparameter sweeps.

Figure 4: CBF activation degree illustrating the reliance of learned policies on online safety filtering across all reward methods and hyperparameters.

Behavioral Analysis

Qualitative analysis of agent behaviors in test scenarios reveals coordinated yielding, order negotiation, and lane discipline. Rather than maximizing only self-progress, CBF-informed agents adaptively yield at conflict points—manifesting globally safer, more cooperative crossings.

Figure 5: Vehicle trajectories and footprints under the CBF-informed policy; yielding and coordinated conflict region traversal are evident.

Implications and Future Directions

The CBF-informed reward framework represents a significant step toward integrating formal safety certificate logic with RL in high-dimensional, multi-agent CAV systems. By translating structured safety constraints into reward signals, one obtains (i) consistent and robust safety guidance, (ii) improved generalization and hyperparameter insensitivity, and (iii) markedly decreased dependence on costly online safety filtering, even in highly coupled domains like intersections.

Practical implications include:

• Deployment: Enhanced safety reliability for RL-driven CAVs in settings where symbolic planners are infeasible.
• Scalability: Decreases the complexity of online filtering, which is critical in dense, interactive traffic scenarios.
• Generalization: Constructed CBFs are extensible to arbitrary vehicle shapes and high-order systems, paving the way for broader adoption in diverse robotic platforms.

Theoretical questions remain on the integration of CBF-based rewards with function approximation and off-policy RL stability. Additionally, adaptive or learning-based CBF construction remains an open research avenue, particularly for mixed-autonomy systems and scenarios with imperfect models or partial observability.

Conclusion

This paper makes a compelling case for CBF-informed reward shaping in MARL for CAVs at intersections. The method achieves state-of-the-art task performance, robust safety guidance, and minimal intervention requirements compared to well-established heuristic methods. The proposed approach marks a pivotal convergence of formal control-theoretic safety and deep multi-agent learning, underscoring new directions in safe, efficient, and scalable autonomous driving policy development.

(2605.16894)