SECRM-2D Autonomous Driving Controller

• SECRM-2D is a reinforcement learning–based autonomous driving controller that rigorously enforces analytic safety constraints to ensure safe route-following in dynamic multi-lane environments.
• It integrates a state-wise constrained Markov decision process with continuous longitudinal and discrete lateral actions using DDPG, balancing efficiency, comfort, and strict safety requirements.
• Analytic safety criteria, based on international road safety norms, guarantee zero crash rates and stable vehicle platooning as demonstrated across complex simulated driving scenarios.

SECRM-2D is a reinforcement learning–based autonomous driving controller framework for efficient, comfortable, and safe route-following in multi-lane traffic environments. Designed to optimize longitudinal and lateral actions with strict analytic safety guarantees, SECRM-2D explicitly enforces headway criteria derived from international road safety conventions and is evaluated against classical and RL-based baselines in a variety of complex simulated driving scenarios (Shi et al., 2024).

1. Problem Formulation and Objectives

SECRM-2D addresses the control of an “ego” vehicle on a multi-lane road, required to follow a fixed route defined by sections and discrete lanes, in the presence of other vehicles. Control occurs at discrete time intervals $r$ (typically $r=0.1\,\text{s}$), with the controller selecting longitudinal acceleration %%%%2%%%% in $[-d_E,a_E]$ and a discrete lateral action (stay, shift left, shift right). The formulation seeks to:

• Maximize efficiency: by approaching the target average speed within local speed and leader-following constraints.
• Maximize comfort: via penalization of jerk, i.e., the rate of change of acceleration.
• Guarantee analytically-derived safety: specifically the Vienna Convention criterion that the follower can stop safely if the leader brakes suddenly.
• Faithfully follow the prescribed route: encompassing both discretionary overtaking and mandatory exits/merges.

Trade-offs arise between efficiency, comfort, and safety: maximizing speed or aggressive maneuvers can degrade comfort or violate safety, while strict safety limits may restrict optimality. SECRM-2D seeks Pareto-optimality under hard safety constraints.

2. State-wise Constrained MDP Formulation

The control problem is structured as a constrained Markov decision process (MDP) $(S,A,P,R,\gamma,C,w)$:

• State space ($S$): Observations include ego position, lane index, speed ($v_E$), prior acceleration ($a_E$), lateral velocity, route flags per lane, and remaining section distance. For each lane within scanning radius, offsets and states of up to $N_\text{front}$ vehicles ahead and $N_\text{back}$ behind are sampled.
• Action space ($A$): Actions are tuples $(\alpha, \ell)$, with $\alpha$ (continuous within safety bounds) and $\ell \in \{\text{left},\text{cur},\text{right}\}$.
• Transition model ($P$): Deterministic kinematics for ego, stochastic updates using SUMO micro-simulator (Krauss CF/SL2015 LC) for other vehicles.
• Reward ($R$): Aggregates efficiency, comfort, discretionary lane-change, and mandatory lane-change followership:

$$r_t = w_\text{eff}\,R_\text{eff}(s_t,a_t) + w_\text{comf}\,R_\text{comf}(s_t,a_t) + w_\text{dlc}\,R_\text{dlc}(s_t,a_t) + w_\text{mlc}\,R_\text{mlc}(s_t)$$

Where efficiency penalizes deviation from safe speed, comfort penalizes jerk:

$$R_\text{comf}(s,a,s') = -\left( \frac{a_E(s')-a_E(s)}{(a_E+d_E)/r} \right)^2$$

and lane-change terms reward or penalize as per route adherence, with $R_\text{dlc}$ and $R_\text{mlc}$ formulated according to geometric and route proximity.

• Constraints ($C$): Encoded as zero-one costs requiring $C_i \leq 0$; longitudinal and lateral constraints are analytically defined below.

3. Analytic Safety Constraints

Safety in SECRM-2D is enforced via closed-form, kinematic headway criteria. The fundamental longitudinal safety rule, derived from the requirement that a follower must maintain sufficient gap to preclude collision if the leader brakes maximally, is:

$$g(t) \geq \frac{v_E(t)+v_E(t+1)}{2}\;r + \frac{v_E^2(t+1)}{2d_E} - \frac{v_L^2(t)}{2d_L} + \epsilon \tag{1}$$

From this, the maximum safe next-step ego speed $s(t+1)$ is computed as:

$$s(t+1) = -\frac{d_E r}{2} + \sqrt{\left(\frac{d_E r}{2}\right)^2 - 2 d_E\left( \frac{r v_E(t)}{2} - \frac{v_L^2(t)}{2} - g(t) + \epsilon \right) } \tag{2}$$

Control is restricted such that $v_E(t+1) \le s(t+1)$, or equivalently:

$a_E(t) \leq \frac{s(t+1) - v_E(t)}{r} \tag{3}$

Lane-changing safety is enforced by recalculating these gap criteria for the hypothesized post-change leader and follower; a change is allowed only if both satisfy the 2-vehicle safe stopping inequalities.

4. RL Algorithm and Safety Enforcement Approach

SECRM-2D utilizes the Deep Deterministic Policy Gradient (DDPG) RL algorithm, with a three-layer multilayer perceptron (MLP) for both the actor policy $\pi_\phi(s)$ and critic $Q_\theta(s,a)$, each with 256 ReLU units and $\tanh$-bounded outputs mapped to $[-3,3]^2$ for action encoding. Actions are mapped to $(\alpha,\ell)$ via affine rescaling and discrete partitioning.

A critical feature is hard constraint enforcement: safety terms are excluded from the reward function. Actor outputs are filtered rigorously so that all analytic constraints (Eqs. (3), lane-change inequalities) are satisfied at every step—violating actions are either suppressed with zero/minuscule negative cost and excluded from transition or reset to “cur” in the event of an invalid lane-change. Replay buffers further reduce the propensity to select unsafe actions during learning.

Training uses OU noise for exploration, 1,000 warm-up steps, and best-checkpoint selection across $\sim5,000$ episodes with randomization of speed limits to promote generalization.

5. Lane-Changing Design and Route Adherence

Lateral actions are modeled with a discrete choice among $\{\text{left},\text{cur},\text{right}\}$. Discretionary lane-changes, motivated by bypass efficiency, are rewarded using $R_\text{dlc}$, proportional to the target speed advantage of the adjacent lane. Mandatory lane-changes for route-following (e.g., exits, merges) incur penalties proportional to the minimum lane hops to the target route lane and inversely with the remaining distance to section end.

All lane-change proposals undergo identical analytic safe-gap checks for longitudinal and adjacent lane safety. Unsafe lane-changes are explicitly disallowed, ensuring strict adherence to the Vienna-derived safety conventions.

6. Experimental Protocol and Performance Metrics

Simulation is executed in SUMO via TraCI at $0.1$ s timestep. SECRM-2D is evaluated in two primary environments:

• Loop network: Circular topology for discretionary overtaking tests.
• QEW interchange: Real-world geometry for mandatory exit and merging.

External traffic is modeled by SUMO’s Krauss car-following and SL2015 lane-change algorithms. Baselines include IDM+MOBIL, Gipps+Greedy, PPO-RL, mandatory-LC planner, highway-exit RL, and adversarial RL merging controllers.

Test scenarios (each with 30 randomized seeds) assess normal/heavy traffic, emergency braking response, zig-zag bypassing, forced braking corridors, and on-ramp merges. Metrics tracked are crash rate, average speed (efficiency), average jerk (comfort), route-miss rate, and merge-fail rate.

7. Quantitative Results and Steady-State Analysis

SECRM-2D consistently attains zero crash rate in all evaluated scenarios:

• Loop, normal: Speed $\approx32.45$ m/s, jerk $\approx0.86$ m/s$^3$, crash 0%
• Loop, heavy: Speed $26.32$ m/s, jerk $0.02$, crash 0%
• Emergency braking: Speed $31.73$ m/s, jerk $0.89$, crash 0% (PPO-RL: 21% crash)
• QEW bypassing ($H=5,10,20$ m): Crash 0%, route miss $\le15\%$, speed $20$–$23$ m/s, jerk $1.08$–$1.13$; other RL/baselines crash 12–24%
• QEW on-ramp merging: Speed $22.53$ m/s, jerk $1.21$, crash 0%, merge-fail $13\%$; baselines crash 11–23%, fail 17–33%
• Reaction time $r_E$ sensitivity: speed drops from $21.3$ to $19.2$ m/s ($r=0.1$–$1$ s), crash remains 0%

Steady-state theoretical analysis shows that a platoon of SECRM-2D vehicles converges to a unique, asymptotically stable equilibrium $(g^*,v^*)$ with constant speed $w$ and gap $g^* = w r + \epsilon$. This stability and headway behavior are confirmed in 4-vehicle platoon simulations, indicating that SECRM-2D supports robust, stable multi-agent following.

8. Context, Significance, and Implications

SECRM-2D demonstrates state-wise hard safety constraint enforcement within RL-based autonomous driving, achieving empirical and analytic guarantees absent in prior RL car-following and lane-change controllers. The methodology confirms that minimizing crash propensity is not achievable by reward shaping alone; explicit analytic filtering is necessary for deployment-grade safety. Performance is on par or superior to classical and RL-based baselines with respect to efficiency and comfort, without trade-off on safety or route following.

A plausible implication is that future RL-based autonomous driving controllers should separate control rewards from safety enforcement, leveraging analytic constraint frameworks aligned with formal conventions. This suggests directions for scaling to mixed autonomy and coupling with more granular traffic norm adherence.

For in-depth mathematical derivation, implementation, and broader context, see (Shi et al., 2024).