Human-Guided DSAC for Autonomous Driving

• The paper introduces H-DSAC, which integrates human interventions via Proxy Value Propagation into the DSAC framework to encode expert intent and penalize risky actions.
• It employs a reward-free, distributional TD update that propagates sparse human labels into a dense return signal, ensuring uncertainty-aware and safe policy learning.
• Empirical results show that H-DSAC delivers faster convergence, enhanced sample efficiency, and reduced safety costs compared to traditional RL approaches in autonomous driving.

Human-Guided Distributional Soft Actor-Critic (H-DSAC) is a human-in-the-loop reinforcement learning method for autonomous driving that combines Distributional Soft Actor-Critic (DSAC) with Proxy Value Propagation (PVP) to achieve safe and sample-efficient policy learning in real-world environments. In the formulation presented for real-world driving, H-DSAC is explicitly described as a reward-free, active human-in-the-loop learning method whose key innovation is the construction of a distributional proxy value function that encodes human intent by assigning higher expected returns to expert demonstrations and penalizing actions that require human intervention; a related 2025 formulation develops the same human-guided DSAC core within a broader confidence-guided human-AI collaboration framework that adds shared control and confidence-based policy switching (Zeqiao et al., 7 Oct 2025, Zeqiao et al., 4 Jun 2025).

1. Problem setting and motivation

H-DSAC is motivated by the mismatch between the promise of end-to-end reinforcement learning for autonomous driving and the constraints of real road deployment. The problem statement emphasizes two critical challenges. The first is safety during training: unconstrained trial-and-error exploration can cause collisions or near-misses. The second is sample efficiency: rare but safety-critical events, including emergency braking and sudden lane changes, require enormous data to learn from scratch. Human-in-the-loop methods are introduced precisely to mitigate these issues by injecting expert corrections at key moments, thereby reducing risky exploration and improving sample efficiency (Zeqiao et al., 7 Oct 2025).

The related C-HAC formulation broadens this diagnosis by noting that both reinforcement learning and imitation learning face safe-exploration and distribution-shift challenges, while existing human-AI collaboration schemes can rely too heavily on extensive human intervention. Within that framework, the human-guided DSAC component is intended to achieve rapid and stable learning of human-guided policies with minimal human interaction, before any subsequent reward-driven enhancement is introduced (Zeqiao et al., 4 Jun 2025).

In this sense, H-DSAC addresses a specific regime of autonomous driving research: online learning under real-world safety constraints, where sparse human interventions must be converted into a dense control signal that is sufficiently informative to shape the policy away from risky regions and toward expert-like behavior.

2. Distributional soft actor-critic backbone

The algorithmic backbone of H-DSAC is DSAC, which augments soft actor-critic with a return-distribution model. In the real-world driving formulation, the policy $\pi_\phi(a \mid s)$ maximizes expected soft returns

$$J_\pi = \mathbb{E}_{(s_i,a_i)\sim \rho_\pi}\!\left[\sum_{i=0}^{\infty}\gamma^i\bigl(r_i + \alpha H(\pi(\cdot \mid s_i))\bigr)\right],$$

with the practical actor objective

$$J_\pi(\phi)=\mathbb{E}_{s\sim D,\;a\sim \pi_\phi}\bigl[Q_\theta(s,a)-\alpha \log \pi_\phi(a \mid s)\bigr].$$

Instead of estimating a scalar $Q$-value, DSAC models the return as a random variable $Z(s,a)$ with distribution $\mathcal Z(s,a)$, and applies the distributional Bellman update

$$\mathcal{T}Z(s,a)\stackrel{D}{=} r(s,a)+\gamma Z(s',a'), \quad s' \sim P(\cdot \mid s,a),\; a' \sim \pi(\cdot \mid s').$$

Critic learning minimizes a KL-based loss between the projected target distribution and the critic distribution $\mathcal Z_\theta(\cdot \mid s,a)$, while policy improvement uses the gradient of the entropy-regularized objective (Zeqiao et al., 7 Oct 2025).

The related technical exposition gives the corresponding soft-return definition

$$Z^\pi(s_t,a_t):=r_t+\gamma\sum_{i=t+1}^\infty \gamma^{\,i-(t+1)}\bigl[r_i-\alpha \log \pi(a_i \mid s_i)\bigr],$$

and writes the distributional soft Bellman operator as

$$\mathcal{T}_{\!\mathcal{D}^\pi}\,\mathcal{Z}(s,a)\overset{D}{=}r+\gamma\bigl(Z'(s',a')-\alpha \log \pi(a' \mid s')\bigr),$$

with KL-divergence minimization against the target measure. It also notes that many implementations discretize the return support in $$J_\pi = \mathbb{E}_{(s_i,a_i)\sim \rho_\pi}\!\left[\sum_{i=0}^{\infty}\gamma^i\bigl(r_i + \alpha H(\pi(\cdot \mid s_i))\bigr)\right],$$0 atoms and perform quantile-regression or Cramér-distance updates, whereas the actor maximizes the entropy-augmented expectation of $$J_\pi = \mathbb{E}_{(s_i,a_i)\sim \rho_\pi}\!\left[\sum_{i=0}^{\infty}\gamma^i\bigl(r_i + \alpha H(\pi(\cdot \mid s_i))\bigr)\right],$$1 (Zeqiao et al., 4 Jun 2025).

For H-DSAC, the significance of the DSAC backbone is that human guidance is not inserted into a scalar critic but into a learned return distribution. This makes the value model compatible with both expectation-based policy optimization and uncertainty- or confidence-oriented extensions.

3. Proxy Value Propagation and human-intention encoding

The defining mechanism of H-DSAC is Proxy Value Propagation. In the real-world formulation, whenever the human expert intervenes at state $$J_\pi = \mathbb{E}_{(s_i,a_i)\sim \rho_\pi}\!\left[\sum_{i=0}^{\infty}\gamma^i\bigl(r_i + \alpha H(\pi(\cdot \mid s_i))\bigr)\right],$$2, the agent’s attempted action $$J_\pi = \mathbb{E}_{(s_i,a_i)\sim \rho_\pi}\!\left[\sum_{i=0}^{\infty}\gamma^i\bigl(r_i + \alpha H(\pi(\cdot \mid s_i))\bigr)\right],$$3 and the expert’s corrective action $$J_\pi = \mathbb{E}_{(s_i,a_i)\sim \rho_\pi}\!\left[\sum_{i=0}^{\infty}\gamma^i\bigl(r_i + \alpha H(\pi(\cdot \mid s_i))\bigr)\right],$$4 are recorded as pairs $$J_\pi = \mathbb{E}_{(s_i,a_i)\sim \rho_\pi}\!\left[\sum_{i=0}^{\infty}\gamma^i\bigl(r_i + \alpha H(\pi(\cdot \mid s_i))\bigr)\right],$$5 in a human buffer $$J_\pi = \mathbb{E}_{(s_i,a_i)\sim \rho_\pi}\!\left[\sum_{i=0}^{\infty}\gamma^i\bigl(r_i + \alpha H(\pi(\cdot \mid s_i))\bigr)\right],$$6. The proxy value function $$J_\pi = \mathbb{E}_{(s_i,a_i)\sim \rho_\pi}\!\left[\sum_{i=0}^{\infty}\gamma^i\bigl(r_i + \alpha H(\pi(\cdot \mid s_i))\bigr)\right],$$7 then labels human actions with a Dirac at $$J_\pi = \mathbb{E}_{(s_i,a_i)\sim \rho_\pi}\!\left[\sum_{i=0}^{\infty}\gamma^i\bigl(r_i + \alpha H(\pi(\cdot \mid s_i))\bigr)\right],$$8 and novice actions at $$J_\pi = \mathbb{E}_{(s_i,a_i)\sim \rho_\pi}\!\left[\sum_{i=0}^{\infty}\gamma^i\bigl(r_i + \alpha H(\pi(\cdot \mid s_i))\bigr)\right],$$9,

$$J_\pi(\phi)=\mathbb{E}_{s\sim D,\;a\sim \pi_\phi}\bigl[Q_\theta(s,a)-\alpha \log \pi_\phi(a \mid s)\bigr].$$0

and optimizes the proxy-value loss

$$J_\pi(\phi)=\mathbb{E}_{s\sim D,\;a\sim \pi_\phi}\bigl[Q_\theta(s,a)-\alpha \log \pi_\phi(a \mid s)\bigr].$$1

When $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\;a\sim \pi_\phi}\bigl[Q_\theta(s,a)-\alpha \log \pi_\phi(a \mid s)\bigr].$$2 is modeled as Gaussian $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\;a\sim \pi_\phi}\bigl[Q_\theta(s,a)-\alpha \log \pi_\phi(a \mid s)\bigr].$$3, each KL term expands to squared-error forms in $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\;a\sim \pi_\phi}\bigl[Q_\theta(s,a)-\alpha \log \pi_\phi(a \mid s)\bigr].$$4 and variance regularizers (Zeqiao et al., 7 Oct 2025).

The same mechanism is presented in the related human-guided DSAC summary using a novice buffer $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\;a\sim \pi_\phi}\bigl[Q_\theta(s,a)-\alpha \log \pi_\phi(a \mid s)\bigr].$$5 of autonomous rollouts and a human buffer $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\;a\sim \pi_\phi}\bigl[Q_\theta(s,a)-\alpha \log \pi_\phi(a \mid s)\bigr].$$6 of interventions. There, the proxy-value loss is written for $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\;a\sim \pi_\phi}\bigl[Q_\theta(s,a)-\alpha \log \pi_\phi(a \mid s)\bigr].$$7 as

$$J_\pi(\phi)=\mathbb{E}_{s\sim D,\;a\sim \pi_\phi}\bigl[Q_\theta(s,a)-\alpha \log \pi_\phi(a \mid s)\bigr].$$8

with closed-form human and novice terms

$$J_\pi(\phi)=\mathbb{E}_{s\sim D,\;a\sim \pi_\phi}\bigl[Q_\theta(s,a)-\alpha \log \pi_\phi(a \mid s)\bigr].$$9

This formulation makes explicit that the human-chosen action is treated as “good” and the novice action as “bad,” but the signal is encoded as a return distribution rather than as a direct action-cloning target (Zeqiao et al., 4 Jun 2025).

A central property of PVP is that the sparse labels attached to intervention states are propagated to unlabeled states. In the real-world H-DSAC description, this is done via a reward-free distributional TD update over the combined buffer $Q$0:

$Q$1

where

$Q$2

The total critic loss is then

$Q$3

The C-HAC summary gives the analogous reward-free TD loss for the proxy-guided critic and reaches the same total value-distribution objective (Zeqiao et al., 7 Oct 2025, Zeqiao et al., 4 Jun 2025).

4. Learning procedure and implementation

The online procedure begins by initializing an actor, a distributional critic, target critic parameters, and separate human and novice replay buffers. At each step, the agent observes state $Q$4 and samples an action from the current actor. If the human decides to intervene, the human corrective action is executed and the tuple $Q$5 is stored in $Q$6; otherwise the agent action is executed and stored in $Q$7. The executed transition $Q$8 is stored in the combined buffer. Every $Q$9 steps, the method samples minibatches from $Z(s,a)$0 and $Z(s,a)$1, updates the critic using $Z(s,a)$2, updates the actor using

$Z(s,a)$3

updates the temperature parameter $Z(s,a)$4, and performs soft target updates for $Z(s,a)$5 and $Z(s,a)$6 (Zeqiao et al., 7 Oct 2025).

The state representation used in the real-world system includes vehicle kinematics, specifically speed, lateral offset, and heading; surrounding obstacles represented as 240-dimensional LiDAR vectors; and navigation waypoints, specifically the next 30 checkpoints and turn commands. The action output is 2-dimensional continuous control, consisting of acceleration and steering angle. The critic $Z(s,a)$7 is a two-layer MLP with 256-256 units and ReLU activations, outputting the mean $Z(s,a)$8 and log-variance so as to model a diagonal Gaussian over returns. The actor $Z(s,a)$9 is also a two-layer 256-256 ReLU MLP and outputs Gaussian policy mean and log-standard-deviation for each action dimension. Reported hyperparameters are $\mathcal Z(s,a)$0, $\mathcal Z(s,a)$1, batch size $\mathcal Z(s,a)$2, discount $\mathcal Z(s,a)$3, target smoothing $\mathcal Z(s,a)$4, initial $\mathcal Z(s,a)$5, novice buffer size $\mathcal Z(s,a)$6, and a dynamically collected human buffer of approximately $\mathcal Z(s,a)$7 in the experiments (Zeqiao et al., 7 Oct 2025).

The significance of these implementation choices lies in their emphasis on practical real-world training times. The state design is explicitly described as “well-designed,” and the method is reported to achieve real-world driving policy learning within practical training times (Zeqiao et al., 7 Oct 2025).

5. Safety properties and empirical performance

The safety rationale of H-DSAC is operational rather than purely post hoc. Human interventions override dangerous actions in real time, and PVP penalizes such actions in the value distribution, steering the policy away from risky regions. In MetaDrive simulation, the reported sample complexity is 0.05 M environment steps for H-DSAC, compared with 1 M for SAC, PPO, and DSAC, together with approximately 15 K human labels. Under that regime, H-DSAC reports episodic return $\mathcal Z(s,a)$8 versus $\mathcal Z(s,a)$9 for DSAC/SAC/PPO, safety cost $$\mathcal{T}Z(s,a)\stackrel{D}{=} r(s,a)+\gamma Z(s',a'), \quad s' \sim P(\cdot \mid s,a),\; a' \sim \pi(\cdot \mid s').$$0 versus $$\mathcal{T}Z(s,a)\stackrel{D}{=} r(s,a)+\gamma Z(s',a'), \quad s' \sim P(\cdot \mid s,a),\; a' \sim \pi(\cdot \mid s').$$1, and success rate $$\mathcal{T}Z(s,a)\stackrel{D}{=} r(s,a)+\gamma Z(s',a'), \quad s' \sim P(\cdot \mid s,a),\; a' \sim \pi(\cdot \mid s').$$2 versus $$\mathcal{T}Z(s,a)\stackrel{D}{=} r(s,a)+\gamma Z(s',a'), \quad s' \sim P(\cdot \mid s,a),\; a' \sim \pi(\cdot \mid s').$$3. The same report states that H-DSAC is human-efficient, using only 14.8 K interventions versus 33.7 K for PVP, and that removing the PVP proxy loss slows convergence by more than $$\mathcal{T}Z(s,a)\stackrel{D}{=} r(s,a)+\gamma Z(s',a'), \quad s' \sim P(\cdot \mid s,a),\; a' \sim \pi(\cdot \mid s').$$4 and increases safety cost by $$\mathcal{T}Z(s,a)\stackrel{D}{=} r(s,a)+\gamma Z(s',a'), \quad s' \sim P(\cdot \mid s,a),\; a' \sim \pi(\cdot \mid s').$$5. In real-world UGV experiments, training uses 100 K steps at a 10 Hz control loop, the takeover rate falls from $$\mathcal{T}Z(s,a)\stackrel{D}{=} r(s,a)+\gamma Z(s',a'), \quad s' \sim P(\cdot \mid s,a),\; a' \sim \pi(\cdot \mid s').$$6 to $$\mathcal{T}Z(s,a)\stackrel{D}{=} r(s,a)+\gamma Z(s',a'), \quad s' \sim P(\cdot \mid s,a),\; a' \sim \pi(\cdot \mid s').$$7 by 80 K steps, and the learned policy successfully completes both training and unseen test routes while handling turns, pedestrian yielding, obstacle avoidance, and intersections robustly. The broader safety summary in the same source also states that the takeover rate drops from about $$\mathcal{T}Z(s,a)\stackrel{D}{=} r(s,a)+\gamma Z(s',a'), \quad s' \sim P(\cdot \mid s,a),\; a' \sim \pi(\cdot \mid s').$$8 to near-zero by 80 K steps in approximately 2 hours of training (Zeqiao et al., 7 Oct 2025).

The related C-HAC experiments report the human-guided DSAC stage and the subsequent confidence-guided enhancement on the MetaDrive safety benchmark against model-free RL baselines, offline RL and imitation learning baselines, and other human-AI collaboration methods. Reported metrics over five seeds include a test return of $$\mathcal{T}Z(s,a)\stackrel{D}{=} r(s,a)+\gamma Z(s',a'), \quad s' \sim P(\cdot \mid s,a),\; a' \sim \pi(\cdot \mid s').$$9 within 1 M frames, surpassing DSAC’s $\mathcal Z_\theta(\cdot \mid s,a)$0; safety cost of $\mathcal Z_\theta(\cdot \mid s,a)$1 in total, compared with unconstrained SAC at approximately $\mathcal Z_\theta(\cdot \mid s,a)$2k and PPO at approximately $\mathcal Z_\theta(\cdot \mid s,a)$3k; and success rate of $\mathcal Z_\theta(\cdot \mid s,a)$4, compared with DSAC at $\mathcal Z_\theta(\cdot \mid s,a)$5 and PVP at $\mathcal Z_\theta(\cdot \mid s,a)$6. The same source states that using only 15 k human steps in Stage I and 950 k autonomous steps, the method outperformed offline-RL methods trained on 50 k human steps. Its ablations report that removing shared control caused return to plunge by 70%, while removing confidence caused cost to rebound by 200%. Real-world UGV experiments on two campus routes are described as showing smooth, safe human-like trajectories (Zeqiao et al., 4 Jun 2025).

Taken together, these results establish the empirical profile repeatedly associated with H-DSAC: fast safe bootstrap from sparse interventions, lower training risk than unconstrained RL, and practical convergence in both simulation and real-world driving tasks.

6. Extensions, scope, and relation to adjacent methods

A key point of scope is that H-DSAC, as presented for real-world autonomous driving, is a reward-free active human-in-the-loop method. The proxy value function does not encode true environmental rewards; instead, it encodes human intent by assigning higher expected returns to expert demonstrations and penalizing actions that require human intervention. The actor is nevertheless trained with the DSAC-style entropy-regularized objective, using the expectation of the proxy-shaped return distribution as its optimization target (Zeqiao et al., 7 Oct 2025).

The C-HAC formulation extends this human-guided core by introducing a self-learning policy $\mathcal Z_\theta(\cdot \mid s,a)$7 after the human-guided policy $\mathcal Z_\theta(\cdot \mid s,a)$8 has converged to a safe base driving style. The behavior policy is then blended as

$\mathcal Z_\theta(\cdot \mid s,a)$9

where $$Z^\pi(s_t,a_t):=r_t+\gamma\sum_{i=t+1}^\infty \gamma^{\,i-(t+1)}\bigl[r_i-\alpha \log \pi(a_i \mid s_i)\bigr],$$0 is a confidence-based intervention function. Confidence is computed from a second distribution network trained on real rewards, using the probability

$$Z^\pi(s_t,a_t):=r_t+\gamma\sum_{i=t+1}^\infty \gamma^{\,i-(t+1)}\bigl[r_i-\alpha \log \pi(a_i \mid s_i)\bigr],$$1

and intervention is triggered when confidence that $$Z^\pi(s_t,a_t):=r_t+\gamma\sum_{i=t+1}^\infty \gamma^{\,i-(t+1)}\bigl[r_i-\alpha \log \pi(a_i \mid s_i)\bigr],$$2 outperforms $$Z^\pi(s_t,a_t):=r_t+\gamma\sum_{i=t+1}^\infty \gamma^{\,i-(t+1)}\bigl[r_i-\alpha \log \pi(a_i \mid s_i)\bigr],$$3 is insufficient. This extension preserves the human-guided DSAC mechanism but places it within a two-stage architecture consisting of human-guided learning followed by RL continuous enhancement (Zeqiao et al., 4 Jun 2025).

A recurrent misconception is to treat H-DSAC as straightforward behavior cloning from corrective demonstrations. The published formulations do not support that simplification. Human interventions are indeed sparse labels, but they are embedded in a distributional critic, optimized through KL objectives, and propagated across unlabeled states by a reward-free distributional TD update over the combined replay buffer (Zeqiao et al., 7 Oct 2025). Another potential source of confusion is terminological: the 2025 literature uses both “H-DSAC” and the broader “C-HAC” label. The former refers to the human-guided DSAC core built from DSAC and distributional proxy-value propagation; the latter denotes a larger confidence-guided human-AI collaboration strategy that incorporates the H-DSAC stage and then adds shared control and dynamic switching (Zeqiao et al., 4 Jun 2025).

This suggests that H-DSAC occupies a hybrid position between corrective human guidance, distributional value learning, and entropy-regularized actor-critic optimization. A plausible implication is that its principal methodological contribution is not merely the use of interventions, but the conversion of sparse interventions into a dense return-distribution signal that can support both safe bootstrap and, in the extended framework, later autonomous improvement.