Diffusion Policy & Hybrid Action Spaces

• The topic defines diffusion policies as methods that integrate generative diffusion models with hybrid action spaces to robustly sample multimodal trajectories in control tasks.
• It explains how supervised learning and reinforcement optimization jointly refine both discrete decisions and continuous parameters, yielding improved performance metrics.
• Applications in autonomous driving and manipulation demonstrate that these methods enhance robustness, generalization, and real-time feasibility through stochastic denoising and hybrid policy integration.

Diffusion Policy methods represent an emerging approach for sequential decision-making and control, notably in complex domains with multimodal behavior and structured action requirements. Particularly, the integration of diffusion models within hybrid action spaces—comprising both continuous and discrete components—has become central to advances in autonomous driving and manipulation. These approaches leverage the expressive generative capacity of diffusion models and the explicit supervision or reinforcement-learning optimization of control signals, yielding policies that are robust, diverse, and generalizable.

1. Hybrid Action Space Construction

A hybrid action space consists of discrete and continuous sub-spaces, enabling policies to model both categorical choices and parametric behaviors. DiffE2E (Zhao et al., 26 May 2025) conceptualizes actions as a composite of: (a) a diffusion-modeled latent trajectory $z_{\rm diff}$, mapping to a future path $\bm x_0 \in \mathbb{R}^{\ell_k \times d_c}$; and (b) a supervised latent vector $z_{\rm sup}$ of length $\ell_s$, used for explicit predictions such as speed state or control signals (steering, throttle, brake). The action input at each diffusion iteration is constructed as

$$\mathcal{Z}_{\rm in} = [ f_{\rm enc}(\tau_t); \mathbf{q}_0 ] + \mathbf{E}_{\rm pos}^{\mathcal{Z}} \in \mathbb{R}^{(\ell_k + \ell_s) \times d}$$

where $\tau_t$ is the current noisy trajectory and $\mathbf{q}_0$ denotes learnable queries for supervision. Outputs are partitioned:

$$\mathcal{Z}_{\rm diff} = \mathcal{Z}_{\rm out}[:\ell_k], \quad \mathcal{Z}_{\rm sup} = \mathcal{Z}_{\rm out}[\ell_k:\ell_k+\ell_s]$$

Hybrid manipulation policies (Le et al., 22 Nov 2024) further factorize the joint policy as

$$\pi_\theta(x, a^m \mid s) = \pi_{\rm loc}(x \mid s)\,\pi_m(a^m \mid s)$$

with $x$ a discrete contact index and $a^m \in \mathbb{R}^d$ continuous motion parameters.

2. Diffusion Process Formalization

Diffusion models stochastically generate continuous actions through iterative denoising, capturing the distribution over feasible future trajectories or motion parameters. Both DiffE2E and HyDo utilize Denoising Diffusion Probabilistic Models (DDPM):

Forward process:

$$q(z_t \mid z_{t-1}) = \mathcal{N}(z_t; \sqrt{1 - \beta_t}\,z_{t-1}, \beta_t I), \quad t = 1 \ldots T$$

Reverse process, parameterized by neural networks (Transformer or U-Net):

$$p_\theta(z_{t-1} \mid z_t) = \mathcal{N}(z_{t-1}; \mu_\theta(z_t, t), \sigma_t^2 I)$$

DiffE2E implements a truncated DDIM sampler with $K \ll T$ steps; typical $K=2$ ensures real-time feasibility. HyDo models the continuous action as

$$\pi_m(a^{m,0} \mid s) = \mathcal{N}(a^{m,K}; 0, I) \prod_{k=1}^K p_\theta(a^{m,k-1} \mid a^{m,k}, s)$$

where $a^{m, 0}$ is sampled for execution.

3. Supervised and Reinforcement Policy Integration

In DiffE2E, explicit supervision anchors latent variables in control-relevant dimensions, with losses derived from regression (MSE) or weighted cross-entropy for classification:

$$\mathcal{L}_{\rm sup} = \sum_{i \in \Omega} \lambda_i \, \mathcal{L}_i(y_i, \hat{y}_i)$$

DiffE2E and HyDo both employ collaborative objectives: supervised/diffusion and actor/critic losses respectively are optimized jointly.

DiffE2E's end-to-end loss:

$$\mathcal{L} = \mathcal{L}_{\rm diff} + \lambda\,\mathcal{L}_{\rm sup}$$

with $\mathcal{L}_{\rm diff}$ as the denoising MSE over the trajectory latent, conditioned on perception features.

HyDo augments Soft Actor-Critic (SAC) with diffusion entropy regularization:

$$J_\pi(\theta) = \sum_t \mathbb{E}_{s_t, a_t^{0:K} \sim \pi_\theta} \left[ r(s_t, a_t^0) - \alpha \sum_{k=0}^K \log \pi_\theta(a_t^{k-1} \mid a_t^k, k, s_t) \right]$$

Structured variational inference establishes a lower bound on optimal trajectory likelihood, supporting the entropy term.

4. Architectural Modules for Hybrid Policy Learning

DiffE2E fuses multi-sensor features via a multi-scale, bidirectional cross-attention backbone reminiscent of Transfuser, aggregating LiDAR BEV, camera, and goal embeddings:

$$\tilde{\mathcal{C}} = [\mathcal{C}; f_{\rm goal}(\mathbf{g}); \mathbf{t}_{\rm emb}] + \mathbf{E}_{\rm pos}^{\mathcal{C}}$$

A hybrid Transformer decoder receives input queries concatenating noisy trajectories and supervision queries, processes them with self-attention and cross-attention against perception features, partitions outputs to diffusion and supervised heads.

In HyDo, state features are constructed for each discrete candidate; the continuous diffusion network (U-Net or consistency model) samples motion primitives, and discrete selection is performed via softmax over Q-values.

5. Impact on Multimodal Behavior, Controllability, and Robustness

Hybrid policies with diffusion components sample diverse and feasible trajectories, avoiding collapse to a single mean and ensuring better coverage of long-tail scenarios. Supervised or RL-optimized branches guarantee outputs adhere to control or task requirements.

DiffE2E demonstrates empirical superiority in closed-loop and simulated benchmarks: on CARLA Longest6, Driving Score 83.0, Route Completion 96, Infraction Score 0.86—surpassing prior baselines by >10 DS points. NAVSIM yields PDMS 92.7 and leading performance across collision, comfort, and progress metrics (Zhao et al., 26 May 2025).

HyDo achieves elevated success rates (e.g., sim2real 6D goals: HACMan 53%, HyDo 68–72%) and increased policy entropy, reflecting greater multimodal exploration and skill transfer in non-prehensile manipulation (Le et al., 22 Nov 2024).

6. Theoretical Foundations and Algorithmic Principles

Diffusion chain integration into policy optimization enables principled stochastic action modeling. Structured variational inference gives a lower bound interpretation to the entropy-regularized RL objective, legitimizing hybrid objectives and connecting them to log-likelihood maximization for optimal policies.

Gradient computation combines policy-gradient for discrete choices, behavior-cloning/regression for diffusion policies, and Q-learning for critic networks. These unified frameworks balance exploration and exploitation, maintaining diversity and fidelity in execution.

7. Applications and Generalization

Diffusion policies with hybrid action modeling have demonstrated state-of-the-art outcomes in autonomous driving—real-time, robust, and generalizable to out-of-distribution conditions—and manipulation where diverse, multimodal strategies are essential for transfer and generalization. A plausible implication is the extensibility of these methods to broader embodied intelligence domains, given their adaptability to hybrid space definition, modular architectures, and collaborative training paradigms.

Empirical and theoretical evidence supports diffusion-based hybrid approaches as a general-purpose strategy for advanced sequential decision-making in structured, multimodal action spaces.