P-DQN: Deep Q-Networks for Hybrid Actions

• P-DQN is a deep reinforcement learning method designed for hybrid discrete-continuous action spaces, integrating categorical decisions with continuous parameter tuning.
• The approach employs a dual-network architecture that fuses Q-value estimation for discrete actions with gradient ascent for continuous parameter optimization, yielding strong empirical performance.
• Variants like MP-DQN address issues such as false gradients by using multiple forward passes, enhancing training stability and sample efficiency.

Parametrized Deep Q-Networks (P-DQN) extend deep reinforcement learning (DRL) methods to environments characterized by discrete-continuous hybrid action spaces. In these settings, each action is encoded as a pair $(k, x_k)$, where $k$ indexes a categorical "high-level" action and $x_k$ parameterizes associated continuous control. Unlike conventional DRL techniques which assume solely discrete (as in DQN) or continuous (as in DDPG) action spaces, P-DQN natively incorporates both, addressing applications such as game agents and robotic control without explicit discretization or relaxation of the hybrid action space. The P-DQN algorithm integrates Q-learning over discrete choices with gradient ascent over action parameters and achieves empirically strong performance and sample efficiency on benchmark domains including RoboCup soccer and commercial video games (Xiong et al., 2018). However, subsequent research identifies theoretical and practical limitations in the original formulation and advances variants such as Multi-Pass DQN (MP-DQN) to address these issues (Bester et al., 2019).

1. Markov Decision Processes with Hybrid Action Spaces

P-DQN operates on Markov Decision Processes (MDPs) with a hybrid action space

$$\mathcal A = \{ (k, x_k) \mid k \in \{1, \dots, K\},\ x_k \in \mathcal X_k \subset \mathbb R^{d_k} \}.$$

Here, each state $s_t \in \mathcal S$ is transformed using a transition kernel $s_{t+1} \sim p(\cdot \mid s_t, a_t)$, reward $r_t = r(s_t, a_t)$, and standard discounted return $R_t = \sum_{i \geq t} \gamma^{i-t} r_i$. Discrete choices typically model high-level "moves," while continuous parameters capture context-specific execution (e.g., action directions, speeds, or spatial coordinates).

2. P-DQN Architecture and Bellman Operator

The canonical P-DQN framework deploys two neural networks:

• A Q-network $Q(s, k, x_k; \omega)$ estimating the state-action value,
• A deterministic “parameter-actor” network $x_k(s; \theta)$ mapping state and discrete action index to the continuous parameter.

The twin-network architecture proceeds as follows:

1. The state $s$ is encoded via a shared feature extractor.
2. For each discrete action $k$, an actor output $x_k(s; \theta)$ provides the continuous action.
3. The Q-head consumes the encoded state and $x_k$ pair and outputs scalar $Q(s, k, x_k; \omega)$.

Policy evaluation is governed by the hybrid Bellman optimality equation: $$Q^*(s_t, k_t, x_{k_t}) = \mathbb E\left[ r_t + \gamma \max_{k'} \sup_{x'\in \mathcal X_{k'}} Q^*(s_{t+1}, k', x') \mid s_t, a_t \right].$$ Direct maximization $\sup_{x_k} Q(\cdot)$ is intractable, so P-DQN trains $x_k(s;\theta)$ to approximate the maximizer by ascending the Q-value landscape, with the practical surrogate objective

$$L_t(\theta) = -\sum_{k=1}^K Q(s_t, k, x_k(s_t;\theta); \omega).$$

The Q-network is trained by minimizing the mean squared error between predicted values and target values defined as

$$y_t = r_t + \gamma \max_{k'} Q(s_{t+1}, k', x_{k'}(s_{t+1}; \theta^-); \omega^-),$$

where $(\omega^-, \theta^-)$ are periodically-updated target networks for stability (Xiong et al., 2018, Bester et al., 2019).

3. Algorithmic Procedure and Practical Considerations

P-DQN operates in an off-policy manner using experience replay. The typical workflow:

1. Observe $s_t$.
2. Compute $x_k(s_t; \theta)$ for each $k$.
3. Select $(k_t, x_{k_t})$ either randomly (with $\epsilon$) or by maximizing $Q(s_t, k, x_k; \omega)$.
4. Execute the action, receive $(r_t, s_{t+1})$, and store the transition.
5. Sample mini-batches from the replay buffer.
6. Update Q-network and actor per their respective losses.
7. Periodically synchronize target networks.

The architecture is agnostic to action set size and does not require explicit discretization or relaxation, which avoids combinatorial explosion and preserves gradient structure. Action parameter bounds are incorporated via output penalties or clipping. The method supports $n$-step returns and asynchronous parallelism (Xiong et al., 2018).

4. Theoretical Analysis of Joint Parameterization

Subsequent analysis observes that the original P-DQN implementation concatenates all $x_k$ into a unified input to a single Q-network: $Q(s, k, x_1, ..., x_K; \omega).$ This induces spurious cross-dependencies: the Q-value for action $k$ can be sensitive to non-associated parameters $x_j$ for $j \neq k$. Two critical issues result (Bester et al., 2019):

• False gradients: During actor updates, $x_j$ where $j \neq k$ receive nonzero gradients, although only $x_{k^*}$ determines the policy’s current execution.
• Policy distortion: Updates to one $x_j$ can inadvertently perturb all Q-values $Q(s, k, x)$, altering the ranking $\arg\max_k Q(s, k, x)$ and destabilizing discrete action selection.

These effects violate the functional separation required for sound Bellman backups in hybrid-action MDPs and represent a key theoretical weakness in the original joint-parameter network design (Bester et al., 2019).

5. Multi-Pass DQN and Empirical Evaluation

To resolve the above, MP-DQN implements $K$ separate forward passes per state:

• For each action $k$, all other action-parameters $x_j$ are set to zero: $x \odot e_k = (0, ..., 0, x_k, 0, ..., 0)$.
• The Q-network thus receives input $(s, x \odot e_k)$, restoring pure functional dependency of $Q_k$ on $x_k$.
• Gradients only flow through relevant parameters, eliminating false updates.

This approach achieves the theoretical behavior of $K$ distinct Q-networks while sharing representations, leading to more accurate training and stable discrete policy ordering (Bester et al., 2019).

Empirically, MP-DQN exhibits superior data efficiency and asymptotic performance on benchmark tasks—Platform, Robot Soccer Goal, and Half Field Offense—relative to P-DQN with joint parameterization, separate-per-action Q-networks (SP-DQN), Q-PAMDP, and PA-DDPG. Final metric summaries are as follows:

Algorithm	Platform Return	Robot Soccer Goal P(goal)	HFO P(goal)	HFO Avg steps to goal
Q-PAMDP	0.789 ± 0.188	0.452 ± 0.093	0 ± 0	n/a
PA-DDPG	0.284 ± 0.061	0.006 ± 0.020	0.875 ± 0.182	95 ± 7
P-DQN (joint)	0.964 ± 0.068	0.701 ± 0.078	0.883 ± 0.085	111 ± 11
SP-DQN	0.941 ± 0.164	0.752 ± 0.131	0.718 ± 0.131	99 ± 7
MP-DQN	0.987 ± 0.039	0.789 ± 0.070	0.913 ± 0.070	99 ± 12

MP-DQN's learning curves show consistently faster convergence and higher final performance, corroborating the necessity of correct Q-function parameterization (Bester et al., 2019).

6. Comparative Advantages and Limitations

P-DQN offers a gradient-based framework for hybrid action spaces, circumventing pitfalls of pure discretization (avoiding exponential blow-up, preserving smooth gradients) and continuous relaxation (avoiding unnecessary over-parameterization and misalignment of action semantics). It enables efficient off-policy training, use of large replay buffers, and injection of demonstration data.

However, naively concatenating action parameters violates the independence assumption underpinning the Bellman operator, leading to detrimental "false gradients." MP-DQN provides an efficient remedy, retaining representational sharing while enforcing correct gradients and functional dependencies. SP-DQN, an alternative based on separate networks per action, avoids false gradients but is less parameter-efficient.

Empirical evaluations demonstrate that careful attention to network parameterization is critical for parameterized-action DRL algorithms, with multi-pass architectures offering a practical and theoretically-justified solution for diverse benchmark domains (Xiong et al., 2018, Bester et al., 2019).

7. Broader Impact and Future Directions

The development of P-DQN and its variants has established a practical template for off-policy DRL in hybrid discrete-continuous spaces commonly encountered in games and robotics. The multi-pass methodology underlying MP-DQN appears widely applicable for parameterized Q-learning architectures and provides a baseline for future work addressing generalization, sample efficiency, and robustness. A plausible implication is that as environments and agent design shift toward richer action parameterizations, architectural choices that respect action structure will remain central for algorithmic progress and empirical success (Bester et al., 2019).