---
title: 'TD3-HD: Hybrid TD3 for Mixed Action Control'
url: https://www.emergentmind.com/topics/td3-hd
type: topic
---

# TD3-HD: Hybrid TD3 for Mixed Action Control

Searching arXiv for the cited papers and term usage to ground the article.
TD3-HD most directly denotes **Hybrid TD3**, a reinforcement-learning method that extends Twin Delayed Deep Deterministic Policy Gradient (TD3) to **parameterized hybrid action spaces** of the form $A = \{(d, x_d): d \in D, x_d \in X_d\}$, where a discrete component selects a mode and a mode-conditioned continuous component specifies control quantities tailored to that mode [2603.01302]. In the cited literature, however, the string “TD3-HD” is not fully standardized. It is used most precisely for the hybrid-action formulation in "Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space" [2603.01302]; it also appears as a convenient label for a **hierarchical/hybrid DQN+TD3 navigation design** [2510.26646], and, in a different control context, as an informal shorthand for TD3 integrated with **Hybrid Disturbance handling** through a Hybrid Disturbance Observer in a cascaded TD3-PID controller [2604.13505]. A separate paper titled "TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendation" concerns dataset distillation and explicitly states that there is **no mention of a variant named “TD3-HD”** in that work [2502.02854].

## 1. Canonical meaning: Hybrid TD3 in parameterized hybrid action spaces

In the hybrid-action formulation, the state space is $S$ and the hybrid action space is parameterized as
$$
A = \{(d, x_d): d \in D, x_d \in X_d\},
$$
where the discrete component $d$ selects a mode and the mode-conditioned continuous parameters $x_d \in X_d \subset \mathbb{R}^{m(d)}$ specify control quantities tailored to that mode [2603.01302]. The policy is factorized as
$$
\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),
$$
with $\pi_c$ taken to be deterministic as an actor $\mu_c$, while $\pi_d$ is a differentiable softmax over logits [2603.01302]. The action-value function is correspondingly hybrid,
$$
Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).
$$

The hybrid Bellman operator for a generic policy $\pi$ is
$$
(T^\pi Q)(s, d, x_d) = r(s, d, x_d) + \gamma \cdot E_{s' \sim P(\cdot \mid s, d, x_d)}
\left[
E_{d' \sim \pi_d(\cdot \mid s')}
E_{x_{d'} \sim \pi_c(\cdot \mid s', d')}
Q(s', d', x_{d'})
\right],
$$
and, with deterministic $\mu_c$ and TD3-style target networks, the backup reduces operationally to a target $y$ for sampled transitions $(s, d, x_d, r, s')$ [2603.01302]. The stated motivation is that existing approaches either discretize continuous components or relax discrete choices into continuous approximations, and those approaches suffer from scalability limitations and training instability in high-dimensional action spaces and under domain randomization [2603.01302].

This formulation places TD3-HD within the literature on **discrete-continuous hybrid action spaces**, especially in robotic manipulation settings where “high-level task decisions and low-level joint-space execution must be jointly optimized” [2603.01302]. A plausible implication is that the method is intended for settings where mode choice and mode-conditioned control are both first-class parts of the MDP, rather than for architectures that merely stack separate RL modules.

## 2. Architecture, objective, and learning rule

TD3-HD uses an actor with a shared trunk that encodes $s$ and branches into a categorical head $\pi_d(d \mid s)$ and a mode-conditioned deterministic head $\mu_c(s, d)$ [2603.01302]. In practice, $d$ is fed as a one-hot or embedding into the continuous head to allow mode-specific parameterization [2603.01302]. The critics are twin Q-networks $Q_{\phi_1}$ and $Q_{\phi_2}$ whose inputs concatenate $s$, a one-hot for $d$, and $x_d$, and they are trained with the standard clipped double Q-learning structure
$$
L_{\text{critic}}(\phi_1, \phi_2) =
E_{(s,d,x_d,r,s')\sim D}
\left[
(Q_{\phi_1}(s,d,x_d)-y)^2 + (Q_{\phi_2}(s,d,x_d)-y)^2
\right]
$$
[2603.01302].

The actor objective departs from standard TD3 in a specific way: unlike standard TD3, which uses a single critic in the actor loss, TD3-HD **retains clipped double Q in the actor update for robustness under hybrid actions** [2603.01302]. With marginalization over discrete modes, the objective is
$$
L_{\text{actor}}(\theta) =
- E_{s\sim D}
\left[
\sum_{d\in D} \pi_d(d \mid s)\cdot \min_{i\in\{1,2\}} Q_{\phi_i}(s, d, \mu_c(s,d))
\right].
$$
Updates are delayed: the actor and target networks are updated less frequently than the critics, for example every `policy_delay` steps [2603.01302].

The corresponding maximization form is
$$
J(\pi) := E_{s\sim D}
\left[
\sum_{d\in D}\pi_d(d \mid s)\cdot Q_{\min}(s,d,\mu_c(s,d))
\right],
$$
where
$$
Q_{\min}(s,d,x) := \min_{i\in\{1,2\}} Q_{\phi_i}(s,d,x),
$$
and the gradients are given explicitly as
$$
\nabla_{\theta_d} J =
E_s\left[\sum_d \nabla_{\theta_d}\pi_d(d\mid s)\cdot Q_{\min}(s,d,\mu_c(s,d))\right],
$$
$$
\nabla_{\theta_c} J =
E_s\left[
\sum_d \pi_d(d\mid s)\cdot
\nabla_x Q_{\min}(s,d,x)\vert_{x=\mu_c(s,d)}\cdot
\nabla_{\theta_c}\mu_c(s,d)
\right]
$$
[2603.01302]. The discrete head is trained by backpropagation through the softmax, so no sampling is needed in the actor update [2603.01302].

Exploration is also split by action type. Continuous exploration uses Gaussian noise at execution time and target policy smoothing for the TD3 target, while discrete exploration can be achieved by sampling $d \sim \pi_d(d \mid s)$ or by $\epsilon$-greedy over $\pi_d$’s logits [2603.01302]. The paper states that the weighted target “naturally leverages the full discrete distribution for target construction, reducing sensitivity to early misclassification” [2603.01302].

## 3. Weighted clipped Q-learning target and bias analysis

A central contribution of TD3-HD is a theoretical analysis of **overestimation bias in hybrid action settings**, including formal bounds under twin-critic architectures and a complete bias ordering across five algorithmic variants [2603.01302]. The overestimation bias is defined as
$$
\delta Q :=
E[y] -
\left(
r + \gamma \cdot E_{s'} E_{d'\sim \pi_d'} E_{x_{d'}'}
[Q^\pi(s', d', x_{d'}')]
\right),
$$
namely the expectation gap between the target’s expected value and the true Bellman backup under the current target policy [2603.01302].

Under the analysis assumptions, the twin-critic noise model is
$$
Q_{\phi_i}(\cdot) = Q^\pi(\cdot) + \epsilon_i,
$$
with $\epsilon_1, \epsilon_2$ independent zero-mean-shifted Gaussian with synchronized mean $\mu$ and variance $\sigma^2$ under heavy domain randomization [2603.01302]. If
$$
X = Q^\pi + \epsilon_1,\qquad Y = Q^\pi + \epsilon_2,
$$
with $\epsilon_i \sim N(\mu,\sigma^2)$ independent, then
$$
E[\min(X,Y)] = Q^\pi + \mu - \sigma/\sqrt{\pi},
$$
which implies a controlled negative bias of magnitude $\sigma/\sqrt{\pi}$ relative to the synchronized mean $\mu$ [2603.01302]. This clipped-min behavior is identified as the cornerstone of stability.

The standard hybrid TD3-style target uses greedy discrete selection:
$$
y_{\text{std}} = r + \gamma \cdot \min_{i\in\{1,2\}}Q_{\phi_i'}(s', d', x_{d'}'),
$$
with
$$
d' = \arg\max_{k\in D}\pi_d'(k \mid s'), \qquad
x_{d'}' = \mu_c'(s', d') + \epsilon,
$$
and $\epsilon \sim N(0,\sigma^2 I)$ [2603.01302]. TD3-HD replaces this with a **weighted clipped Q-learning target**:
$$
y_{\text{HD}} = r + \gamma \cdot \sum_{d\in D}\pi_d'(d \mid s')\cdot \min_{i\in\{1,2\}}Q_{\phi_i'}(s', d, x_d'),
$$
where
$$
x_d' = \mu_c'(s', d) + \epsilon
$$
and $\epsilon \sim N(0,\sigma^2 I)$ is target policy smoothing noise on the continuous parameters [2603.01302].

The key theoretical claim is that, under the synchronized bias shift and independence assumptions, the marginalization **preserves the clipped min’s negative bias magnitude**. Writing
$$
\min_i Q_{\phi_i'}(s', d, x_d') = Q^\pi(s', d, x_d') + m_d,
$$
with
$$
E[m_d] = \mu - \sigma/\sqrt{\pi},
$$
and using $\sum_d \pi_d'(d\mid s') = 1$, the expectation of $y_{\text{HD}}$ becomes
$$
E[y_{\text{HD}}]
=
E[r]
+ \gamma \cdot E_{s'}
\left[
E_{d'\sim \pi_d'}Q^\pi(s', d', x_{d'}')
\right]
+ \gamma\cdot (\mu - \sigma/\sqrt{\pi}),
$$
so “the marginalization preserves the clipped min’s negative bias magnitude $\sigma/\sqrt{\pi}$ relative to $\mu$” [2603.01302]. The paper’s interpretation is that weighted averaging does not introduce additional pessimism beyond the clipped operator; it distributes credit across discrete modes and thereby smooths gradients [2603.01302].

## 4. Bias ordering, stability rationale, and computational implications

Under the stated assumptions, the paper establishes the expected bias ordering
$$
\delta Q_{\text{TD3-HD}} < \delta Q_{\text{HyACC}} \approx \delta Q_{\text{HyTQC}} < \delta Q_{\text{HyDARC}} < \delta Q_{\text{HyDATD3}}
$$
[2603.01302]. The five variants are described as follows.

| Variant | Construction | Bias characterization |
|---|---|---|
| TD3-HD | weighted clipped target over discrete actions | bias equals the clipped double Q bias |
| HyACC | TD3-like structure augmented with quantile truncation beyond TQC | truncation adds further pessimism |
| HyTQC | SAC-based quantile critics with truncation of top atoms | average over low quantiles produces negative bias |
| HyDARC | blends min and max of twin targets via $\lambda \in [0,1]$ | less negative bias than TD3-HD when $\lambda > 0.5$ |
| HyDATD3 | selects the more optimistic of multiple candidate continuous actions before clipping | outer max reduces pessimism |

For HyDARC, the target is
$$
y_{\text{DARC}} =
r + \gamma
\left[
\lambda \cdot \min(S_1,S_2) + (1-\lambda)\cdot \max(S_1,S_2)
\right],
$$
and for HyDATD3,
$$
y_{\text{DATD3}} = r + \gamma \cdot \max(T(a_{c1}'), T(a_{c2}')),
$$
with
$$
T(a_c') = \sum_d \pi_d'(d \mid s')\cdot \min_i Q_{\phi_i'}(s', d, a_c')
$$
[2603.01302]. The comparative statement made in the source is that HyDATD3’s outer max “reduces pessimism, yielding strictly less negative (often closer to zero or positive) bias than clipped min alone” [2603.01302].

The rationale for TD3-HD’s empirical stability is also stated explicitly. Early in training, $\pi_d'$ is diffuse, and committing to a single $\arg\max$ discrete choice amplifies variance and misclassification sensitivity; the weighted backup reduces variance in the targets seen by the critics, resulting in smoother critic learning and more stable actor gradients [2603.01302]. Relative to an $\arg\max d'$, the marginalization increases compute linearly with $|D|$ per target evaluation, but for small-to-moderate $|D|$ this cost is said to be negligible compared to critic forward passes [2603.01302]. The same section reports that, empirically, this translates to improved stability and faster convergence on contact-rich manipulation tasks [2603.01302].

A plausible implication is that TD3-HD is especially appropriate when the discrete branch is semantically meaningful but modest in size, whereas extremely large discrete sets would expose the linear-in-$|D|$ target-construction cost discussed in the paper.

## 5. Implementation details and empirical behavior in robotic manipulation

The implementation described for TD3-HD uses an actor with a shared trunk MLP, for example 2–3 layers with 256–512 units and ReLU activations, followed by a softmax head for $\pi_d$ and a mode-conditioned MLP for $\mu_c$ [2603.01302]. The continuous output may use `tanh` to bound velocities or torques to $[-1,1]$ scaled to action ranges [2603.01302]. The critics are two independent MLPs $Q_{\phi_1}, Q_{\phi_2}$ with concatenated inputs $(s,\text{one-hot}(d),x_d)$, ReLU activations, and optional `LayerNorm` [2603.01302]. Input normalization is performed with **Welford’s online mean/variance normalization for each state component**, maintaining standardized inputs in $[-1,1]$ [2603.01302].

Reported hyperparameter ranges include learning rates of $3\mathrm{e}{-4}$ for actor and critics with Adam, target Polyak averaging $\tau \in [0.005,0.02]$, and target policy smoothing $\epsilon \sim N(0,\sigma^2 I)$ with $\sigma \in [0.1,0.2]$ clipped to action bounds [2603.01302]. Exploration adds Gaussian noise to $\mu_c$ at execution time, and for $\pi_d$ either samples from the softmax or uses $\epsilon$-greedy over logits early in training [2603.01302].

The training loop is specified procedurally. The actor $\theta = \{\theta_d,\theta_c\}$, target $\theta'$, critics $\phi_1,\phi_2$, their targets, and replay buffer $D$ are initialized; then at each step the agent observes $s$, samples or chooses $d \sim \pi_d(d\mid s)$, sets $x_d = \mu_c(s,d) +$ exploration noise, executes $a=(d,x_d)$, stores $(s,d,x_d,r,s')$, computes for each $d' \in D$ the target values
$$
x_{d'}' = \mu_c'(s',d') + \epsilon,\qquad
v_{d'} = \min_i Q_{\phi_i'}(s', d', x_{d'}'),
$$
and then
$$
y = r + \gamma \cdot \sum_{d'}\pi_d'(d' \mid s')\cdot v_{d'}
$$
before critic and delayed actor/target updates [2603.01302].

The experimental setting described in the source is robotic hybrid manipulation with a **UF850 arm in PyBullet**, discrete suction mode $d \in \{\text{off}, \text{on}\}$, and continuous 6-DOF joint velocities $x_d \in \mathbb{R}^6$ across four tasks: **reach, pick, move, put** [2603.01302]. Domain randomization is heavy at every reset over object type, pose, mass, friction, and scene layout [2603.01302]. The reported empirical findings are that TD3-HD exhibits **superior average returns and faster convergence across tasks**, **lower estimation bias magnitude (bounded negative bias near zero) and reduced variance under heavy randomization**, and robustness “in high-dimensional continuous parameters coupled with a small discrete set” [2603.01302]. It is also stated to outperform hybrid SAC variants, P-DQN/PDQN-like approaches, quantile-based hybrids, HyDATD3, and HyDARC in both mean performance and stability [2603.01302].

## 6. Alternative uses of the label “TD3-HD”

The label “TD3-HD” is also used in a distinct sense in "Hybrid DQN-TD3 Reinforcement Learning for Autonomous Navigation in Dynamic Environments" [2510.26646]. There it denotes a **two-level hybrid reinforcement learning framework** in which a high-level DQN selects discrete behaviors or sub-goals and a low-level TD3 executes smooth continuous control to realize those sub-goals [2510.26646]. The high level uses observations derived from pose, heading relative to the goal, and obstacle indicators such as angle-to-goal $\theta$, distance-to-goal, and $d_{\min}$ from LiDAR; the low level outputs bounded continuous commands $(v,\omega)$ with maximum linear/angular velocities $[0,1]\ \mathrm{m/s}$ and $[-1,1]\ \mathrm{rad/s}$ [2510.26646]. The system is implemented in ROS + Gazebo with TurtleBot3, a Gymnasium-compatible wrapper, PathBench analytics, and a unified reward shaping scheme that includes direction alignment, distance to sub-goal or goal, obstacle avoidance, smoothness, collision penalty, and time penalty [2510.26646].

Formally, that work keeps the standard DQN Bellman target
$$
y = r + \gamma \max_{a'}Q_{\theta^-}(s',a')
$$
and the standard TD3 target with target policy smoothing
$$
y = r + \gamma \min_{j\in\{1,2\}}Q_{\bar{\theta}_j}(s', a'),
$$
where
$$
\epsilon \sim \mathrm{clip}(\mathcal{N}(0,\sigma), -c, c),\qquad a' = \pi_{\bar{\phi}}(s') + \epsilon
$$
[2510.26646]. Unlike Hybrid TD3 [2603.01302], this is a **hierarchical decomposition** rather than a single hybrid-action actor-critic.

The empirical status is correspondingly different. The TD3-only baseline rises from negative to positive reward and stabilizes around 80–110 in late stages, whereas the hierarchical DQN+TD3 hybrid is described qualitatively as often rotating in place early in episodes and frequently terminating early or failing to converge [2510.26646]. The paper attributes instability to multi-level non-stationarity, reward misalignment, hyperparameter or scheduling mismatch, environment sensitivity, and reward parameter tuning [2510.26646]. Thus, although the label “TD3-HD” is used there as a TD3-based hierarchical/hybrid design, it does not denote the bias-controlled weighted clipped target of Hybrid TD3 [2603.01302].

A third usage appears in "Cascaded TD3-PID Hybrid Controller for Quadrotor Trajectory Tracking in Wind Disturbance Environments" [2604.13505]. In that work, the closest formal term is **CTPH with HDOB**, where “HD” denotes **Hybrid Disturbance handling through the Hybrid Disturbance Observer** integrated into altitude and attitude loops [2604.13505]. The cascaded architecture assigns outer-loop horizontal position control to TD3 and uses PID for altitude and attitude, with HDOB estimating disturbances from measured accelerations and applied commands and filtering them through a median filter, a gated first-order IIR low-pass filter, and an EMA before fusion [2604.13505]. The authors explicitly state that TD3 itself is not modified to estimate disturbances; instead it is integrated in a cascaded manner with PID+HDOB [2604.13505]. This is therefore another distinct use of a similar label rather than the same algorithmic object as Hybrid TD3.

## 7. Terminological clarification and scope

Across the cited works, “TD3-HD” is not a single universally fixed name. The most technically specific and internally defined use is the Hybrid TD3 algorithm for parameterized hybrid action spaces, with a factorized actor, twin critics over hybrid actions, delayed updates, and a **weighted clipped Q-learning target that marginalizes over the discrete action distribution** [2603.01302]. In that sense, TD3-HD is a direct extension of TD3 to hybrid action MDPs, with formal overestimation-bias analysis and an explicit bias ordering relative to HyACC, HyTQC, HyDARC, and HyDATD3 [2603.01302].

By contrast, the navigation work uses the label for a **hierarchical DQN+TD3 system** with event-driven high-level sub-goal reselection, unified reward shaping, and a LiDAR-based safety gate that is described as a reasonable extension consistent with the paper’s environment and reward signals [2510.26646]. The quadrotor work uses the label only informally for **TD3 integrated with Hybrid Disturbance handling**, but its formal method name is CTPH with HDOB [2604.13505]. Finally, the sequential recommendation paper titled TD3 concerns Tucker decomposition based dataset distillation and explicitly clarifies that it contains **no mention of a variant named “TD3-HD”** [2502.02854].

For research usage, the least ambiguous practice is therefore to reserve “TD3-HD” for **Hybrid TD3** in discrete-continuous hybrid action spaces unless the intended meaning is explicitly qualified by context. This suggests that, in contemporary arXiv usage, the term is best treated as **context-dependent nomenclature** rather than a globally standardized acronym.

Source: https://www.emergentmind.com/topics/td3-hd