TD3-HD: Hybrid TD3 for Mixed Action Control

• TD3-HD is a hybrid TD3 algorithm that handles discrete-continuous action spaces by factorizing the policy into a discrete mode selector and a continuous actor.
• It introduces a weighted clipped Q-learning target that mitigates overestimation bias through twin-critic architecture and marginalization over discrete actions.
• The method demonstrates improved stability and convergence in robotic manipulation tasks compared to alternative hybrid reinforcement learning approaches.

Searching arXiv for the cited papers and term usage to ground the article. TD3-HD most directly denotes Hybrid TD3, a reinforcement-learning method that extends Twin Delayed Deep Deterministic Policy Gradient (TD3) to parameterized hybrid action spaces of the form $A = \{(d, x_d): d \in D, x_d \in X_d\}$, where a discrete component selects a mode and a mode-conditioned continuous component specifies control quantities tailored to that mode (Tran et al., 1 Mar 2026). In the cited literature, however, the string “TD3-HD” is not fully standardized. It is used most precisely for the hybrid-action formulation in "Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space" (Tran et al., 1 Mar 2026); it also appears as a convenient label for a hierarchical/hybrid DQN+TD3 navigation design (He et al., 30 Oct 2025), and, in a different control context, as an informal shorthand for TD3 integrated with Hybrid Disturbance handling through a Hybrid Disturbance Observer in a cascaded TD3-PID controller (Zhang et al., 15 Apr 2026). A separate paper titled "TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendation" concerns dataset distillation and explicitly states that there is no mention of a variant named “TD3-HD” in that work (Zhang et al., 5 Feb 2025).

1. Canonical meaning: Hybrid TD3 in parameterized hybrid action spaces

In the hybrid-action formulation, the state space is $S$ and the hybrid action space is parameterized as

$A = \{(d, x_d): d \in D, x_d \in X_d\},$

where the discrete component $d$ selects a mode and the mode-conditioned continuous parameters $x_d \in X_d \subset \mathbb{R}^{m(d)}$ specify control quantities tailored to that mode (Tran et al., 1 Mar 2026). The policy is factorized as

$$\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),$$

with $\pi_c$ taken to be deterministic as an actor $\mu_c$, while $\pi_d$ is a differentiable softmax over logits (Tran et al., 1 Mar 2026). The action-value function is correspondingly hybrid,

$$Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).$$

The hybrid Bellman operator for a generic policy $S$0 is

$S$1

and, with deterministic $S$2 and TD3-style target networks, the backup reduces operationally to a target $S$3 for sampled transitions $S$4 (Tran et al., 1 Mar 2026). The stated motivation is that existing approaches either discretize continuous components or relax discrete choices into continuous approximations, and those approaches suffer from scalability limitations and training instability in high-dimensional action spaces and under domain randomization (Tran et al., 1 Mar 2026).

This formulation places TD3-HD within the literature on discrete-continuous hybrid action spaces, especially in robotic manipulation settings where “high-level task decisions and low-level joint-space execution must be jointly optimized” (Tran et al., 1 Mar 2026). A plausible implication is that the method is intended for settings where mode choice and mode-conditioned control are both first-class parts of the MDP, rather than for architectures that merely stack separate RL modules.

2. Architecture, objective, and learning rule

TD3-HD uses an actor with a shared trunk that encodes $S$5 and branches into a categorical head $S$6 and a mode-conditioned deterministic head $S$7 (Tran et al., 1 Mar 2026). In practice, $S$8 is fed as a one-hot or embedding into the continuous head to allow mode-specific parameterization (Tran et al., 1 Mar 2026). The critics are twin Q-networks $S$9 and $A = \{(d, x_d): d \in D, x_d \in X_d\},$0 whose inputs concatenate $A = \{(d, x_d): d \in D, x_d \in X_d\},$1, a one-hot for $A = \{(d, x_d): d \in D, x_d \in X_d\},$2, and $A = \{(d, x_d): d \in D, x_d \in X_d\},$3, and they are trained with the standard clipped double Q-learning structure

$A = \{(d, x_d): d \in D, x_d \in X_d\},$4

(Tran et al., 1 Mar 2026).

The actor objective departs from standard TD3 in a specific way: unlike standard TD3, which uses a single critic in the actor loss, TD3-HD retains clipped double Q in the actor update for robustness under hybrid actions (Tran et al., 1 Mar 2026). With marginalization over discrete modes, the objective is

$A = \{(d, x_d): d \in D, x_d \in X_d\},$5

Updates are delayed: the actor and target networks are updated less frequently than the critics, for example every policy_delay steps (Tran et al., 1 Mar 2026).

The corresponding maximization form is

$A = \{(d, x_d): d \in D, x_d \in X_d\},$6

where

$A = \{(d, x_d): d \in D, x_d \in X_d\},$7

and the gradients are given explicitly as

$A = \{(d, x_d): d \in D, x_d \in X_d\},$8

$A = \{(d, x_d): d \in D, x_d \in X_d\},$9

(Tran et al., 1 Mar 2026). The discrete head is trained by backpropagation through the softmax, so no sampling is needed in the actor update (Tran et al., 1 Mar 2026).

Exploration is also split by action type. Continuous exploration uses Gaussian noise at execution time and target policy smoothing for the TD3 target, while discrete exploration can be achieved by sampling $d$0 or by $d$1-greedy over $d$2’s logits (Tran et al., 1 Mar 2026). The paper states that the weighted target “naturally leverages the full discrete distribution for target construction, reducing sensitivity to early misclassification” (Tran et al., 1 Mar 2026).

3. Weighted clipped Q-learning target and bias analysis

A central contribution of TD3-HD is a theoretical analysis of overestimation bias in hybrid action settings, including formal bounds under twin-critic architectures and a complete bias ordering across five algorithmic variants (Tran et al., 1 Mar 2026). The overestimation bias is defined as

$d$3

namely the expectation gap between the target’s expected value and the true Bellman backup under the current target policy (Tran et al., 1 Mar 2026).

Under the analysis assumptions, the twin-critic noise model is

$d$4

with $d$5 independent zero-mean-shifted Gaussian with synchronized mean $d$6 and variance $d$7 under heavy domain randomization (Tran et al., 1 Mar 2026). If

$d$8

with $d$9 independent, then

$x_d \in X_d \subset \mathbb{R}^{m(d)}$0

which implies a controlled negative bias of magnitude $x_d \in X_d \subset \mathbb{R}^{m(d)}$1 relative to the synchronized mean $x_d \in X_d \subset \mathbb{R}^{m(d)}$2 (Tran et al., 1 Mar 2026). This clipped-min behavior is identified as the cornerstone of stability.

The standard hybrid TD3-style target uses greedy discrete selection:

$x_d \in X_d \subset \mathbb{R}^{m(d)}$3

with

$x_d \in X_d \subset \mathbb{R}^{m(d)}$4

and $x_d \in X_d \subset \mathbb{R}^{m(d)}$5 (Tran et al., 1 Mar 2026). TD3-HD replaces this with a weighted clipped Q-learning target:

$x_d \in X_d \subset \mathbb{R}^{m(d)}$6

where

$x_d \in X_d \subset \mathbb{R}^{m(d)}$7

and $x_d \in X_d \subset \mathbb{R}^{m(d)}$8 is target policy smoothing noise on the continuous parameters (Tran et al., 1 Mar 2026).

The key theoretical claim is that, under the synchronized bias shift and independence assumptions, the marginalization preserves the clipped min’s negative bias magnitude. Writing

$x_d \in X_d \subset \mathbb{R}^{m(d)}$9

with

$$\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),$$0

and using $$\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),$$1, the expectation of $$\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),$$2 becomes

$$\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),$$3

so “the marginalization preserves the clipped min’s negative bias magnitude $$\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),$$4 relative to $$\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),$$5” (Tran et al., 1 Mar 2026). The paper’s interpretation is that weighted averaging does not introduce additional pessimism beyond the clipped operator; it distributes credit across discrete modes and thereby smooths gradients (Tran et al., 1 Mar 2026).

4. Bias ordering, stability rationale, and computational implications

Under the stated assumptions, the paper establishes the expected bias ordering

$$\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),$$6

(Tran et al., 1 Mar 2026). The five variants are described as follows.

Variant	Construction	Bias characterization
TD3-HD	weighted clipped target over discrete actions	bias equals the clipped double Q bias
HyACC	TD3-like structure augmented with quantile truncation beyond TQC	truncation adds further pessimism
HyTQC	SAC-based quantile critics with truncation of top atoms	average over low quantiles produces negative bias
HyDARC	blends min and max of twin targets via $$\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),$$7	less negative bias than TD3-HD when $$\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),$$8
HyDATD3	selects the more optimistic of multiple candidate continuous actions before clipping	outer max reduces pessimism

For HyDARC, the target is

$$\pi(d, x_d \mid s) = \pi_d(d \mid s)\cdot \pi_c(x_d \mid s, d),$$9

and for HyDATD3,

$\pi_c$0

with

$\pi_c$1

(Tran et al., 1 Mar 2026). The comparative statement made in the source is that HyDATD3’s outer max “reduces pessimism, yielding strictly less negative (often closer to zero or positive) bias than clipped min alone” (Tran et al., 1 Mar 2026).

The rationale for TD3-HD’s empirical stability is also stated explicitly. Early in training, $\pi_c$2 is diffuse, and committing to a single $\pi_c$3 discrete choice amplifies variance and misclassification sensitivity; the weighted backup reduces variance in the targets seen by the critics, resulting in smoother critic learning and more stable actor gradients (Tran et al., 1 Mar 2026). Relative to an $\pi_c$4, the marginalization increases compute linearly with $\pi_c$5 per target evaluation, but for small-to-moderate $\pi_c$6 this cost is said to be negligible compared to critic forward passes (Tran et al., 1 Mar 2026). The same section reports that, empirically, this translates to improved stability and faster convergence on contact-rich manipulation tasks (Tran et al., 1 Mar 2026).

A plausible implication is that TD3-HD is especially appropriate when the discrete branch is semantically meaningful but modest in size, whereas extremely large discrete sets would expose the linear-in-$\pi_c$7 target-construction cost discussed in the paper.

5. Implementation details and empirical behavior in robotic manipulation

The implementation described for TD3-HD uses an actor with a shared trunk MLP, for example 2–3 layers with 256–512 units and ReLU activations, followed by a softmax head for $\pi_c$8 and a mode-conditioned MLP for $\pi_c$9 (Tran et al., 1 Mar 2026). The continuous output may use tanh to bound velocities or torques to $\mu_c$0 scaled to action ranges (Tran et al., 1 Mar 2026). The critics are two independent MLPs $\mu_c$1 with concatenated inputs $\mu_c$2, ReLU activations, and optional LayerNorm (Tran et al., 1 Mar 2026). Input normalization is performed with Welford’s online mean/variance normalization for each state component, maintaining standardized inputs in $\mu_c$3 (Tran et al., 1 Mar 2026).

Reported hyperparameter ranges include learning rates of $\mu_c$4 for actor and critics with Adam, target Polyak averaging $\mu_c$5, and target policy smoothing $\mu_c$6 with $\mu_c$7 clipped to action bounds (Tran et al., 1 Mar 2026). Exploration adds Gaussian noise to $\mu_c$8 at execution time, and for $\mu_c$9 either samples from the softmax or uses $\pi_d$0-greedy over logits early in training (Tran et al., 1 Mar 2026).

The training loop is specified procedurally. The actor $\pi_d$1, target $\pi_d$2, critics $\pi_d$3, their targets, and replay buffer $\pi_d$4 are initialized; then at each step the agent observes $\pi_d$5, samples or chooses $\pi_d$6, sets $\pi_d$7 exploration noise, executes $\pi_d$8, stores $\pi_d$9, computes for each $$Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).$$0 the target values

$$Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).$$1

and then

$$Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).$$2

before critic and delayed actor/target updates (Tran et al., 1 Mar 2026).

The experimental setting described in the source is robotic hybrid manipulation with a UF850 arm in PyBullet, discrete suction mode $$Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).$$3, and continuous 6-DOF joint velocities $$Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).$$4 across four tasks: reach, pick, move, put (Tran et al., 1 Mar 2026). Domain randomization is heavy at every reset over object type, pose, mass, friction, and scene layout (Tran et al., 1 Mar 2026). The reported empirical findings are that TD3-HD exhibits superior average returns and faster convergence across tasks, lower estimation bias magnitude (bounded negative bias near zero) and reduced variance under heavy randomization, and robustness “in high-dimensional continuous parameters coupled with a small discrete set” (Tran et al., 1 Mar 2026). It is also stated to outperform hybrid SAC variants, P-DQN/PDQN-like approaches, quantile-based hybrids, HyDATD3, and HyDARC in both mean performance and stability (Tran et al., 1 Mar 2026).

6. Alternative uses of the label “TD3-HD”

The label “TD3-HD” is also used in a distinct sense in "Hybrid DQN-TD3 Reinforcement Learning for Autonomous Navigation in Dynamic Environments" (He et al., 30 Oct 2025). There it denotes a two-level hybrid reinforcement learning framework in which a high-level DQN selects discrete behaviors or sub-goals and a low-level TD3 executes smooth continuous control to realize those sub-goals (He et al., 30 Oct 2025). The high level uses observations derived from pose, heading relative to the goal, and obstacle indicators such as angle-to-goal $$Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).$$5, distance-to-goal, and $$Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).$$6 from LiDAR; the low level outputs bounded continuous commands $$Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).$$7 with maximum linear/angular velocities $$Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).$$8 and $$Q: S \times D \times X_d \to \mathbb{R}, \qquad Q(s, d, x_d).$$9 (He et al., 30 Oct 2025). The system is implemented in ROS + Gazebo with TurtleBot3, a Gymnasium-compatible wrapper, PathBench analytics, and a unified reward shaping scheme that includes direction alignment, distance to sub-goal or goal, obstacle avoidance, smoothness, collision penalty, and time penalty (He et al., 30 Oct 2025).

Formally, that work keeps the standard DQN Bellman target

$S$00

and the standard TD3 target with target policy smoothing

$S$01

where

$S$02

(He et al., 30 Oct 2025). Unlike Hybrid TD3 (Tran et al., 1 Mar 2026), this is a hierarchical decomposition rather than a single hybrid-action actor-critic.

The empirical status is correspondingly different. The TD3-only baseline rises from negative to positive reward and stabilizes around 80–110 in late stages, whereas the hierarchical DQN+TD3 hybrid is described qualitatively as often rotating in place early in episodes and frequently terminating early or failing to converge (He et al., 30 Oct 2025). The paper attributes instability to multi-level non-stationarity, reward misalignment, hyperparameter or scheduling mismatch, environment sensitivity, and reward parameter tuning (He et al., 30 Oct 2025). Thus, although the label “TD3-HD” is used there as a TD3-based hierarchical/hybrid design, it does not denote the bias-controlled weighted clipped target of Hybrid TD3 (Tran et al., 1 Mar 2026).

A third usage appears in "Cascaded TD3-PID Hybrid Controller for Quadrotor Trajectory Tracking in Wind Disturbance Environments" (Zhang et al., 15 Apr 2026). In that work, the closest formal term is CTPH with HDOB, where “HD” denotes Hybrid Disturbance handling through the Hybrid Disturbance Observer integrated into altitude and attitude loops (Zhang et al., 15 Apr 2026). The cascaded architecture assigns outer-loop horizontal position control to TD3 and uses PID for altitude and attitude, with HDOB estimating disturbances from measured accelerations and applied commands and filtering them through a median filter, a gated first-order IIR low-pass filter, and an EMA before fusion (Zhang et al., 15 Apr 2026). The authors explicitly state that TD3 itself is not modified to estimate disturbances; instead it is integrated in a cascaded manner with PID+HDOB (Zhang et al., 15 Apr 2026). This is therefore another distinct use of a similar label rather than the same algorithmic object as Hybrid TD3.

7. Terminological clarification and scope

Across the cited works, “TD3-HD” is not a single universally fixed name. The most technically specific and internally defined use is the Hybrid TD3 algorithm for parameterized hybrid action spaces, with a factorized actor, twin critics over hybrid actions, delayed updates, and a weighted clipped Q-learning target that marginalizes over the discrete action distribution (Tran et al., 1 Mar 2026). In that sense, TD3-HD is a direct extension of TD3 to hybrid action MDPs, with formal overestimation-bias analysis and an explicit bias ordering relative to HyACC, HyTQC, HyDARC, and HyDATD3 (Tran et al., 1 Mar 2026).

By contrast, the navigation work uses the label for a hierarchical DQN+TD3 system with event-driven high-level sub-goal reselection, unified reward shaping, and a LiDAR-based safety gate that is described as a reasonable extension consistent with the paper’s environment and reward signals (He et al., 30 Oct 2025). The quadrotor work uses the label only informally for TD3 integrated with Hybrid Disturbance handling, but its formal method name is CTPH with HDOB (Zhang et al., 15 Apr 2026). Finally, the sequential recommendation paper titled TD3 concerns Tucker decomposition based dataset distillation and explicitly clarifies that it contains no mention of a variant named “TD3-HD” (Zhang et al., 5 Feb 2025).

For research usage, the least ambiguous practice is therefore to reserve “TD3-HD” for Hybrid TD3 in discrete-continuous hybrid action spaces unless the intended meaning is explicitly qualified by context. This suggests that, in contemporary arXiv usage, the term is best treated as context-dependent nomenclature rather than a globally standardized acronym.