---
title: Hierarchical Twin-Dueling MARL (HiTDMA)
url: https://www.emergentmind.com/topics/hierarchical-twin-dueling-multi-agent-algorithm-hitdma
type: topic
---

# Hierarchical Twin-Dueling MARL (HiTDMA)

HiTDMA, short for **Hierarchical Twin-Dueling Multi-agent Algorithm**, is a hierarchical multi-agent deep reinforcement learning scheme introduced for **fluid antenna (FA)-assisted mobile edge computing (MEC)** uplink offloading, where channel estimation, FA port selection, beamforming, power control, and computation resource allocation are tightly coupled. In the proposed framework, the joint optimization target is to minimize the **maximum delay among all users** under imperfect channel state information (CSI), mixed discrete-continuous control, and a non-convex system objective. The method is not a generic name for hierarchical MARL; it denotes a specific hybrid design that combines **D3QN** on the user side, **TD3** on the base-station side, and a **game theory-assisted** reduction of the power-control search space, with CSI supplied by **IBM-CCS** channel estimation [2509.19340].

## 1. Problem domain and optimization target

HiTDMA is formulated for an uplink MEC system consisting of an **MEC server**, \(N\) users, and a base station equipped with \(K\) fluid antennas, with \(N \le K\). Each user is associated with one FA receiver at the FA-BS, and each FA can switch among \(M\) preset positions over a linear space \(\mathcal{C}_d\) of length \(Wl\), where \(l\) is the wavelength. For user \(n\), the FA receiver’s antenna position vector is
$$
\mathbf{d}_n =  \left [ d_1,d_2,\cdot \cdot \cdot,d_{N_p} \right ]^T \in\mathcal{C}_d.
$$
The received uplink signal at the FA-BS is written as
$$
\mathbf{y}=\mathbf{W}^H\mathbf{H}(\tilde{\mathbf{d})\mathbf{P}^{\frac{1}{2}\mathbf{x}+\mathbf{W}^H\mathbf{n}
$$
where \(\mathbf{W}\) is the receive beamforming matrix, \(\mathbf{H}(\tilde{\mathbf{d}})\) is the channel matrix depending on FA positions, and \(\mathbf{P}^{\frac{1}{2}}\) is the user power matrix. The per-user uplink channel is modeled by
$$
\tilde{\mathbf{h}_n(\mathbf{d}_n)=\rho_n\tilde{\mathbf{g}_n\odot \mathbf{\alpha }_n (\mathbf{d}_n,\theta_n ),
$$
with
$$
\mathbf{\alpha }_n (\mathbf{d}_n,\theta_n )=\left [ e^{j\frac{2\pi}{l}d_1\cos(\theta_n)},e^{j\frac{2\pi}{l}d_2\cos(\theta_n)},\cdots, e^{j\frac{2\pi}{l}d_{N_p}\cos(\theta_n)} \right ]^T.
$$

Communication performance enters the control objective through
$$
\text{SINR}_n=\frac{\left | \mathbf{w}_n^H\tilde{\mathbf{h}_n \right |^2 p_n}{\sum^N_{k= 1,k\ne n} \left | \mathbf{w}_k^H\tilde{\mathbf{h}_k \right |^2 p_k+\left \| \mathbf{w}_n \right \|_2^2\sigma^2  },
$$
and
$$
R_n = B\log (1+\text{SINR}_n).
$$
Delay compares local execution and edge offloading:
$$
t_n^l = \frac{C_n}{f_n^l}, \qquad
t_n^t = \frac{C_n}{R_n}, \qquad
t_n^{exe}=\frac{C_n}{\beta_nF^{max} }.
$$
The total user delay is
$$
t_n = \min \left \{ (t_n^t+t_n^{exe}),t_n^l \right \}.
$$

The global problem is to jointly choose FA antenna position vectors \(\tilde{\mathbf d}\), receive beamforming matrix \(\mathbf W\), user transmit power vector \(\mathbf P\), and MEC computation resource allocation vector \(\mathbf Z\), while minimizing the maximum delay among all users:
$$
\begin{equation}
\begin{aligned}
\mathcal{P} 1: \min_{\{\tilde{\mathbf{d}, \mathbf{W}, \mathbf{P},\mathbf{Z}\} & \max_n \ t_n \\
\text { s.t. } & \mathrm{C} 1: \mathbf{d}_n\in\mathcal{C}_d,n\in \{ 1,\cdots ,N\}, \\
& \mathrm{C} 2: tr(\mathbf{w_n} \mathbf{w_n}^H )\le 1,n\in\{1,\cdots ,N\}, \\
& \mathrm{C} 3: p_n\le p_{max},n\in\{1,\cdots ,N\},\\
& \mathrm{C} 4: \sum_{n=1}^{N} \beta_iF^{max} \le F^{max}.
\end{aligned}
\end{equation}
$$
The paper characterizes this problem as high-dimensional and non-convex because it combines **discrete** FA port selection with **continuous** beamforming, power, and MEC allocation, while FA port choice changes the channel itself [2509.19340].

## 2. Hierarchical organization of HiTDMA

The term “Hierarchical” in HiTDMA refers to an explicit two-layer decomposition of the control problem. The **user-side lower layer** handles discrete FA port selection through **DUAs**, while the **BS-side upper layer** handles continuous optimization of beamforming, pricing factor, and MEC resource allocation through **TBAs**. The lower layer acts first, and the upper layer observes an augmented state containing the lower-layer decisions. This dependence is written as
$$
s_t^b=\left [ s_t^u,\tilde{\mathbf{d}   \right ].
$$

The lower-layer user state is
$$
s_t^u=\left [ \mathbf{g}^t,loc^t,\mathbf{t}^{t-1}  \right ],
$$
where \(\mathbf g^t=[\mathbf g_1^{a,t},\ldots,\mathbf g_N^{a,t}]\), \(loc^t\) is user position information, and \(\mathbf t^{t-1}\) is prior delay information. The user-side action is discrete FA port selection,
$$
a_{t,n}^u= \mathbf{d}_n.
$$
The BS-side action is continuous:
$$
a_t^b=\left [ \text{Re}\{\mathbf{W}\}, \text{Im}\{\mathbf{W}\},\lambda,\mathbf{Z} \right ].
$$
Beamforming weights are split into real and imaginary parts, \(\lambda\) is a scalar pricing factor, and \(\mathbf Z\) is the MEC allocation vector.

This hierarchy is motivated by structural heterogeneity in the control variables. Port selection is discrete and user-local, whereas beamforming, pricing, and computation allocation are continuous and BS-side. A plausible implication is that the hierarchy does not merely reduce action dimensionality; it also permits algorithm-specific treatment of qualitatively different subproblems. In the paper’s implementation, the DUAs and TBAs share information “within each category” to avoid suboptimal convergence, and the overall control loop is coordinated through sequential state construction and joint environment execution [2509.19340].

## 3. Twin and dueling components

HiTDMA is not a monolithic RL primitive. Its “Twin” and “Dueling” components arise from two different algorithmic branches.

On the **user side**, HiTDMA uses a **Dueling Double Deep Q Network (D3QN)** for FA port selection. The dueling decomposition is
$$
Q(s_t^u,a_t^u;\theta, \psi, \phi)=V(s_t^u;\theta, \psi)+A(s_t^u,a_t^u;\theta, \phi).
$$
Here the network has a shared feature extractor, a state-value stream, and an action-advantage stream. The paper states that this is used because FA port selection is discrete and because many actions may have similar consequences in a given state. The same branch also incorporates a DDQN-style target:
$$
a_t^*=\arg\max \limits_{a^u} Q(s_{t+1}^u,a_t^u;\theta),
$$
$$
y_t^{\text{D3QN}=r_{t+1}+ \gamma^* Q(s_{t+1}^u,\arg\max \limits_{a^u} Q(s_{t+1}^u,a_t^u;\theta);\theta^-),
$$
with loss
$$
loss_{\text{D3QN} (\theta)=[y_t^{\text{D3QN}-Q(s_t^u,a_t^u;\theta)]^2,
$$
and soft target update
$$
\theta^- = \tau_2 \theta + (1- \tau_2) \theta^-.
$$

On the **BS side**, HiTDMA uses **TD3**, so the “Twin” component refers to the twin-critic architecture used to reduce overestimation bias. The target is
$$
y_t^{\text{TD3}=R_{t+1}+ \gamma^* \min_{m=1,2} Q(s_{t}^{i,j},a_t^{i,j},\theta_{Q_m}^-).
$$
The critic losses are
$$
loss_{\text{TD3}(\theta_{Q_m})=[y_t^{\text{TD3}-Q(s_t^{i,j},a_t^{i,j},\theta_{Q_m})]^2,\quad m=1,2,
$$
and the actor objective is
$$
loss_{\text{TD3}(\theta_{\mu})=Q(s_t^{i,j},a_t^{i,j},\theta_{Q_1}).
$$
Target networks are updated by
$$
\begin{aligned}
\theta_{\mu}^- = \tau_1 \theta_{\mu} + (1- \tau_1) \theta_{\mu}^-, \\
\theta_{Q_1}^-= \tau_1 \theta_{Q_1} + (1- \tau_1) \theta_{Q_1}^-, \\
\theta_{Q_2}^-= \tau_1 \theta_{Q_2} + (1- \tau_1) \theta_{Q_2}^-.
\end{aligned}
$$

The title therefore compresses a hybrid design into a single acronym: **hierarchical** because decision making is split across user-side and BS-side layers, **dueling** because the lower layer uses D3QN, and **twin** because the upper layer uses TD3 [2509.19340].

## 4. Game-theoretic reduction and reinforcement-learning formulation

A central component of HiTDMA is the replacement of direct \(N\)-dimensional power optimization with a **single pricing factor** \(\lambda\). The original power-control problem is modeled as a non-cooperative game
$$
G=\left[\mathbf{U},{\mathbf{P},\left\{f_{ n}^{c}(\cdot)\right\}\right].
$$
The net utility is
$$
f_{ n}^{c}(p_n,{p_n}' )=f_{ n}(p_n,{p_n}' )-c_n(p_n,{p_n}' ),
$$
with
$$
f_{ n}(p_n,{p_n}' )=R_n, \qquad c_n(p_n,{p_n}' )=\xi_n p_n.
$$
From
$$
\frac{\partial f_{ n}^{c}(p_n,{p_n}')}{\partial p_n} = \frac{\phi_n }{\ln2(\tilde{I}_n+\delta_n ^2 +\phi_n p_n )  } - \xi_n ,
$$
the paper derives
$$
p_n=\frac{1}{\xi_n\ln2} -\frac{\tilde{I}_n + \delta _n^2 }{\phi_n}.
$$
The pricing-factor construction leads to the iterative power update
$$
p_n^{\tau+1 }=\frac{\left [ 1- e^{-\lambda(\phi_n\vartheta)^\varphi}\right ](\tilde{I}_n^{(\tau)}+\delta_n ^2) p_n^{(0)} }{\tilde{I}_n^{(\tau)}+\delta_n ^2+e^{-\lambda(\phi_n\vartheta)^\varphi} p_n^{(0)} \phi_n }.
$$
As a result, the original optimization problem is reduced to
$$
\begin{equation}
\begin{aligned}
\mathcal{P} 2: \min_{\{\tilde{\mathbf{d}, \mathbf{W}, \lambda,\mathbf{Z}\} & \max_n t_n \\
\text { s.t. } & \mathrm{C} {3}' : \lambda >0,\\
&  \mathrm{C} 1,\ \mathrm{C} 2,\ \mathrm{C} 4.
\end{aligned}
\end{equation}
$$
The paper treats this reduction as crucial because it allows RL agents to learn one scalar \(\lambda\) rather than an entire power vector.

The control problem is formalized as an MDP or Markov game tuple \((S,A,P,R)\), with states including user locations, channel information, offloading rates, and FA port locations; actions including FA port locations, beamforming, MEC resource allocation, and pricing factor; and rewards tied to delay minimization. Both levels use the same reward:
$$
r_t=\left\{
\begin{array}{ll}
\delta_{1}, & \text { if } T_{s} \leq t_{1}, \\
\frac{\delta_{1}\left(t_{2}-T_{s}\right)}{t_{2}-t_{1}, & \text { if } t_{1}<T_{s} \leq t_{2}, \\
0, & \text { otherwise},
\end{array}
\right.
$$
where
$$
T_{s} = {\max_n} t_n.
$$
The paper characterizes this as a piecewise decreasing function of the maximum user delay: it is highest when the system delay is below target \(t_1\), linearly decays until \(t_2\), and becomes zero beyond that [2509.19340].

## 5. Training pipeline, CSI integration, and empirical behavior

HiTDMA is trained together with **IBM-CCS**, an “Information Bottleneck Metric-enhanced Channel Compressed Sensing” method used to reconstruct CSI under dynamic FA port configuration. The RL state therefore depends on estimated rather than exact channels. Algorithmically, the loop is: initialize environment and DUA/TBA networks; estimate the full channel parameter vector using IBM-CCS from partial CSI; construct \(s_t^u\); let DUAs choose \(a_t^u\); construct \(s_t^b\); let TBAs choose \(a_t^b\); convert \(\lambda\) into \(\mathbf P\) using the game-theoretic power update; execute the joint action; receive the reward and next states; store transitions; update D3QN and TD3 losses; and soft-update target networks. Separate memory queues are used for \([s_t^u,a_t^u,r_t,s_{t+1}^u]\) and \([s_t^b,a_t^b,r_t,s_{t+1}^b]\) [2509.19340].

The reported network and optimization settings are concrete. Both D3QN and TD3 use **three fully connected layers** with hidden sizes **64, 128, 64**. The D3QN branch uses **ReLU**, while the TD3 branch is described as using **Sigmod**. Optimization uses **Adam** with learning rate **0.0001**, gradient clipping threshold **0.25**, and discount factor **0.8**. Exploration decays linearly from **1** to **0.02** over **800 training epochs**, then remains constant. The implementation runs on an **NVIDIA GTX 1660 GPU**. The paper does **not** explicitly report replay memory size, batch size, target update interval, or the TD3 policy-delay parameter.

The empirical results reported for HiTDMA are mainly delay- and reward-oriented. The cumulative reward curve shows high variance during roughly the first **30 iterations**, a sharp rise during iterations **40–70**, and convergence around reward values **88–90**. In delay evaluations, the proposed scheme under **IBM-CCS estimated CSI** remains closest to the **perfect-CSI** case and outperforms **FPA**, **FP**, **ZF**, and **MADDPG**. Delay decreases steadily as FA length increases, per-user delay variance is lower than with MADDPG, and the performance advantage over MADDPG grows with the number of users. The paper presents these findings as evidence that the combination of hierarchy, dueling-double Q-learning, twin critics, game-theoretic power compression, and improved CSI estimation is effective for the target offloading problem [2509.19340].

## 6. Position within the hierarchical MARL literature

HiTDMA occupies a specific niche within recent hierarchical MARL: it is a **hybrid hierarchical MARL framework specialized to FA-assisted MEC offloading**, rather than a generic theory of hierarchy. Several recent papers illuminate which parts of HiTDMA are shared with broader hierarchical RL and which are distinctive.

A close contrast is the realistic air-combat framework in “Coordinated Strategies in Realistic Air Combat by Hierarchical Multi-Agent Reinforcement Learning” [2510.11474]. That work has a genuine hierarchical multi-agent architecture with temporal abstraction through tactical options and CTDE at the commander level, but it does **not** include dueling heads, twin critics, double Q-learning, or value-decomposition mixing in its proposed method. Its central optimizer is **MA-SPO**, not a twin-dueling design. This makes it highly relevant to the **hierarchical multi-agent** part of HiTDMA, but not to the **twin** or **dueling** parts.

“Hierarchical Reinforcement Learning for Optimal Agent Grouping in Cooperative Systems” [2501.06554] is similarly aligned with hierarchy, CTDE, option-based temporal abstraction, and permutation-invariant encoders, but the paper explicitly lacks **twin critics**, **dueling decomposition**, and **double Q-learning**. It is more relevant to grouping-aware hierarchical design than to HiTDMA’s D3QN-TD3 construction. “TAG: A Decentralized Framework for Multi-Agent Hierarchical Reinforcement Learning” [2502.15425] broadens the comparison further: TAG supports **arbitrary depth** hierarchy through the **LevelEnv** abstraction and can host heterogeneous learners at different levels, but it contains **no twin critics** and **no dueling architecture**.

Among the cited related works, “Hierarchical Lead Critic based Multi-Agent Reinforcement Learning” [2602.21680] is the closest to HiTDMA on the **twin** dimension. HLC is a hierarchical multi-critic CTDE method that uses **two lead critics** per group and **clipped double Q-learning**, so it partially matches the “Twin” aspect. However, it has **no dueling architecture** and no manager-worker or option-based temporal hierarchy; its hierarchy is in critic scope rather than in a discrete/continuous action decomposition. “HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism” [2110.07246] offers a different contrast: HAVEN is a two-level value-decomposition method for Dec-POMDPs that stabilizes learning through **advantage-based intrinsic reward** and **dual coordination**, not through twin or dueling critics. The older MOBA hierarchy in “Hierarchical Reinforcement Learning for Multi-agent MOBA Game” [1901.08004] likewise matches the macro/micro organizational idea, but it uses imitation learning for macro strategy and a PPO-like actor-critic at the micro level, without twin critics or dueling decomposition.

Taken together, these comparisons suggest a precise interpretation. HiTDMA is not merely “a hierarchical multi-agent algorithm”; it is a specialized composition of **hierarchy**, **dueling-double Q-learning** for discrete user-side control, and **twin-critic TD3** for continuous BS-side control. A plausible implication is that its most distinctive contribution lies less in inventing a new universal MARL principle than in aligning the hierarchical split with the mixed discrete-continuous structure of FA-assisted MEC offloading, while embedding power-control structure through game theory and channel-estimation structure through IBM-CCS [2509.19340].

Source: https://www.emergentmind.com/topics/hierarchical-twin-dueling-multi-agent-algorithm-hitdma