Hierarchical Twin-Dueling MARL (HiTDMA)

• The paper introduces HiTDMA, a hierarchical twin-dueling multi-agent framework that minimizes the maximum delay in fluid antenna-assisted MEC offloading.
• It decomposes the control problem by using D3QN for discrete FA port selection on the user side and TD3 for continuous beamforming and resource allocation on the base-station side.
• The approach leverages game-theoretic power control and IBM-CCS-based CSI estimation to outperform conventional schemes like MADDPG in both delay and reward performance.

HiTDMA, short for Hierarchical Twin-Dueling Multi-agent Algorithm, is a hierarchical multi-agent deep reinforcement learning scheme introduced for fluid antenna (FA)-assisted mobile edge computing (MEC) uplink offloading, where channel estimation, FA port selection, beamforming, power control, and computation resource allocation are tightly coupled. In the proposed framework, the joint optimization target is to minimize the maximum delay among all users under imperfect channel state information (CSI), mixed discrete-continuous control, and a non-convex system objective. The method is not a generic name for hierarchical MARL; it denotes a specific hybrid design that combines D3QN on the user side, TD3 on the base-station side, and a game theory-assisted reduction of the power-control search space, with CSI supplied by IBM-CCS channel estimation (Ju et al., 16 Sep 2025).

1. Problem domain and optimization target

HiTDMA is formulated for an uplink MEC system consisting of an MEC server, $N$ users, and a base station equipped with $K$ fluid antennas, with $N \le K$. Each user is associated with one FA receiver at the FA-BS, and each FA can switch among $M$ preset positions over a linear space $\mathcal{C}_d$ of length $Wl$, where $l$ is the wavelength. For user $n$, the FA receiver’s antenna position vector is

$$\mathbf{d}_n = \left [ d_1,d_2,\cdot \cdot \cdot,d_{N_p} \right ]^T \in\mathcal{C}_d.$$

The received uplink signal at the FA-BS is written as

$\mathbf{y}=\mathbf{W}^H\mathbf{H}(\tilde{\mathbf{d})\mathbf{P}^{\frac{1}{2}\mathbf{x}+\mathbf{W}^H\mathbf{n}$

where $K$0 is the receive beamforming matrix, $K$1 is the channel matrix depending on FA positions, and $K$2 is the user power matrix. The per-user uplink channel is modeled by

$K$3

with

$K$4

Communication performance enters the control objective through

$K$5

and

$K$6

Delay compares local execution and edge offloading:

$K$7

The total user delay is

$K$8

The global problem is to jointly choose FA antenna position vectors $K$9, receive beamforming matrix $N \le K$0, user transmit power vector $N \le K$1, and MEC computation resource allocation vector $N \le K$2, while minimizing the maximum delay among all users:

$N \le K$3

The paper characterizes this problem as high-dimensional and non-convex because it combines discrete FA port selection with continuous beamforming, power, and MEC allocation, while FA port choice changes the channel itself (Ju et al., 16 Sep 2025).

2. Hierarchical organization of HiTDMA

The term “Hierarchical” in HiTDMA refers to an explicit two-layer decomposition of the control problem. The user-side lower layer handles discrete FA port selection through DUAs, while the BS-side upper layer handles continuous optimization of beamforming, pricing factor, and MEC resource allocation through TBAs. The lower layer acts first, and the upper layer observes an augmented state containing the lower-layer decisions. This dependence is written as

$N \le K$4

The lower-layer user state is

$N \le K$5

where $N \le K$6, $N \le K$7 is user position information, and $N \le K$8 is prior delay information. The user-side action is discrete FA port selection,

$N \le K$9

The BS-side action is continuous:

$M$0

Beamforming weights are split into real and imaginary parts, $M$1 is a scalar pricing factor, and $M$2 is the MEC allocation vector.

This hierarchy is motivated by structural heterogeneity in the control variables. Port selection is discrete and user-local, whereas beamforming, pricing, and computation allocation are continuous and BS-side. A plausible implication is that the hierarchy does not merely reduce action dimensionality; it also permits algorithm-specific treatment of qualitatively different subproblems. In the paper’s implementation, the DUAs and TBAs share information “within each category” to avoid suboptimal convergence, and the overall control loop is coordinated through sequential state construction and joint environment execution (Ju et al., 16 Sep 2025).

3. Twin and dueling components

HiTDMA is not a monolithic RL primitive. Its “Twin” and “Dueling” components arise from two different algorithmic branches.

On the user side, HiTDMA uses a Dueling Double Deep Q Network (D3QN) for FA port selection. The dueling decomposition is

$M$3

Here the network has a shared feature extractor, a state-value stream, and an action-advantage stream. The paper states that this is used because FA port selection is discrete and because many actions may have similar consequences in a given state. The same branch also incorporates a DDQN-style target:

$M$4

$M$5

with loss

$M$6

and soft target update

$M$7

On the BS side, HiTDMA uses TD3, so the “Twin” component refers to the twin-critic architecture used to reduce overestimation bias. The target is

$M$8

The critic losses are

$M$9

and the actor objective is

$\mathcal{C}_d$0

Target networks are updated by

$\mathcal{C}_d$1

The title therefore compresses a hybrid design into a single acronym: hierarchical because decision making is split across user-side and BS-side layers, dueling because the lower layer uses D3QN, and twin because the upper layer uses TD3 (Ju et al., 16 Sep 2025).

4. Game-theoretic reduction and reinforcement-learning formulation

A central component of HiTDMA is the replacement of direct $\mathcal{C}_d$2-dimensional power optimization with a single pricing factor $\mathcal{C}_d$3. The original power-control problem is modeled as a non-cooperative game

$\mathcal{C}_d$4

The net utility is

$\mathcal{C}_d$5

with

$\mathcal{C}_d$6

From

$\mathcal{C}_d$7

the paper derives

$\mathcal{C}_d$8

The pricing-factor construction leads to the iterative power update

$\mathcal{C}_d$9

As a result, the original optimization problem is reduced to

$Wl$0

The paper treats this reduction as crucial because it allows RL agents to learn one scalar $Wl$1 rather than an entire power vector.

The control problem is formalized as an MDP or Markov game tuple $Wl$2, with states including user locations, channel information, offloading rates, and FA port locations; actions including FA port locations, beamforming, MEC resource allocation, and pricing factor; and rewards tied to delay minimization. Both levels use the same reward:

$Wl$3

where

$Wl$4

The paper characterizes this as a piecewise decreasing function of the maximum user delay: it is highest when the system delay is below target $Wl$5, linearly decays until $Wl$6, and becomes zero beyond that (Ju et al., 16 Sep 2025).

5. Training pipeline, CSI integration, and empirical behavior

HiTDMA is trained together with IBM-CCS, an “Information Bottleneck Metric-enhanced Channel Compressed Sensing” method used to reconstruct CSI under dynamic FA port configuration. The RL state therefore depends on estimated rather than exact channels. Algorithmically, the loop is: initialize environment and DUA/TBA networks; estimate the full channel parameter vector using IBM-CCS from partial CSI; construct $Wl$7; let DUAs choose $Wl$8; construct $Wl$9; let TBAs choose $l$0; convert $l$1 into $l$2 using the game-theoretic power update; execute the joint action; receive the reward and next states; store transitions; update D3QN and TD3 losses; and soft-update target networks. Separate memory queues are used for $l$3 and $l$4 (Ju et al., 16 Sep 2025).

The reported network and optimization settings are concrete. Both D3QN and TD3 use three fully connected layers with hidden sizes 64, 128, 64. The D3QN branch uses ReLU, while the TD3 branch is described as using Sigmod. Optimization uses Adam with learning rate 0.0001, gradient clipping threshold 0.25, and discount factor 0.8. Exploration decays linearly from 1 to 0.02 over 800 training epochs, then remains constant. The implementation runs on an NVIDIA GTX 1660 GPU. The paper does not explicitly report replay memory size, batch size, target update interval, or the TD3 policy-delay parameter.

The empirical results reported for HiTDMA are mainly delay- and reward-oriented. The cumulative reward curve shows high variance during roughly the first 30 iterations, a sharp rise during iterations 40–70, and convergence around reward values 88–90. In delay evaluations, the proposed scheme under IBM-CCS estimated CSI remains closest to the perfect-CSI case and outperforms FPA, FP, ZF, and MADDPG. Delay decreases steadily as FA length increases, per-user delay variance is lower than with MADDPG, and the performance advantage over MADDPG grows with the number of users. The paper presents these findings as evidence that the combination of hierarchy, dueling-double Q-learning, twin critics, game-theoretic power compression, and improved CSI estimation is effective for the target offloading problem (Ju et al., 16 Sep 2025).

6. Position within the hierarchical MARL literature

HiTDMA occupies a specific niche within recent hierarchical MARL: it is a hybrid hierarchical MARL framework specialized to FA-assisted MEC offloading, rather than a generic theory of hierarchy. Several papers illuminate which parts of HiTDMA are shared with broader hierarchical RL and which are distinctive.

A close contrast is the realistic air-combat framework in “Coordinated Strategies in Realistic Air Combat by Hierarchical Multi-Agent Reinforcement Learning” (Selmonaj et al., 13 Oct 2025). That work has a genuine hierarchical multi-agent architecture with temporal abstraction through tactical options and CTDE at the commander level, but it does not include dueling heads, twin critics, double Q-learning, or value-decomposition mixing in its proposed method. Its central optimizer is MA-SPO, not a twin-dueling design. This makes it highly relevant to the hierarchical multi-agent part of HiTDMA, but not to the twin or dueling parts.

“Hierarchical Reinforcement Learning for Optimal Agent Grouping in Cooperative Systems” (Hu, 11 Jan 2025) is similarly aligned with hierarchy, CTDE, option-based temporal abstraction, and permutation-invariant encoders, but the paper explicitly lacks twin critics, dueling decomposition, and double Q-learning. It is more relevant to grouping-aware hierarchical design than to HiTDMA’s D3QN-TD3 construction. “TAG: A Decentralized Framework for Multi-Agent Hierarchical Reinforcement Learning” (Paolo et al., 21 Feb 2025) broadens the comparison further: TAG supports arbitrary depth hierarchy through the LevelEnv abstraction and can host heterogeneous learners at different levels, but it contains no twin critics and no dueling architecture.

Among the cited related works, “Hierarchical Lead Critic based Multi-Agent Reinforcement Learning” (Eckel et al., 25 Feb 2026) is the closest to HiTDMA on the twin dimension. HLC is a hierarchical multi-critic CTDE method that uses two lead critics per group and clipped double Q-learning, so it partially matches the “Twin” aspect. However, it has no dueling architecture and no manager-worker or option-based temporal hierarchy; its hierarchy is in critic scope rather than in a discrete/continuous action decomposition. “HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism” (Xu et al., 2021) offers a different contrast: HAVEN is a two-level value-decomposition method for Dec-POMDPs that stabilizes learning through advantage-based intrinsic reward and dual coordination, not through twin or dueling critics. The older MOBA hierarchy in “Hierarchical Reinforcement Learning for Multi-agent MOBA Game” (Zhang et al., 2019) likewise matches the macro/micro organizational idea, but it uses imitation learning for macro strategy and a PPO-like actor-critic at the micro level, without twin critics or dueling decomposition.

Taken together, these comparisons suggest a precise interpretation. HiTDMA is not merely “a hierarchical multi-agent algorithm”; it is a specialized composition of hierarchy, dueling-double Q-learning for discrete user-side control, and twin-critic TD3 for continuous BS-side control. A plausible implication is that its most distinctive contribution lies less in inventing a new universal MARL principle than in aligning the hierarchical split with the mixed discrete-continuous structure of FA-assisted MEC offloading, while embedding power-control structure through game theory and channel-estimation structure through IBM-CCS (Ju et al., 16 Sep 2025).