Meta-TD3: Meta-Learning Enhanced TD3

• Meta-TD3 is a meta-learning augmented variant of TD3 that adds an online meta-critic to guide actor updates.
• It integrates twin-critic mechanisms with a bi-level optimization process to improve stability, sample efficiency, and final performance.
• Empirical studies show Meta-TD3 achieves higher cumulative rewards in tasks like wireless communications and microservice scaling compared to standard TD3 and SAC.

Searching arXiv for the specified paper and closely related Meta-TD3 work. Meta-TD3 denotes a family of Twin Delayed Deep Deterministic Policy Gradient (TD3) methods in which meta-learning is introduced into the actor-critic training loop. In the wireless-communications formulation of "Hybrid FIM and STAR-BD-RIS-Aided Wireless Communications with Short Packet Length: A Meta-TD3 Approach," Meta-TD3 is the learning-based method used to jointly optimize flexible intelligent metasurface (FIM) surface configuration, transmit beamforming, and STAR-BD-RIS phase shifts under short-block-length constraints (Eftekhari et al., 19 Sep 2025). In a broader reinforcement-learning sense, the term is rooted in the online meta-critic framework of "Online Meta-Critic Learning for Off-Policy Actor-Critic Methods," which augments off-policy actor-critic algorithms, including TD3, with a learned auxiliary loss intended to accelerate and improve actor updates (Zhou et al., 2020). Across the literature, the name therefore refers either to a specific TD3 variant with an online meta-critic or, more loosely, to meta-learning-enhanced TD3 pipelines in domain-specific systems.

1. Definition and algorithmic lineage

Meta-TD3 inherits its base structure from TD3, an off-policy deterministic actor-critic algorithm for continuous control. The canonical TD3 substrate consists of twin Q-networks, target policy smoothing, and delayed actor updates. In the online meta-critic formulation, TD3 is extended by a meta-critic network whose output is interpreted as an auxiliary loss for the actor; the meta-critic is explicitly trained to accelerate the learning process, and it is learned online for a single task rather than over a family of tasks (Zhou et al., 2020).

The defining distinction from vanilla TD3 is therefore not the presence of twin critics alone, but the addition of a second learning signal beyond the standard critic. In the 2020 formulation, the actor is still optimized against the ordinary TD3 objective, but the update is supplemented by a meta-learned term. In the 2025 wireless formulation, the same general idea is instantiated as a learnable meta-critic $m_{\varkappa}$ that guides actor updates within a bi-level optimization procedure (Eftekhari et al., 19 Sep 2025).

This usage establishes Meta-TD3 as a meta-learning augmentation of TD3 rather than a separate RL family. A plausible implication is that the term should be understood operationally: TD3 remains the control backbone, while meta-learning modifies how the actor update is produced or how the surrounding optimization pipeline adapts.

2. TD3 substrate and control-theoretic core

The standard TD3 component of Meta-TD3 uses two critics and one deterministic actor. In the general off-policy formulation, the policy is $a=\pi_{\theta}(s)$, while the critic approximates the action-value function $Q_{\omega}(s,a)$ (Zhou et al., 2020). TD3 addresses over-estimation and instability through three mechanisms.

First, twin critics are maintained, and target construction uses the minimum of the two value estimates. In the standard notation of the online meta-critic work, the TD target is

$$y_j = r_j + \gamma \min_{i=1,2} Q_{\bar\omega_i}\!\bigl(s_j', \pi_{\bar\theta}(s_j')+\epsilon\bigr), \quad \epsilon\sim \mathrm{clip}\bigl(\mathcal{N}(0,\sigma^2), -c, c\bigr).$$

Second, target policy smoothing perturbs the target action with clipped noise. Third, delayed actor and target updates update the actor and target networks only once every $d$ critic updates (Zhou et al., 2020).

In the wireless Meta-TD3 formulation, these same TD3 mechanisms appear explicitly as: twin critics $Q_{\psi_1},Q_{\psi_2}$, use of $\min_j Q_{\psi_j}$ to form targets, target-action smoothing via

$$\tilde a' = \pi_{\phi'}(s')+\epsilon,\qquad \epsilon\sim\mathrm{clip}(\mathcal{N}(0,\sigma),-c,c),$$

and delayed actor and target updates (Eftekhari et al., 19 Sep 2025).

The critic target in that system is

$$y = \text{rew} +\gamma\,\min_j Q_{\psi_j'}(s',\tilde a').$$

This places Meta-TD3 squarely within the deterministic-policy, off-policy control regime, with meta-learning modifying the actor-optimization pathway rather than replacing the Bellman-based critic training.

3. Meta-learning mechanisms

Two closely related but not identical meta-learning constructions appear in the cited literature.

In the online meta-critic formulation, the meta-critic is a neural network $h_{\phi}$ whose output is treated as an auxiliary loss for the actor. Its input depends on the current actor parameters and is permutation-invariant over the minibatch. One concrete realization is

$a=\pi_{\theta}(s)$0

where $a=\pi_{\theta}(s)$1 is the actor penultimate-layer representation and $a=\pi_{\theta}(s)$2 is a small 3-layer MLP with two hidden layers of 100 ReLU units and a Softplus output (Zhou et al., 2020).

The meta-objective is bi-level. A training batch produces an intermediate actor update using the standard actor loss and then a second update using the meta-critic loss. A validation batch then measures whether the updated actor improves the critic-derived objective. The paper defines

$a=\pi_{\theta}(s)$3

and updates $a=\pi_{\theta}(s)$4 by automatic differentiation through the one-step unrolling (Zhou et al., 2020).

The 2025 wireless Meta-TD3 paper uses a learnable meta-critic $a=\pi_{\theta}(s)$5 in a structurally similar bi-level procedure. The inner loop updates the two critics and forms an intermediate actor $a=\pi_{\theta}(s)$6 from the critic objective, then produces $a=\pi_{\theta}(s)$7 using the meta-critic objective. The outer loop updates $a=\pi_{\theta}(s)$8 from

$a=\pi_{\theta}(s)$9

with $Q_{\omega}(s,a)$0 and $Q_{\omega}(s,a)$1 sampled as disjoint replay batches (Eftekhari et al., 19 Sep 2025).

The common principle is that the meta-critic is not merely another value estimator. It is trained to shape actor updates so that subsequent validation performance improves. This suggests that Meta-TD3 should be interpreted as a bi-level optimization method layered on top of TD3 rather than a small architectural tweak.

4. Short-block-length wireless communications formulation

In "Hybrid FIM and STAR-BD-RIS-Aided Wireless Communications with Short Packet Length: A Meta-TD3 Approach," Meta-TD3 is used in a multi-user wireless system that deploys an FIM antenna at the base station and a simultaneously transmitting and reflecting beyond diagonal RIS along the transmission path to ensure sufficient received power for single-antenna users (Eftekhari et al., 19 Sep 2025).

The optimization objective is to maximize the sum rate under short-block-length constraints by jointly optimizing FIM element positions $Q_{\omega}(s,a)$2, beamformers $Q_{\omega}(s,a)$3, and STAR-BD-RIS phase shifts $Q_{\omega}(s,a)$4. The paper formulates

$Q_{\omega}(s,a)$5

subject to minimum SINR, total power, FIM-position, and STAR-BD-RIS phase-shift constraints (Eftekhari et al., 19 Sep 2025). The achievable rate for user $Q_{\omega}(s,a)$6 uses a finite-block-length expression: $Q_{\omega}(s,a)$7

The corresponding Markov decision process has state space

$Q_{\omega}(s,a)$8

action space

$Q_{\omega}(s,a)$9

and reward

$$y_j = r_j + \gamma \min_{i=1,2} Q_{\bar\omega_i}\!\bigl(s_j', \pi_{\bar\theta}(s_j')+\epsilon\bigr), \quad \epsilon\sim \mathrm{clip}\bigl(\mathcal{N}(0,\sigma^2), -c, c\bigr).$$0

This reward directly encodes constraint satisfaction rather than introducing a softer penalty construction (Eftekhari et al., 19 Sep 2025).

The reported network architecture uses $$y_j = r_j + \gamma \min_{i=1,2} Q_{\bar\omega_i}\!\bigl(s_j', \pi_{\bar\theta}(s_j')+\epsilon\bigr), \quad \epsilon\sim \mathrm{clip}\bigl(\mathcal{N}(0,\sigma^2), -c, c\bigr).$$1 FIM elements, STAR-BD-RIS size $$y_j = r_j + \gamma \min_{i=1,2} Q_{\bar\omega_i}\!\bigl(s_j', \pi_{\bar\theta}(s_j')+\epsilon\bigr), \quad \epsilon\sim \mathrm{clip}\bigl(\mathcal{N}(0,\sigma^2), -c, c\bigr).$$2, $$y_j = r_j + \gamma \min_{i=1,2} Q_{\bar\omega_i}\!\bigl(s_j', \pi_{\bar\theta}(s_j')+\epsilon\bigr), \quad \epsilon\sim \mathrm{clip}\bigl(\mathcal{N}(0,\sigma^2), -c, c\bigr).$$3 users, and $$y_j = r_j + \gamma \min_{i=1,2} Q_{\bar\omega_i}\!\bigl(s_j', \pi_{\bar\theta}(s_j')+\epsilon\bigr), \quad \epsilon\sim \mathrm{clip}\bigl(\mathcal{N}(0,\sigma^2), -c, c\bigr).$$4 paths. The actor and critics have three fully connected hidden layers of sizes $$y_j = r_j + \gamma \min_{i=1,2} Q_{\bar\omega_i}\!\bigl(s_j', \pi_{\bar\theta}(s_j')+\epsilon\bigr), \quad \epsilon\sim \mathrm{clip}\bigl(\mathcal{N}(0,\sigma^2), -c, c\bigr).$$5. The learning rates are $$y_j = r_j + \gamma \min_{i=1,2} Q_{\bar\omega_i}\!\bigl(s_j', \pi_{\bar\theta}(s_j')+\epsilon\bigr), \quad \epsilon\sim \mathrm{clip}\bigl(\mathcal{N}(0,\sigma^2), -c, c\bigr).$$6, the meta-step is $$y_j = r_j + \gamma \min_{i=1,2} Q_{\bar\omega_i}\!\bigl(s_j', \pi_{\bar\theta}(s_j')+\epsilon\bigr), \quad \epsilon\sim \mathrm{clip}\bigl(\mathcal{N}(0,\sigma^2), -c, c\bigr).$$7, the discount factor is $$y_j = r_j + \gamma \min_{i=1,2} Q_{\bar\omega_i}\!\bigl(s_j', \pi_{\bar\theta}(s_j')+\epsilon\bigr), \quad \epsilon\sim \mathrm{clip}\bigl(\mathcal{N}(0,\sigma^2), -c, c\bigr).$$8, soft-update parameters satisfy $$y_j = r_j + \gamma \min_{i=1,2} Q_{\bar\omega_i}\!\bigl(s_j', \pi_{\bar\theta}(s_j')+\epsilon\bigr), \quad \epsilon\sim \mathrm{clip}\bigl(\mathcal{N}(0,\sigma^2), -c, c\bigr).$$9, replay batch size is $d$0, and the target-action clip parameter is $d$1. Exploration noise is Ornstein-Uhlenbeck or Gaussian, and training continues until convergence at roughly $d$2–$d$3 episodes with update steps $d$4 per episode such as $d$5 (Eftekhari et al., 19 Sep 2025).

The same study reports computational complexity per update of approximately $d$6 due to forward and backward passes in four networks: actor, two critics, and meta-critic. Formal complexity bounds are not provided (Eftekhari et al., 19 Sep 2025).

5. Empirical behavior and domain-specific variants

The wireless short-packet study reports that Meta-TD3 converges to approximately $d$7 higher cumulative reward than SAC. At $d$8 dBm, Meta-TD3 with the hybrid FIM+STAR-BD-RIS configuration achieves about $d$9 bps/Hz, compared with $Q_{\psi_1},Q_{\psi_2}$0 bps/Hz for SAC. Under stricter SINR thresholds, it retains more than $Q_{\psi_1},Q_{\psi_2}$1 of high-SINR throughput, and with increasing numbers of RIS elements it remains approximately $Q_{\psi_1},Q_{\psi_2}$2–$Q_{\psi_1},Q_{\psi_2}$3 above SAC. The simulation parameters include noise power $Q_{\psi_1},Q_{\psi_2}$4 dBm/Hz, error probability $Q_{\psi_1},Q_{\psi_2}$5, and block length $Q_{\psi_1},Q_{\psi_2}$6; benchmarks include SAC, fixed-position antenna, and STAR-RIS only (Eftekhari et al., 19 Sep 2025).

The original online meta-critic study evaluates Meta-TD3, denoted TD3-MC, on seven MuJoCo-Gym v2 tasks, two rllab tasks, and simulated racing in TORCS. TD3-MC’s maximum average return exceeds TD3’s in 6 of 7 Gym tasks, both rllab tasks, and TORCS. The reported learning curves show faster early learning and a higher asymptotic return with lower variance (Zhou et al., 2020).

A separate system-level use of Meta-TD3 appears in "MSARS: A Meta-Learning and Reinforcement Learning Framework for SLO Resource Allocation and Adaptive Scaling for Microservices," where meta-learning is applied to a GCN-based SLO allocator and an improved TD3 model generates adaptive auto-scaling policies. In Kubernetes-based Sock-Shop experiments with 10 versions and 5 chains, the framework reports $Q_{\psi_1},Q_{\psi_2}$7 faster adaptation to new LLP/chain changes versus a non-meta GCN baseline, $Q_{\psi_1},Q_{\psi_2}$8 fewer end-to-end SLO violations compared to standard TD3 without SLO guidance, and $Q_{\psi_1},Q_{\psi_2}$9 reduction in overall resource cost versus vanilla TD3 (Hu et al., 2024).

A further application appears in SIM-assisted MU-MISO downlink optimization, where TD3 is used for joint SIM phase-shift configuration and power allocation. The reported results show that TD3 converges faster and smoother than DDPG, the final sum-rate ordering is TD3 $\min_j Q_{\psi_j}$0 DDPG $\min_j Q_{\psi_j}$1 AO $\min_j Q_{\psi_j}$2 IWF, increasing the number of meta-atoms per layer is beneficial, and performance saturates beyond roughly five layers (Yang et al., 2024).

Context	Meta-TD3 role	Reported outcome
Hybrid FIM + STAR-BD-RIS wireless system	Meta-critic-guided TD3 for joint surface, beamforming, and phase-shift optimization	$\min_j Q_{\psi_j}$3 higher cumulative reward than SAC; $\min_j Q_{\psi_j}$4 bps/Hz at $\min_j Q_{\psi_j}$5 dBm
Online meta-critic continuous control	TD3 actor augmented with online meta-learned auxiliary loss	Exceeds TD3 in 6 of 7 Gym tasks, both rllab tasks, and TORCS
MSARS microservices	Meta-learning for SLO allocation plus improved TD3 for adaptive scaling	$\min_j Q_{\psi_j}$6 faster adaptation, $\min_j Q_{\psi_j}$7 fewer SLO violations, $\min_j Q_{\psi_j}$8 lower resource cost

These results collectively indicate that the strongest empirical case for Meta-TD3 lies in stability, sample-efficiency-related gains, and improved final reward under high-dimensional continuous control. The exact source of the gain, however, differs across papers: sometimes it is an online meta-critic for actor updates, and sometimes it is a broader meta-learning-plus-TD3 system design.

6. Interpretation, limitations, and recurring misconceptions

A recurring misconception is that Meta-TD3 denotes a single universally fixed algorithm. The cited literature does not support that interpretation. In the online meta-critic formulation, Meta-TD3 is specifically TD3 plus a meta-critic that meta-learns an auxiliary actor loss (Zhou et al., 2020). In the FIM and STAR-BD-RIS formulation, Meta-TD3 again refers to TD3 enhanced by a learnable meta-critic trained through bi-level optimization (Eftekhari et al., 19 Sep 2025). By contrast, in MSARS the meta-learning component is attached to the GCN that allocates SLO resources, while TD3 governs adaptive scaling; the name therefore refers to a composite framework rather than solely to a meta-critic actor update (Hu et al., 2024). This suggests that the label is semantically stable at the level of “meta-learning-enhanced TD3,” but not always at the level of a single canonical update rule.

Another misconception is that Meta-TD3 is primarily a convergence-speed method. The evidence is more specific. In the wireless short-block-length system, Meta-TD3 requires more episodes than SAC to converge, but attains a strictly higher final reward and shows improved stability and reduced Q-overestimation; formal complexity bounds are not provided (Eftekhari et al., 19 Sep 2025). In the online meta-critic study, it improves early learning and final performance across continuous-control tasks (Zhou et al., 2020). The available evidence therefore supports a trade-off in which more elaborate optimization may yield better asymptotic performance and stability, but not necessarily fewer training iterations in every setting.

A further limitation is computational. The 2025 wireless formulation explicitly attributes per-update complexity of approximately $\min_j Q_{\psi_j}$9 to forward and backward passes through four networks, and its training loop requires separate meta-training and meta-validation replay batches (Eftekhari et al., 19 Sep 2025). This added structure is intrinsic to the bi-level design. A plausible implication is that Meta-TD3 is most attractive in regimes where the performance gain from better actor shaping offsets the extra optimization overhead.

Overall, Meta-TD3 is best understood as a class of TD3-based methods in which meta-learning is used to improve actor adaptation, actor supervision, or system-level adaptation around the TD3 core. Within that class, the meta-critic formulation provides the clearest algorithmic definition, while recent application papers demonstrate how the same design principle is specialized to wireless communications, microservice orchestration, and intelligent metasurface control.