Model-Assisted MA-CSAC: Constrained Multi-Agent RL

• The paper introduces MA-CSAC, a constrained RL approach that reformulates a non-convex satellite optimization problem as a CMDP to boost network performance.
• MA-CSAC employs a multi-agent actor-critic framework with heterogeneous policies (Gaussian, von Mises, Beta) tailored for continuous, circular, and bounded control variables.
• Experiments demonstrate MA-CSAC achieves the highest long-term spectral efficiency and favorable SE-EE trade-off in large-scale satellite and SIM-integrated systems.

Searching arXiv for the specified paper and related context. Model-Assisted Multi-Agent Constraint Soft Actor-Critic (MA-CSAC) is a constrained multi-agent reinforcement-learning method introduced as one of three optimization approaches for the joint beamforming, ASIM tuning, and backscatter allocation problem in an active SIM-equipped low Earth orbit satellite system serving multiple ground users via RSMA and IoT devices through a symbiotic radio network. In the underlying study, the method is positioned alongside BCD-SCA and MCPPO, with MA-CSAC reported to attain the highest long-term spectral and energy efficiency in large-scale networks, while incurring longer training times than MCPPO and lacking the deterministic convergence behavior of BCD-SCA in convex scenarios (Yeganeh et al., 23 Aug 2025).

1. System role and optimization context

MA-CSAC is formulated for the non-convex joint optimization problem that couples the satellite precoding matrix, ASIM phase-shifts, ASIM amplitude gains, backscatter-related variables, common-rate splits, and efficiency trade-off terms. The broader communication architecture includes an active stacked intelligent metasurface mounted on the backplate of the satellite solar panels, rate-splitting multiple access for ground users, and a symbiotic radio network for IoT devices. Within that setting, the paper evaluates three optimization approaches: block coordinate descent with successive convex approximation (BCD-SCA), model-assisted multi-agent constraint soft actor-critic (MA-CSAC), and multi-constraint proximal policy optimization (MCPPO) (Yeganeh et al., 23 Aug 2025).

The specific role of MA-CSAC is to address the original non-convex program by recasting it as a constrained multi-agent Markov Decision Process (CMDP). This places the method in the intersection of constrained RL, entropy-regularized actor-critic learning, and CTDE-based multi-agent coordination. A plausible implication is that the method is intended for operating regimes where deterministic convex approximations are either insufficient or difficult to maintain as the network scales.

2. CMDP formulation

The method casts the optimization problem into a CMDP

$$(\mathcal S,\mathcal A,\mathcal P,\,r,\{\mathcal C_j,\bar c_j\}_{j=1}^J,\gamma)\,.$$

The state space is defined at time $t$ as

$$s_t=\big\{\mathbf F(t),\{\mathbf H^{(q)}(t)\},\{\mathbf g_\ell(t)\},\{\mathbf h_i(t)\},\{\tau_i(t)\}\big\},$$

where the global state contains the time-varying satellite-SIM channel $\mathbf F$, inter-layer SIM couplings $\{\mathbf H^{(q)}\}$, SIM-user channels $\{\mathbf g_\ell\}$, SBD harvesting channels $\{\mathbf h_i\}$, and current backscatter time-split $\{\tau_i\}$ (Yeganeh et al., 23 Aug 2025).

The joint action is

$$a_t = \{\mathbf W_t,\{\Phi^{(q)}_t\},\boldsymbol\tau_t,\boldsymbol\eta_t,\boldsymbol\sigma_t,\mathbf C_t,\alpha,\beta,\vartheta_{\rm sat},\vartheta_{\rm SIM}\},$$

comprising the satellite precoder $\mathbf W$, SIM-tuning matrices $t$0, BD-EH/BD durations $t$1, reflection coefficients $t$2, common-rate splits $t$3, the trade-off weights $t$4, and PA-efficiency factors $t$5.

The one-step reward is Lagrangian-penalized: $t$6 with $t$7, $t$8, and $t$9 taken from the system model. The costs $$s_t=\big\{\mathbf F(t),\{\mathbf H^{(q)}(t)\},\{\mathbf g_\ell(t)\},\{\mathbf h_i(t)\},\{\tau_i(t)\}\big\},$$0 encode instantaneous constraint violations such as power limits and QoS bounds.

The constraints are represented as $$s_t=\big\{\mathbf F(t),\{\mathbf H^{(q)}(t)\},\{\mathbf g_\ell(t)\},\{\mathbf h_i(t)\},\{\tau_i(t)\}\big\},$$1, with examples including

$$s_t=\big\{\mathbf F(t),\{\mathbf H^{(q)}(t)\},\{\mathbf g_\ell(t)\},\{\mathbf h_i(t)\},\{\tau_i(t)\}\big\},$$2

total SIM power $$s_t=\big\{\mathbf F(t),\{\mathbf H^{(q)}(t)\},\{\mathbf g_\ell(t)\},\{\mathbf h_i(t)\},\{\tau_i(t)\}\big\},$$3, min-rate constraints, and the equality condition $$s_t=\big\{\mathbf F(t),\{\mathbf H^{(q)}(t)\},\{\mathbf g_\ell(t)\},\{\mathbf h_i(t)\},\{\tau_i(t)\}\big\},$$4. Each such constraint is enforced through a Lagrange multiplier $$s_t=\big\{\mathbf F(t),\{\mathbf H^{(q)}(t)\},\{\mathbf g_\ell(t)\},\{\mathbf h_i(t)\},\{\tau_i(t)\}\big\},$$5. This formulation makes the algorithm explicitly constraint-aware rather than purely reward-maximizing.

3. Multi-agent decomposition and CTDE architecture

MA-CSAC employs three agent types under a centralized-training decentralized-execution paradigm:

• TX-agent: controlling the satellite precoding matrix $$s_t=\big\{\mathbf F(t),\{\mathbf H^{(q)}(t)\},\{\mathbf g_\ell(t)\},\{\mathbf h_i(t)\},\{\tau_i(t)\}\big\},$$6 and power splits $$s_t=\big\{\mathbf F(t),\{\mathbf H^{(q)}(t)\},\{\mathbf g_\ell(t)\},\{\mathbf h_i(t)\},\{\tau_i(t)\}\big\},$$7.
• SIM-phase agent: controlling the ASIM phase-shifts $$s_t=\big\{\mathbf F(t),\{\mathbf H^{(q)}(t)\},\{\mathbf g_\ell(t)\},\{\mathbf h_i(t)\},\{\tau_i(t)\}\big\},$$8.
• SIM-gain agent: controlling ASIM amplitude gains $$s_t=\big\{\mathbf F(t),\{\mathbf H^{(q)}(t)\},\{\mathbf g_\ell(t)\},\{\mathbf h_i(t)\},\{\tau_i(t)\}\big\},$$9.

Training is centralized through a single replay buffer $\mathbf F$0 that stores joint transitions $\mathbf F$1. The critic is centralized and takes full $\mathbf F$2 as input, whereas the actors are decentralized in the sense that each agent outputs only its portion of the joint action. Execution is decentralized: each agent samples from its own policy $\mathbf F$3 (Yeganeh et al., 23 Aug 2025).

This decomposition is technically significant because it aligns heterogeneous control variables with distinct policy families. Beamforming and power variables are treated differently from phase angles and amplitude gains, which is consistent with their domains and constraints. This suggests that the method is tailored to the mixed continuous-and-structured action space induced by satellite precoding, metasurface tuning, and backscatter operation.

4. Model assistance, entropy regularization, and constraint handling

The paper gives a specific interpretation of the phrase “model-assisted.” It states that MA-CSAC does not rely on separate surrogate predictors of the environment; rather, the “model assistance” refers to the use of explicit channel/state knowledge in the centralized critic to guide policy updates. No additional learned surrogate model is introduced beyond the actor-critic networks (Yeganeh et al., 23 Aug 2025).

This clarification addresses a likely misconception. In many RL contexts, “model-assisted” can imply model-based rollouts, learned world models, or auxiliary predictors. Here, the term instead denotes critic-side use of explicit channel/state knowledge. The method therefore remains within an actor-critic framework rather than introducing a separate surrogate environment.

Constraint handling is implemented through Lagrangian relaxation. The augmented objective is

$\mathbf F$4

where $\mathbf F$5 is the entropy bonus. The multipliers are updated by projected gradient ascent on constraint slack: $\mathbf F$6 with $\mathbf F$7 the average cost for constraint $\mathbf F$8 in the sampled mini-batch and $\mathbf F$9 a small tolerance.

The use of entropy regularization places MA-CSAC in the SAC family, while the explicit multiplier updates make it a constrained variant. A plausible implication is that feasibility management and exploration are coupled rather than treated as separate stages.

5. SAC extensions and learning dynamics

MA-CSAC uses a twin-Q centralized critic and stochastic per-agent policies. The soft Q-function is defined through two Q-networks $\{\mathbf H^{(q)}\}$0, with

$\{\mathbf H^{(q)}\}$1

which is used to reduce overestimation (Yeganeh et al., 23 Aug 2025).

The critic training objective is the Bellman-residual

$\{\mathbf H^{(q)}\}$2

where

$\{\mathbf H^{(q)}\}$3

While standard SAC uses an explicit value network, here the value function is computed implicitly as

$\{\mathbf H^{(q)}\}$4

Each agent $\{\mathbf H^{(q)}\}$5 uses a stochastic policy $\{\mathbf H^{(q)}\}$6 chosen according to the variable type:

• Gaussian for continuous beamforming and power.
• von Mises for phase angles $\{\mathbf H^{(q)}\}$7.
• Beta for SIM gains $\{\mathbf H^{(q)}\}$8.

The actor update is

$\{\mathbf H^{(q)}\}$9

The entropy coefficient $\{\mathbf g_\ell\}$0 is automatically tuned by minimizing

$\{\mathbf g_\ell\}$1

These design choices give the method a heterogeneous action-modeling structure: Euclidean variables, circular variables, and bounded interval variables are each assigned a policy family compatible with their domains. This suggests that the algorithm is not merely “multi-agent SAC” in a generic sense, but a domain-specific constrained SAC construction adapted to mixed-geometry control variables.

6. Training procedure, hyperparameters, and reported behavior

The training procedure initializes actor parameters $\{\mathbf g_\ell\}$2, critic parameters $\{\mathbf g_\ell\}$3, Lagrange multipliers $\{\mathbf g_\ell\}$4, target networks $\{\mathbf g_\ell\}$5, and replay buffer $\{\mathbf g_\ell\}$6. During each episode, each agent samples its action from its current policy, the joint action is executed, and the tuple $\{\mathbf g_\ell\}$7 is stored in $\{\mathbf g_\ell\}$8. Gradient steps then perform critic updates, actor updates, soft target updates

$\{\mathbf g_\ell\}$9

and multiplier updates

$\{\mathbf h_i\}$0

The reported key hyperparameters are:

• $\{\mathbf h_i\}$1
• $\{\mathbf h_i\}$2
• $\{\mathbf h_i\}$3
• Entropy $\{\mathbf h_i\}$4: “auto”
• Replay buffer: $\{\mathbf h_i\}$5
• Batch size $\{\mathbf h_i\}$6
• $\{\mathbf h_i\}$7
• $\{\mathbf h_i\}$8

The reported performance summary distinguishes convergence, spectral efficiency, energy efficiency, and scalability. For convergence, MCPPO converges faster early, by approximately 200 episodes, but with higher variance; MA-CSAC converges more slowly, at approximately 400 episodes, but reaches the highest stable average reward, approximately 150, and the lowest variance; BCD-SCA deterministically converges in 9 iterations to objective approximately 130 (Yeganeh et al., 23 Aug 2025).

For spectral efficiency as a function of the number of ASIM elements $\{\mathbf h_i\}$9, all methods improve SE with $\{\tau_i\}$0, and MA-CSAC achieves the highest SE, approximately $\{\tau_i\}$1 bps/Hz at $\{\tau_i\}$2, compared with BCD-SCA at approximately $\{\tau_i\}$3 bps/Hz and MCPPO at approximately $\{\tau_i\}$4 bps/Hz. For energy efficiency versus $\{\tau_i\}$5, EE grows and then saturates; BCD-SCA attains the highest peak EE, approximately $\{\tau_i\}$6 Mbps/J, MA-CSAC is next at approximately $\{\tau_i\}$7 Mbps/J, and MCPPO is approximately $\{\tau_i\}$8 Mbps/J. For the SE-EE trade-off and scalability, when $\{\tau_i\}$9 users, MA-CSAC maintains approximately $$a_t = \{\mathbf W_t,\{\Phi^{(q)}_t\},\boldsymbol\tau_t,\boldsymbol\eta_t,\boldsymbol\sigma_t,\mathbf C_t,\alpha,\beta,\vartheta_{\rm sat},\vartheta_{\rm SIM}\},$$0–$$a_t = \{\mathbf W_t,\{\Phi^{(q)}_t\},\boldsymbol\tau_t,\boldsymbol\eta_t,\boldsymbol\sigma_t,\mathbf C_t,\alpha,\beta,\vartheta_{\rm sat},\vartheta_{\rm SIM}\},$$1 bps/Hz at approximately $$a_t = \{\mathbf W_t,\{\Phi^{(q)}_t\},\boldsymbol\tau_t,\boldsymbol\eta_t,\boldsymbol\sigma_t,\mathbf C_t,\alpha,\beta,\vartheta_{\rm sat},\vartheta_{\rm SIM}\},$$2 Mbps/J, outperforming both MCPPO and BCD-SCA across the full SE range (Yeganeh et al., 23 Aug 2025).

Taken together, these results establish a specific profile for MA-CSAC. It is not the fastest method in early learning, nor the highest-peak method for the single EE statistic reported against $$a_t = \{\mathbf W_t,\{\Phi^{(q)}_t\},\boldsymbol\tau_t,\boldsymbol\eta_t,\boldsymbol\sigma_t,\mathbf C_t,\alpha,\beta,\vartheta_{\rm sat},\vartheta_{\rm SIM}\},$$3, and it does not inherit the deterministic convergence of BCD-SCA. Its reported strength is highest stable long-term reward and superior spectral-efficiency and SE-EE trade-off behavior in large-scale networks. A plausible implication is that the method is best interpreted as a scalable, constraint-aware RL optimizer for large satellite-ASIM-SR settings rather than as a universally dominant replacement for deterministic solvers.