Multi-User PPO (MUPPO) for UAV Video Streaming

• Multi-User PPO (MUPPO) is a PPO-based reinforcement learning method that jointly optimizes UAV flight paths, transmit power, and bitrate for semantic video streaming.
• It employs an actor-critic framework with multi-threaded updates to aggregate experiences across UAVs, ensuring stable and fast convergence.
• Empirical results show that, compared to D3QN, the joint MUPPO framework improves overall QoE by over 13% under challenging bandwidth and channel conditions.

Multi-User Proximal Policy Optimization (MUPPO) denotes a PPO-based deep reinforcement learning method introduced for a multi-UAV semantic video transmission system, where it serves as the control layer that jointly optimizes UAV flight trajectories, transmission power allocation, and bitrate selection to maximize long-term Quality of Experience (QoE) under dynamic wireless conditions. In the cited formulation, MUPPO is integrated with the semantic self-correcting video transmission framework SSCV-G, whose bitrate control is realized by selectively transmitting a subset of semantic indices and whose receiver employs ST-ViT for reconstruction of dropped semantic indices. The term is application-specific rather than a universally standardized name for a single PPO variant; related literature more commonly uses labels such as MAPPO, coordinated PPO, or full-pipeline PPO for multi-entity PPO extensions (Chen et al., 9 Jul 2025).

1. Definition and terminological scope

In its original formulation, MUPPO is the paper’s deep reinforcement learning method for solving a non-convex QoE optimization problem in a multi-UAV real-time semantic video streaming system. The controller operates inside SSCV-G and is used to jointly optimize UAV flight trajectories, transmit power allocation, and semantic/video bitrate selection. The system context is explicitly multi-UAV and multi-user: multiple UAVs capture and stream aerial video, SSCV-G provides fine-grained bitrate control by selectively transmitting a subset of semantic indices, ST-ViT reconstructs dropped or missing semantic indices, and MUPPO decides communication and streaming actions under varying channel conditions (Chen et al., 9 Jul 2025).

The literature around PPO extensions shows that the name “MUPPO” is not universal. Closely related work instead formalizes cooperative multi-agent PPO under centralized training and decentralized execution, coordinated step-size adaptation, or parameter-sharing-versatile pipelines. “Muti-Agent Proximal Policy Optimization For Data Freshness in UAV-assisted Networks” formulates a MAPPO-based AoU minimization method for multiple UAVs; “Coordinated Proximal Policy Optimization” extends PPO to cooperative Dec-POMDPs via coordinated step-size adaptation; and “FP3O: Enabling Proximal Policy Optimization in Multi-Agent Cooperation with Parameter-Sharing Versatility” develops a trust-region-style cooperative multi-agent PPO that supports full, partial, and non-parameter sharing (Ndiaye et al., 2023, Wu et al., 2021, Feng et al., 2023).

This suggests that MUPPO is best understood as an application-specific designation for a PPO-based controller in a coupled multi-user or multi-UAV environment, rather than as a canonical algorithmic family with a single agreed formalism. In the cited MUPPO paper, the “multi-user” aspect is operationalized through a network-wide QoE objective, inter-UAV interference, common slot delay, and aggregated interaction data during policy updates, not through an explicitly specified CTDE multi-agent architecture (Chen et al., 9 Jul 2025).

2. Optimization problem and system model

The MUPPO formulation is built on a constrained long-horizon optimization problem over $M$ UAVs, indexed by

$m \in \mathcal{M} = \{1,2,\dots,M\}.$

Video transmission lasts for $T$ time slots, each of duration $\hat t$. The position of UAV $m$ at time $t$ is

$U_t(x_m,y_m,z_m),$

and evolves as

$U_{t+1}(x_m,y_m,z_m)=U_t(x_m,y_m,z_m)+\Vec{a}_m^t(x,y,z),$

where $\Vec{a}_m^t(x,y,z)$ is the UAV’s flight vector. The UAV-to-ground channel coefficient is $h_m^t$, and the downlink transmission rate is

$m \in \mathcal{M} = \{1,2,\dots,M\}.$0

with bandwidth $m \in \mathcal{M} = \{1,2,\dots,M\}.$1, transmit power $m \in \mathcal{M} = \{1,2,\dots,M\}.$2, noise PSD $m \in \mathcal{M} = \{1,2,\dots,M\}.$3, and inter-UAV interference $m \in \mathcal{M} = \{1,2,\dots,M\}.$4. Per-slot transmission delay is

$m \in \mathcal{M} = \{1,2,\dots,M\}.$5

where $m \in \mathcal{M} = \{1,2,\dots,M\}.$6 is the transmitted video chunk size, and since all UAV streams are processed in parallel, total slot delay is

$m \in \mathcal{M} = \{1,2,\dots,M\}.$7

These definitions establish the coupling between mobility, radio resource allocation, and bitrate control (Chen et al., 9 Jul 2025).

QoE is defined as

$m \in \mathcal{M} = \{1,2,\dots,M\}.$8

where

$m \in \mathcal{M} = \{1,2,\dots,M\}.$9

The three terms are a quality reward, a bitrate smoothness penalty, and a delay or rebuffer penalty. The paper does not provide numerical values for $T$0, and QoE does not explicitly include fairness terms, packet loss penalties, semantic reconstruction quality metrics, or energy cost terms (Chen et al., 9 Jul 2025).

The optimization problem is

$T$1

subject to

$T$2

$T$3

$T$4

$T$5

$T$6

The paper explicitly characterizes $T$7 as a non-convex mixed-integer nonlinear programming problem (Chen et al., 9 Jul 2025).

3. MDP formulation and PPO objective

The optimization is reformulated as an MDP

$T$8

For UAV $T$9 at time slot $\hat t$0, the state is

$\hat t$1

and the action is

$\hat t$2

The immediate reward is

$\hat t$3

and the discounted return is

$\hat t$4

with trajectory

$\hat t$5

The paper reports $\hat t$6, $\hat t$7, and a slot duration of $\hat t$8 s (Chen et al., 9 Jul 2025).

The PPO formulation begins with importance sampling under the old policy: $\hat t$9 The advantage is defined by

$m$0

with the TD-form

$m$1

MUPPO uses generalized advantage estimation: $m$2 where

$m$3

The clipped surrogate is

$m$4

where

$m$5

The clipping function is

$m$6

The critic loss is

$m$7

Actor and critic are optimized separately: $m$8

$m$9

The manuscript reports a notation inconsistency between $t$0 in the equation and policy entropy bonus weight $t$1 in Table I; the safely stated experimental entropy weight is $t$2 (Chen et al., 9 Jul 2025).

4. Algorithmic characteristics and relation to other PPO extensions

The paper characterizes MUPPO as a PPO-based multi-user controller trained from aggregated experiences across multiple UAV or user interactions, using actor and critic networks and optimizing a network-wide QoE reward rather than isolated per-user rewards. It further states that a multi-threaded environment-interaction mechanism is used, in which each thread independently collects environment and reward data and all interaction information is aggregated during policy updates. The main multi-user extension is therefore not a new PPO objective, but the combination of a coupled multi-UAV QoE objective, a multi-user or multi-UAV environment with interference and shared delay impact, and multi-threaded trajectory aggregation (Chen et al., 9 Jul 2025).

This distinguishes MUPPO from several other PPO-derived lines. Standard PPO is a first-order on-policy method based on the clipped surrogate objective and repeated minibatch optimization on on-policy data (Schulman et al., 2017). “Truly Proximal Policy Optimization” argues that standard clipping neither strictly bounds the likelihood ratio nor enforces a true trust region, and proposes rollback and trust-region-triggered clipping to recover a more principled proximal update (Wang et al., 2019). “Coordinated Proximal Policy Optimization” introduces coordinated step-size adaptation across agents in cooperative MARL, while FP3O formulates a full-pipeline paradigm for cooperative MARL compatible with full, partial, and non-parameter sharing (Wu et al., 2021, Feng et al., 2023).

Against that background, the cited MUPPO paper is comparatively modest in architectural specification. It shows an actor-critic framework and speaks of “the” actor network and “the” critic network, but it does not explicitly specify whether there is one actor per UAV, a shared actor across UAVs, a centralized joint-action policy, or a centralized critic. It likewise does not specify number of hidden layers, hidden dimensions, activations, output parameterization, optimizer, batch size, PPO epochs, or the numerical episode length $t$3. This suggests that the paper should be read primarily as an application-level PPO deployment in a coupled multi-user wireless system, not as a fully formalized general-purpose multi-agent PPO framework (Chen et al., 9 Jul 2025).

5. Empirical performance and reported operating regime

The reported experimental configuration uses $t$4 UAVs, bandwidth $t$5, noise PSD $t$6, target region radius $t$7, flight height range $t$8, flight vector $t$9, transmit power $U_t(x_m,y_m,z_m),$0, learning rate $U_t(x_m,y_m,z_m),$1, $U_t(x_m,y_m,z_m),$2, $U_t(x_m,y_m,z_m),$3, $U_t(x_m,y_m,z_m),$4, policy entropy bonus weight $U_t(x_m,y_m,z_m),$5, and time slot $U_t(x_m,y_m,z_m),$6. The MUPPO evaluation compares four schemes: MUPPO + SSCV-G, D3QN + SSCV-G, MUPPO + SVC, and D3QN + SVC (Chen et al., 9 Jul 2025).

The paper reports that, compared with D3QN, MUPPO shows faster convergence and more stable convergence. The stated reasons are multi-threaded environment interaction with aggregated updates, policy-gradient optimization of cumulative return, and clipping-based stabilized policy updates. Under bandwidth degradation, when bandwidth drops to $U_t(x_m,y_m,z_m),$7, QoE of MUPPO + SSCV-G decreases by only $U_t(x_m,y_m,z_m),$8, while competing methods decrease by more than $U_t(x_m,y_m,z_m),$9. Under channel degradation, when the Rician factor drops to $U_{t+1}(x_m,y_m,z_m)=U_t(x_m,y_m,z_m)+\Vec{a}_m^t(x,y,z),$0, QoE of MUPPO + SSCV-G decreases by only $U_{t+1}(x_m,y_m,z_m)=U_t(x_m,y_m,z_m)+\Vec{a}_m^t(x,y,z),$1, while competing benchmarks decrease by more than $U_{t+1}(x_m,y_m,z_m)=U_t(x_m,y_m,z_m)+\Vec{a}_m^t(x,y,z),$2. In the conclusion, the joint SSCV-G + MUPPO framework is reported to improve overall user QoE by $U_{t+1}(x_m,y_m,z_m)=U_t(x_m,y_m,z_m)+\Vec{a}_m^t(x,y,z),$3 compared to competitive baselines (Chen et al., 9 Jul 2025).

These reported gains must be interpreted with care. The strongest quantitative statements are attributed to the combined framework rather than to MUPPO in isolation. The comparative results also isolate codec choice and reinforcement-learning choice only partially: MUPPO is evaluated alongside SSCV-G and SVC, and the best-performing configuration is the joint MUPPO + SSCV-G system. A plausible implication is that the reported QoE gains arise from the interaction between fine-grained semantic bitrate control, ST-ViT-based loss recovery, and policy optimization, rather than from a standalone change in PPO machinery (Chen et al., 9 Jul 2025).

6. Limitations, ambiguities, and broader significance

The paper leaves several implementation-critical aspects unspecified. It does not clarify whether power is continuous, bitrate is discrete or continuous, or trajectory vectors are discrete or continuous, and it does not explain how PPO outputs are mapped to valid control variables. It also does not provide the neural network architecture, batch size, number of PPO epochs, episode length $U_{t+1}(x_m,y_m,z_m)=U_t(x_m,y_m,z_m)+\Vec{a}_m^t(x,y,z),$4, exact reward weights $U_{t+1}(x_m,y_m,z_m)=U_t(x_m,y_m,z_m)+\Vec{a}_m^t(x,y,z),$5, or a scalability analysis for larger numbers of UAVs or users. The multi-user coordination structure is similarly unclear: centralized joint policy, shared actor, one actor per UAV, CTDE, and centralized critic are all unstated (Chen et al., 9 Jul 2025).

A second source of ambiguity is terminological. In adjacent wireless-networking literature, essentially similar control patterns are often described as MAPPO rather than MUPPO. For instance, the UAV data-freshness work on AoU uses decentralized actors, a centralized value function, and a shared global reward under CTDE; CoPPO formalizes coordinated step-size adaptation; FP3O derives a shared lower bound compatible with diverse parameter-sharing configurations. By contrast, the cited MUPPO paper does not formalize these design choices, even though its problem is similarly multi-UAV and coupled (Ndiaye et al., 2023, Wu et al., 2021, Feng et al., 2023).

The broader significance of MUPPO therefore lies less in introducing a new universally adopted PPO variant than in showing how PPO can be embedded as a high-level controller in a coupled semantic communication system. Its concrete contribution is a formulation where a single QoE objective couples mobility, interference-aware transmission rate, bitrate smoothness, and delay, and where the action space simultaneously spans communication variables, semantic bitrate control, and flight control. This suggests that in multi-user wireless systems, “multi-user PPO” may often denote not a novel surrogate loss, but a PPO-based control architecture operating over coupled users, shared rewards, and aggregated experiences (Chen et al., 9 Jul 2025).