PAC-MCoFL: Pareto Actor-Critic in Federated Learning

• PAC-MCoFL is a game-theoretic multi-agent RL framework that achieves Pareto-optimal equilibria in federated learning by optimizing client assignment, adaptive quantization, and resource allocation.
• It leverages Pareto Actor-Critic, expectile regression, and ternary Cartesian decomposition to manage high-dimensional mixed action spaces under privacy constraints.
• The approach outperforms conventional methods in reward, latency, energy efficiency, and scalability, enabling robust decision-making in non-cooperative service provider environments.

PAC-MCoFL is a game-theoretic multi-agent reinforcement learning framework for communication-and-computation co-optimization in non-cooperative federated learning services. In the formulation introduced in "Pareto Actor-Critic for Communication and Computation Co-Optimization in Non-Cooperative Federated Learning Services" (Tan et al., 22 Aug 2025), service providers act as agents in a multi-service-provider federated learning ecosystem and jointly optimize client assignment, adaptive quantization, and resource allocation under privacy constraints, competing interests, and limited information sharing. The framework integrates Pareto Actor-Critic principles with expectile regression, uses ternary Cartesian decomposition to manage a high-dimensional mixed action space, and includes a scalable variant, PAC-MCoFL-p, with a parameterized conjecture generator and a provably bounded approximation error (Tan et al., 22 Aug 2025).

1. Definition and problem domain

PAC-MCoFL is defined for a federated learning environment with a set of service providers

$\mathcal{R}=\{1,\dots,r,\dots,R\},$

and a set of clients

$\mathcal{N}=\{1,\dots,i,\dots,N\}.$

Each service provider owns a different federated learning service or model, and clients may participate in training for different providers using their local private datasets (Tan et al., 22 Aug 2025).

The motivating setting is explicitly non-cooperative. Centralized joint optimization is treated as unrealistic because of privacy constraints, competing interests, partial observability, and shared communication and computation resources. PAC-MCoFL therefore models each service provider as a strategic agent that optimizes its own federated learning operation while accounting for simultaneous competition from other providers (Tan et al., 22 Aug 2025).

The control variables optimized over time are

$\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},$

corresponding to client assignment or selection, adaptive quantization, communication resource allocation, and computation resource allocation (Tan et al., 22 Aug 2025). The stated objective is to balance model accuracy or federated learning service quality against communication overhead, latency, and energy consumption (Tan et al., 22 Aug 2025).

A central feature of the framework is that it does not target purely selfish equilibrium behavior. The paper argues that standard multi-agent reinforcement learning may converge to a risk-averse or otherwise suboptimal equilibrium, and instead seeks a Pareto-optimal equilibrium in which no service provider can improve without harming another (Tan et al., 22 Aug 2025).

2. System model and optimization objective

For service $r$, client $i$ has a local dataset $\mathcal{D}_{i,r}$, and the local empirical objective is

$$L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}$$

The global federated objective for service $r$ is

$$\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}$$

with aggregation weight

$$\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}.$$

Local training uses

$\mathcal{N}=\{1,\dots,i,\dots,N\}.$0

followed by server aggregation

$\mathcal{N}=\{1,\dots,i,\dots,N\}.$1

Communication compression is modeled through stochastic quantization. For element $\mathcal{N}=\{1,\dots,i,\dots,N\}.$2, the quantizer is written as

$\mathcal{N}=\{1,\dots,i,\dots,N\}.$3

and the communication overhead for client $\mathcal{N}=\{1,\dots,i,\dots,N\}.$4, service $\mathcal{N}=\{1,\dots,i,\dots,N\}.$5, round $\mathcal{N}=\{1,\dots,i,\dots,N\}.$6, is

$\mathcal{N}=\{1,\dots,i,\dots,N\}.$7

Computation energy and latency are modeled as

$\mathcal{N}=\{1,\dots,i,\dots,N\}.$8

$\mathcal{N}=\{1,\dots,i,\dots,N\}.$9

Using FDMA, the transmission rate is

$\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},$0

with communication latency and energy

$\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},$1

$\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},$2

Total service-level energy and latency are then

$\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},$3

$\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},$4

The optimization problem is stated as

$\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},$5

subject to energy, latency, client-selection, CPU, bandwidth, and quantization constraints (Tan et al., 22 Aug 2025). The function $\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},$6 is described as a monotonically decreasing cost function, and the formulation is subsequently turned into a reward-maximization problem for reinforcement learning (Tan et al., 22 Aug 2025).

3. Stochastic game formulation

The problem is modeled as an MDP and, more specifically, as a multi-agent stochastic game

$\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},$7

where each service provider is a player (Tan et al., 22 Aug 2025).

For agent $\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},$8, the observation at round $\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},$9 is

$r$0

with

$r$1

$r$2

$r$3

This means the local observation contains federated learning status, operational status, and public bandwidth allocations (Tan et al., 22 Aug 2025).

The action is

$r$4

and the joint action is

$r$5

The paper further models client-side realized computation and quantization stochastically around service-level decisions: $r$6

The per-agent reward is defined as

$r$7

Here $r$8 is test accuracy, and the adversarial quantization term is

$r$9

The term explicitly couples each service provider’s utility to the decisions of competing providers (Tan et al., 22 Aug 2025).

The return under joint policy $i$0 is

$i$1

and the associated $i$2-function is given by

$i$3

4. Core method: Pareto Actor-Critic, expectile regression, and TCAD

PAC-MCoFL combines four components: multi-agent reinforcement learning for decentralized decision-making, Pareto Actor-Critic for conjecturing others’ policies, an expectile-regression critic for heterogeneous risk modeling, and ternary Cartesian decomposition for tractable control over the mixed action space (Tan et al., 22 Aug 2025).

A concise component view is as follows.

Component	Role	Formal object
Pareto Actor-Critic	Conjectures favorable joint policies of other SPs	$i$4
Expectile critic	Models asymmetric risk preferences	$i$5
TCAD	Decomposes action control into ternary directional updates	$i$6
PAC-MCoFL-p	Replaces exhaustive conjecture with a generator	$i$7

The Pareto Actor-Critic mechanism is expressed through the conjectured opponent policy

$i$8

and the induced actor objective

$i$9

Operationally, the conjecture is approximated through the $\mathcal{D}_{i,r}$0-function: $\mathcal{D}_{i,r}$1 This is the framework’s defining PAC element: each service provider optimizes against a conjectured Pareto-favorable response of the others (Tan et al., 22 Aug 2025).

Without expectile regression, the critic loss is written as

$\mathcal{D}_{i,r}$2

PAC-MCoFL replaces this with the expectile objective

$\mathcal{D}_{i,r}$3

where $\mathcal{D}_{i,r}$4 is the expectile coefficient. The critic update becomes

$\mathcal{D}_{i,r}$5

The paper interprets $\mathcal{D}_{i,r}$6 as more sensitive to underestimation and promoting aggressive behavior, $\mathcal{D}_{i,r}$7 as promoting conservative behavior, and $\mathcal{D}_{i,r}$8 as the symmetric case (Tan et al., 22 Aug 2025).

The actor update is

$\mathcal{D}_{i,r}$9

with the policy-gradient expression

$$L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}$$0

TCAD is introduced because the action space

$$L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}$$1

is high-dimensional and mixed (Tan et al., 22 Aug 2025). The ternary projection operator is

$$L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}$$2

so the actor outputs directional increments rather than full joint actions. The actual update is

$$L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}$$3

with feasible sets

$$L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}$$4

The paper states that this reduces the effective action-space burden from a product over discretized action dimensions to constant-size ternary directional control per dimension (Tan et al., 22 Aug 2025).

5. Scalable variant and theoretical properties

PAC-MCoFL-p is introduced because exhaustive conjecture over the other agents’ joint action space has cost

$$L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}$$5

which becomes impractical as the number of service providers increases (Tan et al., 22 Aug 2025).

The scalable variant replaces exhaustive enumeration with a parameterized conjecture generator: $$L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}$$6 where $$L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}$$7 is a hidden state derived from the observation. The generator is trained with

$$L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}$$8

The paper proves that if the true optimal joint action distribution is approximated by the generator, then

$$L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}$$9

where $r$0 is the Lipschitz constant (Tan et al., 22 Aug 2025). This is the formal bounded-error guarantee for PAC-MCoFL-p.

The critic iteration is

$r$1

The associated Pareto operator is

$r$2

The paper proves the contraction property

$r$3

and, under finite observation and action spaces, infinite visitation, bounded reward, a structural assumption on Nash equilibria, and standard stochastic approximation learning-rate conditions, proves convergence to a Nash $r$4-value

$r$5

A corollary states that the expectile-modified operator preserves the almost sure convergence property (Tan et al., 22 Aug 2025).

This suggests that PAC-MCoFL should be understood not merely as a heuristic equilibrium-selection device, but as a Pareto-oriented actor-critic construction with explicit contraction and convergence claims under the paper’s assumptions.

6. Empirical evaluation, scope, and relation to adjacent PAC literature

The experimental setup in (Tan et al., 22 Aug 2025) uses $r$6 clients, $r$7 service providers, $r$8 global rounds, $r$9 local SGD steps, and default $$\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}$$0. The three federated learning tasks are CIFAR-10, Fashion-MNIST, and MNIST, with model sizes $$\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}$$1, $$\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}$$2, and $$\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}$$3 parameters, respectively (Tan et al., 22 Aug 2025). Data heterogeneity is varied through $$\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}$$4, with $$\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}$$5 as the IID-like default (Tan et al., 22 Aug 2025). Baselines include FedAvg, FedProx-u, FedDQ-h, AdaQuantFL-h, MAPPO, and RSM-MASAC (Tan et al., 22 Aug 2025).

The framework is evaluated with per-task and total communication overhead, latency, energy, reward, test accuracy, and hypervolume indicator. For an algorithm, the reward-vector set is

$$\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}$$6

and the hypervolume indicator is

$$\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}$$7

From Table III, PAC-MCoFL attains total reward

$$\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}$$8

and HVI

$$\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}$$9

while PAC-MCoFL-p attains

$$\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}.$$0

and

$$\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}.$$1

respectively (Tan et al., 22 Aug 2025). The paper summarizes PAC-MCoFL’s gains as approximately $$\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}.$$2 in total reward and $$\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}.$$3 in hypervolume indicator over the latest MARL solutions (Tan et al., 22 Aug 2025). It also reports that TCAD improves total reward by about $$\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}.$$4 and HVI by about $$\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}.$$5, and that extreme expectiles such as $$\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}.$$6 and $$\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}.$$7 can degrade performance by up to $$\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}.$$8 (Tan et al., 22 Aug 2025). Under more severe non-IID conditions $$\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}.$$9, PAC-MCoFL still outperforms MAPPO and RSM-MASAC, with reward improvements reported in the ranges $\mathcal{N}=\{1,\dots,i,\dots,N\}.$00–$\mathcal{N}=\{1,\dots,i,\dots,N\}.$01 and $\mathcal{N}=\{1,\dots,i,\dots,N\}.$02–$\mathcal{N}=\{1,\dots,i,\dots,N\}.$03, depending on task (Tan et al., 22 Aug 2025).

The paper also reports a scalability distinction: standard PAC-MCoFL becomes OOM for $\mathcal{N}=\{1,\dots,i,\dots,N\}.$04, whereas PAC-MCoFL-p remains feasible and outperforms scalable MAPPO by about $\mathcal{N}=\{1,\dots,i,\dots,N\}.$05 average reward in broader multi-agent settings (Tan et al., 22 Aug 2025).

A recurring source of confusion is nomenclature. PAC-MCoFL is unrelated to polarization-adjusted convolutional codes, which are coding-theoretic PAC codes (Moradi et al., 2021, Yao et al., 2020, Rowshan et al., 2020, Moradi, 2020). It is also distinct from FedPAC, which uses PAC learning to quantify privacy, utility, and efficiency in federated learning via sample complexity (Zhang et al., 2023), and from PAC-PFL, which is a PAC-Bayesian framework for personalized federated learning of probabilistic models (Boroujeni et al., 2024). A looser theoretical precursor is the centralized cooperative multi-agent reinforcement learning setting with noisy and resource-limited communication analyzed in (Raveh et al., 2019), but that setting uses a central learner and tabular $\mathcal{N}=\{1,\dots,i,\dots,N\}.$06-tables rather than a non-cooperative multi-service-provider federated learning game.

Within its own stated scope, PAC-MCoFL is therefore best characterized as a Pareto-oriented multi-agent actor-critic framework for decentralized resource co-optimization in non-cooperative federated learning services, distinguished by three design choices: optimistic Pareto conjecture over other agents’ policies, expectile-based asymmetric value learning, and TCAD-based control over client assignment, quantization, bandwidth, and computation (Tan et al., 22 Aug 2025).