---
title: 'PAC-MCoFL: Pareto Actor-Critic in Federated Learning'
url: https://www.emergentmind.com/topics/pac-mcofl
type: topic
---

# PAC-MCoFL: Pareto Actor-Critic in Federated Learning

PAC-MCoFL is a game-theoretic multi-agent reinforcement learning framework for communication-and-computation co-optimization in non-cooperative federated learning services. In the formulation introduced in "Pareto Actor-Critic for Communication and Computation Co-Optimization in Non-Cooperative Federated Learning Services" [2508.16037], service providers act as agents in a multi-service-provider federated learning ecosystem and jointly optimize client assignment, adaptive quantization, and resource allocation under privacy constraints, competing interests, and limited information sharing. The framework integrates Pareto Actor-Critic principles with expectile regression, uses ternary Cartesian decomposition to manage a high-dimensional mixed action space, and includes a scalable variant, PAC-MCoFL-p, with a parameterized conjecture generator and a provably bounded approximation error [2508.16037].

## 1. Definition and problem domain

PAC-MCoFL is defined for a federated learning environment with a set of service providers
\[
\mathcal{R}=\{1,\dots,r,\dots,R\},
\]
and a set of clients
\[
\mathcal{N}=\{1,\dots,i,\dots,N\}.
\]
Each service provider owns a different federated learning service or model, and clients may participate in training for different providers using their local private datasets [2508.16037].

The motivating setting is explicitly non-cooperative. Centralized joint optimization is treated as unrealistic because of privacy constraints, competing interests, partial observability, and shared communication and computation resources. PAC-MCoFL therefore models each service provider as a strategic agent that optimizes its own federated learning operation while accounting for simultaneous competition from other providers [2508.16037].

The control variables optimized over time are
\[
\{n_{r,t},\, q_{r,t},\, B_{r,t},\, f_{r,t}\},
\]
corresponding to client assignment or selection, adaptive quantization, communication resource allocation, and computation resource allocation [2508.16037]. The stated objective is to balance model accuracy or federated learning service quality against communication overhead, latency, and energy consumption [2508.16037].

A central feature of the framework is that it does not target purely selfish equilibrium behavior. The paper argues that standard multi-agent reinforcement learning may converge to a risk-averse or otherwise suboptimal equilibrium, and instead seeks a Pareto-optimal equilibrium in which no service provider can improve without harming another [2508.16037].

## 2. System model and optimization objective

For service \(r\), client \(i\) has a local dataset \(\mathcal{D}_{i,r}\), and the local empirical objective is
\[
L_{i,r}\left( \boldsymbol{\omega} \right) =\frac{1}{|\mathcal{D}_{i,r}|}\sum_{\xi\in \mathcal{D}_{i,r}} l\left( \boldsymbol{\omega};\xi \right). \tag{1}
\]
The global federated objective for service \(r\) is
\[
\boldsymbol{\omega}^\ast =\arg\min_{\boldsymbol{\omega}} L_r\left( \boldsymbol{\omega} \right) =\arg\min_{\boldsymbol{\omega}} \sum_{i=1}^N \kappa_{i,r} L_{i,r}\left( \boldsymbol{\omega}\right), \tag{2}
\]
with aggregation weight
\[
\kappa_{i,r}=\frac{|\mathcal{D}_{i,r}|}{\sum_{i=1}^N |\mathcal{D}_{i,r}|}. 
\]
Local training uses
\[
\boldsymbol{\omega}_{i,r,t}[k] =\boldsymbol{\omega}_{i,r,t}[k-1] -\eta_k \tilde{\nabla} L_{i,r}\left( \boldsymbol{\omega}_{i,r,t}[k-1] \right), \tag{3}
\]
followed by server aggregation
\[
\boldsymbol{\omega}_{r,t+1} =\sum_{i=1}^N \kappa_{i,r} \left( \boldsymbol{\omega}_{i,r,t}[\iota] \right). \tag{4}
\]

Communication compression is modeled through stochastic quantization. For element \(\omega_d\), the quantizer is written as
\[
Q_q\left( \omega_d \right) =\left\| \boldsymbol{\omega} \right\|_p\cdot \mathrm{sgn}(\omega_d) \cdot \Xi_q \left(\omega_d,q \right),
\]
and the communication overhead for client \(i\), service \(r\), round \(t\), is
\[
{\rm vol}_{i,r,t} =|\boldsymbol{\omega}|\left(\lceil \log_2 q_{i,r,t} \rceil+1\right)+32. \tag{5}
\]

Computation energy and latency are modeled as
\[
E_{i,r,t}^{\rm cmp} =\mu_i c_{i,r} |\mathcal{D}_{i,r}| f_{i,r,t}^{2}, \tag{6}
\]
\[
T_{i,r,t}^{\rm cmp} =\frac{c_{i,r} |\mathcal{D}_{i,r}|}{f_{i,r,t}}. \tag{7}
\]
Using FDMA, the transmission rate is
\[
{\rm rate}_{i,r,t} =B_{i,r,t}\log_2\left( 1+\frac{g_{i,t} p_{i,t}}{B_{i,r,t} N_0} \right), \tag{8}
\]
with communication latency and energy
\[
T_{i,r,t}^{\rm com} =\frac{{\rm vol}_{i,r,t}}{{\rm rate}_{i,r,t}}, \tag{9}
\]
\[
E_{i,r,t}^{\rm com} =\frac{{\rm vol}_{i,r,t} p_{i,t}}{{\rm rate}_{i,r,t}}. \tag{10}
\]
Total service-level energy and latency are then
\[
E_{r,t}^{\rm total} =\sum_{i\in \mathcal{N}}\left( E_{i,r,t}^{\rm com}+E_{i,r,t}^{\rm cmp} \right), \tag{11a}
\]
\[
T_{r,t}^{\rm total} =\max_{i\in \mathcal{N}} \left( T_{i,r,t}^{\rm cmp}+T_{i,r,t}^{\rm com} \right). \tag{11b}
\]

The optimization problem is stated as
\[
\min _{\bm f, \bm n, \bm q, \bm B} \sum_{t=0}^{T-1} \gamma^t \Upsilon \left( L_r(\boldsymbol{\omega}_{r,t}), {\rm vol}_{r,t}, E_{r,t}^{\rm total}, T_{r,t}^{\rm total}\right) \tag{12}
\]
subject to energy, latency, client-selection, CPU, bandwidth, and quantization constraints [2508.16037]. The function \(\Upsilon(\cdot)\) is described as a monotonically decreasing cost function, and the formulation is subsequently turned into a reward-maximization problem for reinforcement learning [2508.16037].

## 3. Stochastic game formulation

The problem is modeled as an MDP and, more specifically, as a multi-agent stochastic game
\[
(\mathcal{O},\mathcal{A},{\rm rwd},\Lambda,\gamma),
\]
where each service provider is a player [2508.16037].

For agent \(r\), the observation at round \(t\) is
\[
o_{r,t} =\left\{ Z_{r,t}(\boldsymbol{\omega}),\Theta_{r,t},\boldsymbol{B}_t \right\}, \tag{13}
\]
with
\[
Z_{r,t}(\boldsymbol{\omega}) = \left\{ t,L_r(\boldsymbol{\omega}_{r,t}),\Gamma_{r,t},q_{r,t} \right\},
\]
\[
\Theta_{r,t} = \left\{ T_{r,t}^{\rm total}, E_{r,t}^{\rm total},{\rm vol}_{r,t} \right\},
\]
\[
\boldsymbol{B}_t=\{B_{r,t}\}_{r\in\mathcal{R}}.
\]
This means the local observation contains federated learning status, operational status, and public bandwidth allocations [2508.16037].

The action is
\[
a = \left\{ n_{r,t},f_{r,t},B_{r,t},q_{r,t} \right\} \in \mathcal{A}, \tag{14}
\]
and the joint action is
\[
\boldsymbol{a}_t=[a_{1,t},\dots,a_{R,t}].
\]
The paper further models client-side realized computation and quantization stochastically around service-level decisions:
\[
f_{i,r,t}\sim G(f_{r,t},\Sigma_f),\qquad q_{i,r,t}\sim G(q_{r,t},\Sigma_s). 
\]

The per-agent reward is defined as
\[
{\rm rwd}_{r,t} = \sigma_1 \Gamma_{r,t} +\sigma_2 \Phi_{r,t}(\boldsymbol{q}) -\sigma_3 E_{r,t}^{\rm total} -\sigma_4 T_{r,t}^{\rm total}. \tag{15}
\]
Here \(\Gamma_{r,t}\) is test accuracy, and the adversarial quantization term is
\[
\Phi_{r,t}(\boldsymbol{q}) = \frac{n_{r,t} q_{r,t}}{\epsilon \times {\rm vol}_{r,t} +\sum_{j\in \mathcal{R}\setminus\{r\}} n_{j,t} q_{j,t}}. \tag{16}
\]
The term explicitly couples each service provider’s utility to the decisions of competing providers [2508.16037].

The return under joint policy \(\boldsymbol{\pi}=(\pi_r,\pi_{-r})\) is
\[
J_r\left( \boldsymbol{\pi} \right) = \mathbb{E}_{o_{r,0}} \left[ \sum_{t=0}^{T-1}\gamma^t \mathbb{E}_{\boldsymbol{\pi}} \left[ {\rm rwd}_{r,t}\mid o_{r,0},\boldsymbol{\pi}\right] \right], \tag{17}
\]
and the associated \(Q\)-function is given by
\[
Q(o_{r,t},\boldsymbol{a}_t) = {\rm rwd}_{r,t}(o_{r,t},\boldsymbol{a}_t) +\gamma \mathbb{E}[V(o_{r,t+1})]. 
\]

## 4. Core method: Pareto Actor-Critic, expectile regression, and TCAD

PAC-MCoFL combines four components: multi-agent reinforcement learning for decentralized decision-making, Pareto Actor-Critic for conjecturing others’ policies, an expectile-regression critic for heterogeneous risk modeling, and ternary Cartesian decomposition for tractable control over the mixed action space [2508.16037].

A concise component view is as follows.

| Component | Role | Formal object |
|---|---|---|
| Pareto Actor-Critic | Conjectures favorable joint policies of other SPs | \(\pi_{-r}^{\dagger}\) |
| Expectile critic | Models asymmetric risk preferences | \(M_r^{(\tau)}\) |
| TCAD | Decomposes action control into ternary directional updates | \(\psi_m\in\{-1,0,1\}\) |
| PAC-MCoFL-p | Replaces exhaustive conjecture with a generator | \({\rm Gen}_{\varphi_r}\) |

The Pareto Actor-Critic mechanism is expressed through the conjectured opponent policy
\[
\pi_{-r}^{\dagger} = \arg\max_{\pi_{-r}} J_r(\pi_r,\pi_{-r}), \tag{19}
\]
and the induced actor objective
\[
\pi_r = \arg\max_{\pi_r} J_r(\pi_r,\pi_{-r}^{\dagger}). \tag{20}
\]
Operationally, the conjecture is approximated through the \(Q\)-function:
\[
\pi_{-r}^{\dagger} = \arg\max_{\pi_{-r}} Q\left( o_{r,t},a,\pi_{-r} \right). \tag{21}
\]
This is the framework’s defining PAC element: each service provider optimizes against a conjectured Pareto-favorable response of the others [2508.16037].

Without expectile regression, the critic loss is written as
\[
M_r(\pi_{-r}^{\dagger}) \triangleq \mathbb{E}\Big[ \big( {\rm rwd}_{r,t} +\gamma \max_{\pi_{-r}} Q(o_{r,t+1}, a_{r,t+1}, \pi_{-r}) - Q(o_{r,t}, a, \pi_{-r}) \big)^2 \Big]. \tag{22}
\]
PAC-MCoFL replaces this with the expectile objective
\[
M_r^{(\tau)}(\pi_{-r}^{\dagger}) \triangleq \mathbb{E} \left[ \tau \cdot \delta^2_{+} + (1-\tau)\cdot \delta^2_{-} \right], \tag{23}
\]
where \(\tau\in(0,1)\) is the expectile coefficient. The critic update becomes
\[
\theta_r = \theta_r - \alpha \nabla_{\theta_r} M_r^{(\tau)}(\pi_{-r}^{\dagger}). \tag{24}
\]
The paper interprets \(\tau<0.5\) as more sensitive to underestimation and promoting aggressive behavior, \(\tau>0.5\) as promoting conservative behavior, and \(\tau=0.5\) as the symmetric case [2508.16037].

The actor update is
\[
\phi_r = \phi_r + \beta \nabla_{\phi_r} J_r(\pi_r,\pi_{-r}^{\dagger}), \tag{25}
\]
with the policy-gradient expression
\[
\begin{aligned}
\nabla_{\phi_r} J_r\left(\pi_r,\pi_{-r}^{\dagger}\right)
&= \sum_{o_{r,t}} d^{\pi}(o_{r,t}) \sum_{a,\pi_{-r}^{\dagger}} \pi_r(a\mid o_{r,t})\pi_{-r}^{\dagger}(\cdot \mid o_{r,t},a) \\
&\qquad \cdot \nabla_{\phi_r}\log \pi_r(a\mid o_{r,t}) Q(o_{r,t},a,\pi_{-r}^{\dagger}).
\end{aligned} \tag{26}
\]

TCAD is introduced because the action space
\[
a=\{n_{r,t},f_{r,t},B_{r,t},q_{r,t}\}
\]
is high-dimensional and mixed [2508.16037]. The ternary projection operator is
\[
\mathcal{T}_{\rm TCAD}:a_{r,t} \rightarrow \{\varsigma_m \cdot \psi_m\}_{m=1}^4,\quad \psi_m\in\{-1,0,1\}, \tag{28}
\]
so the actor outputs directional increments rather than full joint actions. The actual update is
\[
a_{r,t+1}^{\prime (m)} = {\rm Proj}_{\mathcal{C}_m} \left( a_{r,t}^{\prime (m)} + \psi_m \cdot \varsigma_m \right), \tag{29}
\]
with feasible sets
\[
\mathcal{C}_m= \begin{cases}
\{1, \ldots, N\}, & m=n \\
\left[f^{\min }, f^{\max }\right], & m=f \\
\left[B^{\min }, B^{\max }\right], & m=B \\
\mathbb{Z}\cap\left[q^{\min }, q^{\max }\right], & m=q.
\end{cases} \tag{30}
\]
The paper states that this reduces the effective action-space burden from a product over discretized action dimensions to constant-size ternary directional control per dimension [2508.16037].

## 5. Scalable variant and theoretical properties

PAC-MCoFL-p is introduced because exhaustive conjecture over the other agents’ joint action space has cost
\[
O(|\mathcal{A}|^{R-1}),
\]
which becomes impractical as the number of service providers increases [2508.16037].

The scalable variant replaces exhaustive enumeration with a parameterized conjecture generator:
\[
\pi_{-r}^{\dagger} \triangleq {\rm softmax}\left({\rm Gen}_{\varphi_r}(a_{r,t},o_{r,t},h)\right), \tag{31}
\]
where \(h\) is a hidden state derived from the observation. The generator is trained with
\[
\Phi(\varphi_{r}) = \mathbb{E}_{\tilde{a}\sim \pi_{-r}^{\dagger}} \left[ -Q_r^{\pi}(o_{r,t},a_{r,t},\tilde{a})\right] + \chi \mathbb{E}\left[ D_{\rm KL}\left( \pi_{-r}^{\dagger} \,\|\, \pi_{-r}^{\dagger,\rm tar} \right) \right]. \tag{32}
\]

The paper proves that if the true optimal joint action distribution is approximated by the generator, then
\[
\left| \mathbb{E}_{\hat{\pi}_{-r}}\left[ Q \right] - \max_{a}Q \right| \leq C \cdot \sqrt{D_{\rm KL}\left( \hat{\pi}_{-r} \,\|\, \pi_{-r}^{\dagger} \right)}, \tag{33}
\]
where \(C\) is the Lipschitz constant [2508.16037]. This is the formal bounded-error guarantee for PAC-MCoFL-p.

The critic iteration is
\[
Q_{r,t+1}\left(o_{r,t},\boldsymbol{a}_t\right)= Q_{r,t}\left(o_{r,t},\boldsymbol{a}_t\right)
+\alpha\Big( {\rm rwd}_{r,t} +\gamma \mathbb{E}_{o_{r,t+1}, a_{r,t+1}} \left[ \max_{\pi_{-r}} Q_{r,t}\left(o_{r,t+1}, a_{r,t+1}, \pi_{-r}\right) \right]
-Q_{r,t}\left(o_{r,t},\boldsymbol{a}_t\right) \Big). \tag{34}
\]
The associated Pareto operator is
\[
\mathscr{H}^p Q_{r,t} \left(o_{r,t},\boldsymbol{a}_t\right) =
\mathbb{E}_{o_{r,t+1} \sim \Lambda\left(\cdot \mid o_{r,t},\boldsymbol{a}_t\right)}
\left[ {\rm rwd}_{r,t} +\gamma\max_{\pi_{-r}} Q_{r,t}\left(o_{r,t+1}, a, \pi_{-r}\right) \right]. \tag{35}
\]
The paper proves the contraction property
\[
\| \mathscr{H}^P Q_1-\mathscr{H}^P Q_2 \|_\infty \leq \gamma \| Q_1-Q_2\|_\infty, \tag{36}
\]
and, under finite observation and action spaces, infinite visitation, bounded reward, a structural assumption on Nash equilibria, and standard stochastic approximation learning-rate conditions, proves convergence to a Nash \(Q\)-value
\[
\boldsymbol{Q}^{\ast}=\left[ Q_{1}^{\ast},Q_{2}^{\ast},\cdots ,Q_{R}^{\ast} \right]. 
\]
A corollary states that the expectile-modified operator preserves the almost sure convergence property [2508.16037].

This suggests that PAC-MCoFL should be understood not merely as a heuristic equilibrium-selection device, but as a Pareto-oriented actor-critic construction with explicit contraction and convergence claims under the paper’s assumptions.

## 6. Empirical evaluation, scope, and relation to adjacent PAC literature

The experimental setup in [2508.16037] uses \(N=5\) clients, \(R=3\) service providers, \(T=35\) global rounds, \(\iota=3\) local SGD steps, and default \(\tau=0.5\). The three federated learning tasks are CIFAR-10, Fashion-MNIST, and MNIST, with model sizes \(9{,}074{,}474\), \(21{,}840\), and \(101{,}770\) parameters, respectively [2508.16037]. Data heterogeneity is varied through \(\rho\in\{0.9,0.7,0.5\}\), with \(\rho=1\) as the IID-like default [2508.16037]. Baselines include FedAvg, FedProx-u, FedDQ-h, AdaQuantFL-h, MAPPO, and RSM-MASAC [2508.16037].

The framework is evaluated with per-task and total communication overhead, latency, energy, reward, test accuracy, and hypervolume indicator. For an algorithm, the reward-vector set is
\[
\mathcal{P}_{(\rm alg)} = \left\{ \mathbf{v}^{(m)} = \bigl({\rm rwd}^{(m)}_{1},\,{\rm rwd}^{(m)}_{2},\,\ldots,\,{\rm rwd}^{(m)}_{R}\bigr) \right\}_{m=1}^{\rm total\ runs}, \tag{37}
\]
and the hypervolume indicator is
\[
{\rm HVI} = \varpi \left( \bigcup_{\mathbf{v}\in \mathcal{P}_{\rm alg}} [{\rm rwd}_1,{\rm rwd}_1^{\rm ref}] \times\cdots\times [{\rm rwd}_R,{\rm rwd}_R^{\rm ref}] \right). \tag{38}
\]

From Table III, PAC-MCoFL attains total reward
\[
245.27 \pm 24.48
\]
and HVI
\[
0.7258,
\]
while PAC-MCoFL-p attains
\[
237.82 \pm 27.4
\]
and
\[
0.6576
\]
respectively [2508.16037]. The paper summarizes PAC-MCoFL’s gains as approximately \(5.8\%\) in total reward and \(4.2\%\) in hypervolume indicator over the latest MARL solutions [2508.16037]. It also reports that TCAD improves total reward by about \(7.6\%\) and HVI by about \(14.2\%\), and that extreme expectiles such as \(0.1\) and \(0.99\) can degrade performance by up to \(10.4\%\) [2508.16037]. Under more severe non-IID conditions \(\rho=0.5\), PAC-MCoFL still outperforms MAPPO and RSM-MASAC, with reward improvements reported in the ranges \(1.8\%\)–\(3.1\%\) and \(2.1\%\)–\(5.6\%\), depending on task [2508.16037].

The paper also reports a scalability distinction: standard PAC-MCoFL becomes OOM for \(R\ge 4\), whereas PAC-MCoFL-p remains feasible and outperforms scalable MAPPO by about \(21.3\%\) average reward in broader multi-agent settings [2508.16037].

A recurring source of confusion is nomenclature. PAC-MCoFL is unrelated to polarization-adjusted convolutional codes, which are coding-theoretic PAC codes [2106.08118], [2005.13711], [2002.06805], [2012.05511]. It is also distinct from FedPAC, which uses PAC learning to quantify privacy, utility, and efficiency in federated learning via sample complexity [2304.04641], and from PAC-PFL, which is a PAC-Bayesian framework for personalized federated learning of probabilistic models [2401.08351]. A looser theoretical precursor is the centralized cooperative multi-agent reinforcement learning setting with noisy and resource-limited communication analyzed in [1905.09951], but that setting uses a central learner and tabular \(Q\)-tables rather than a non-cooperative multi-service-provider federated learning game.

Within its own stated scope, PAC-MCoFL is therefore best characterized as a Pareto-oriented multi-agent actor-critic framework for decentralized resource co-optimization in non-cooperative federated learning services, distinguished by three design choices: optimistic Pareto conjecture over other agents’ policies, expectile-based asymmetric value learning, and TCAD-based control over client assignment, quantization, bandwidth, and computation [2508.16037].

Source: https://www.emergentmind.com/topics/pac-mcofl