---
title: Mean Actor-Critic (MAC) Methods
url: https://www.emergentmind.com/topics/mean-actor-critic-mac
type: topic
---

# Mean Actor-Critic (MAC) Methods

Mean Actor-Critic (MAC) is a class of reinforcement learning (RL) algorithms that modify the classic actor-critic framework by analytically averaging policy gradients over the entire action set, rather than estimating gradients using only the actions actually executed. This approach yields lower-variance policy gradient estimates, improved sample efficiency, and, in the context of mean-field control, extends to learning over distributions of states and actions. The term “Mean Actor-Critic” has been used in both standard RL for discrete-action spaces [1709.00503] and in mean-field control/game theory (continuous/distributional MAC) [2309.04317, 2303.06993, 1910.07498].

## 1. The Mean Actor-Critic Principle in Reinforcement Learning

Classical on-policy policy-gradient algorithms aim to maximize the expected return:
\[
J(\theta) = \mathbb{E}_{s\sim d^\pi,\,a\sim\pi_\theta}[Q^\pi(s,a)]
\]
The policy-gradient theorem yields:
\[
\nabla_\theta J(\theta) = \mathbb{E}_{s\sim d^\pi,\,a\sim\pi_\theta}[\nabla_\theta\log\pi_\theta(a|s) Q^\pi(s,a)]
\]
In standard actor-critic (AC) frameworks, this expectation is estimated by sampling both states $s$ and executed actions $a$. The MAC algorithm explores the equivalence:
\[
\mathbb{E}_{a\sim\pi_\theta}\big[\nabla_\theta\log\pi_\theta(a|s) Q^\pi(s,a)\big] = \sum_{a\in\mathcal{A}} \pi_\theta(a|s) \nabla_\theta\log\pi_\theta(a|s) Q^\pi(s,a)
\]
By pushing the sum inside the expectation over states, MAC computes the policy gradient for each sampled state by analytically averaging over all actions—using the explicit form of $\pi_\theta(a|s)$ and $Q^\pi(s,a)$—rather than relying on the particular action sampled. This results in a policy gradient estimator of the form:
\[
\nabla_\theta J_{\rm MAC}(\theta) = \mathbb{E}_{s\sim d^\pi}\left[\sum_{a\in\mathcal{A}} \pi_\theta(a|s) \nabla_\theta\log\pi_\theta(a|s) Q^\pi(s,a)\right]
\]
This estimator applies to discrete $\mathcal{A}$ and can be implemented efficiently if the action set is modest in size [1709.00503].

## 2. Theoretical Properties: Variance Reduction

The MAC gradient estimator is unbiased: it corresponds exactly to the true policy gradient as per the policy-gradient theorem. However, by replacing stochastic sampling over actions with an exact sum (or expectation), MAC eliminates the variance associated with action sampling. Specifically, for parameter component $i$, the MAC estimator
\[
\mathrm{MAC}_i = \frac{1}{T} \sum_{t=1}^T \sum_{a} \pi_\theta(a|s_t) \nabla_{\theta_i}\log\pi_\theta(a|s_t) \widehat Q(s_t, a; \omega)
\]
has strictly lower variance than its standard actor-critic counterpart, as long as $\pi_\theta(\cdot|s)$ is not deterministic, assuming independence of $\widehat Q(s,a)$ errors across $(s, a)$. This follows directly from Jensen's inequality applied to the variance across the discrete action space [1709.00503]. No additional baseline function is required—subtracting an action-independent baseline cancels out exactly in the MAC formulation.

## 3. Implementation and Algorithmic Steps

A high-level algorithmic outline for the discrete-action MAC in RL is as follows [1709.00503]:

1. **Initialize** policy parameters $\theta$ and critic parameters $\omega$.
2. **Rollout** the current policy to collect a batch of $T$ state transitions $\{(s_t, a_t, r_t, s_{t+1})\}$.
3. **Critic Update**: Fit $\widehat Q(s, a; \omega)$ (e.g., by TD(0), TD($\lambda$), or Monte Carlo) to approximate $Q^\pi(s,a)$.
4. **Policy Update**: For each $s_t$ in the batch, compute the policy gradient
   \[
   g_t = \sum_{a \in \mathcal{A}} \pi_\theta(a|s_t) \nabla_\theta\log\pi_\theta(a|s_t)\,\widehat Q(s_t,a;\omega)
   \]
   Perform a gradient ascent step: $\theta \leftarrow \theta + \alpha \cdot (1/T)\sum_{t=1}^T g_t$.
5. Repeat until convergence, optionally decaying the step size.

The MAC framework naturally applies to cases where actions are discrete and $\mathcal{A}$ is not prohibitively large. Over-sampling critic updates per policy update can further enhance stability.

## 4. Applications in Mean-Field Control and Mean-Field Games

The MAC nomenclature has also been widely adopted in continuous-time reinforcement learning for mean-field control (MFC) and mean-field games (MFG). These settings generalize the single-agent RL context by introducing population-level effects (state distribution $\mu_t$ evolving with the agents).

In continuous-time mean-field MAC algorithms [2309.04317, 2303.06993], the state process evolves as a McKean–Vlasov SDE:
\[
dX_t = b(t, X_t, \mu_t, \alpha_t)\,dt + \sigma(t, X_t, \mu_t, \alpha_t)\,dW_t
\]
The learning task is to minimize $\mathbb{E}[\int_0^T f(X_t, \mu_t, \alpha_t)\,dt + g(X_T, \mu_T)]$ over randomized policies $\pi_\theta$.

Key features distinctive to mean-field MAC approaches:
- **Parameterization**: The policy (“actor”) is typically a Gaussian whose mean is represented by a moment neural network (moment-NN) depending on $x$ and low-order empirical moments of $\mu_t$. The critic is also a moment-NN approximating the value function over the Wasserstein space of distributions.
- **Policy gradient**: Incorporates both standard policy-gradient terms and additional terms arising from the mean-field (distributional) dependence, involving functional derivatives and explicit mean-field operators.
- **Critic loss**: Enforces a martingale (HJB) condition using temporal-difference losses adapted to Wasserstein space inputs.
- **Monte Carlo over distributions**: Training samples entire trajectories of particle systems to approximate distributional expectations.

The essential computation remains: MAC marginalizes action randomness in the policy gradient, using either exact summation or analytic expectation, which is particularly tractable in the Gaussian parameterization context for continuous action spaces.

## 5. Empirical Findings and Benchmarks

**Classic RL (Discrete-action, Non-Mean-Field):**

- MAC achieved faster learning than REINFORCE, Advantage REINFORCE, actor-critic, and advantage actor-critic baselines on CartPole; matched top baselines on LunarLander.
- In six Atari 2600 games (Beamrider, Breakout, Pong, Q*bert, Seaquest, SpaceInvaders), MAC was competitive with state-of-the-art policy search methods (TRPO, Evolution Strategies, A2C), frequently outperforming or matching them after 50 million frames.
- Lower empirical variance translated to smoother learning curves and higher sample efficiency, with particularly stable updates in high-variance games [1709.00503].

**Mean-Field Control and Games:**

- Moment-NN MAC delivers sub-percent errors for linear-quadratic (LQ) models and robust tracking for nonlinear settings, even in two and three-dimensional MFC problems [2309.04317].
- In large particle regimes ($M \sim 10^4$ or higher) and with moderate moment order (L=2 or 3), MAC’s learned policies and value functions closely match analytic solutions.
- Actor-critic MAC in discrete-time linear-quadratic MFGs achieves provable global linear convergence to Nash equilibria, provided standard LQ-MFG structural assumptions are satisfied [1910.07498].

A summary table of select empirical domains is given below:

| Domain                    | MAC Variant          | Key Outcome                                               |
|---------------------------|---------------------|-----------------------------------------------------------|
| CartPole, LunarLander     | Discrete-action MAC | Faster or equal learning; lower gradient variance         |
| Atari 2600 (6 games)      | Discrete-action MAC | Competitive/stable vs TRPO, ES, A2C (after 50M frames)    |
| Systemic-risk LQ          | Moment-NN MAC       | Sub-percent value errors, matched control trajectories    |
| 1D/2D/3D LQ optimal trade | Moment-NN MAC       | Errors 1–2%; robust for nonlinear, controlled-volatility  |
| LQ mean-field games       | Linear MAC (MFG)    | Provable linear convergence to Nash equilibria            |

## 6. Connections to Broader RL and Mean-Field Literature

The MAC approach formalizes a long-standing method for reducing variance in policy gradient estimators by replacing stochastic action sampling with analytical averaging, made feasible by tractable action spaces or efficient parameterizations. In RL, MAC is especially effective for discrete, moderate-size action sets. In mean-field settings, the combination of MAC with moment-NNs enables function approximation on spaces of distributions, a central technical problem in mean-field control and games.

The MAC formulation is distinct from baseline subtraction, advantage estimation, or entropy regularization. It is orthogonal to improvements in critic learning, function approximation architecture, or policy regularization. In mean-field game theory, MAC supports model-free learning of Nash equilibria under linear-quadratic structure with global convergence guarantees and extends naturally to high-dimensional, nonlinear, and volatility-controlled settings.

## 7. Limitations and Computational Considerations

MAC’s principal advantage is variance reduction without extra bias. However, the inner expectation over actions or analytic computation of integrals/sums can be prohibitive for large discrete or high-dimensional continuous action spaces. In MFC/MFG, efficiency is preserved by parameterizing policies and critics with moment-NNs and fitting over low-order empirical moments, but the computational cost scales with the number of particles and the order of moments used. Training in high dimensions or with high-order moments may require considerable compute time (reported as $\sim 4 \times 10^4$ to $1.5 \times 10^5$ seconds on V100 GPUs for certain mean-field problems) [2309.04317]. A plausible implication is that scalability to large-scale real-world MFC depends on further advances in neural architecture and sampling strategy.

## References

- [1709.00503] "Mean Actor Critic" — original discrete-action MAC in RL.
- [2309.04317] "Actor critic learning algorithms for mean-field control with moment neural networks" — continuous-time mean-field MAC with moment-NN parameterization.
- [2303.06993] "Actor-Critic learning for mean-field control in continuous time" — continuous-time mean-field MAC with Wasserstein space parametrization.
- [1910.07498] "Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games" — discrete-time MFG with provable convergence.

Source: https://www.emergentmind.com/topics/mean-actor-critic-mac