Action Collapse Policy Gradient (ACPG)

• ACPG is an umbrella term for policy-gradient methods that modify learning via divergence regularization, geometric constraints, or quantum decoding.
• The diversity-promoting variant uses φ-divergence and MMD regularizers to prevent entropy collapse and improve action exploration in personalization tasks.
• The ETF-based approach fixes the action-head to a simplex equiangular tight frame, enforcing a neural collapse-like geometry for stable and efficient policy learning.

Searching arXiv for the cited ACPG-related papers and any closely related usages of the acronym. Action Collapse Policy Gradient (ACPG) is not a single uniformly defined algorithm in the arXiv literature, but a shared acronym used for multiple policy-gradient modifications that target different failure modes and architectural constraints. In one line of work, ACPG augments standard policy-gradient optimization with $\varphi$-divergence and Maximum Mean Discrepancy penalties to prevent entropy collapse in personalization tasks (Starnes et al., 2023). In a second line, ACPG fixes the policy head to a simplex equiangular tight frame (ETF) so that policy-network features provably exhibit an Action Collapse geometry analogous to Neural Collapse (Zhou et al., 2 Sep 2025). In a third, quantum ACPG denotes a variational quantum policy-gradient algorithm with optimized action decoding, in which a single computational-basis measurement and a classical decoding map define the policy (Meyer et al., 2022). The commonality across these uses is a modification of policy-gradient structure rather than a shared mathematical objective.

1. Terminological scope and conceptual distinctions

A concise way to organize the literature is to distinguish three ACPG meanings by the mechanism each paper changes: regularization of the policy objective, geometric constraints on the action head, or quantum action decoding. The acronym therefore names different interventions at different layers of the policy-gradient pipeline.

Usage of ACPG	Core mechanism	Reported setting
Diversity-promoting ACPG (Starnes et al., 2023)	$\varphi$-divergence and/or MMD regularization	MNIST, CIFAR-10, Spotify personalization
ETF-based ACPG (Zhou et al., 2 Sep 2025)	Freeze the final action head as a simplex ETF	OpenAI Gym with REINFORCE, PPO, TRPO, A3C
Quantum ACPG (Meyer et al., 2022)	Optimized classical decoding of quantum measurements	CartPole-v0, FrozenLake, ContextualBandits, 5-qubit hardware

A recurrent source of confusion is the phrase action collapse. In the personalization paper, action collapse is also called entropy collapse and is a pathology: a learned policy $\pi_\theta(a\mid s)$ assigns near-zero probability to many actions, reducing exploration and coverage (Starnes et al., 2023). In the ETF-based paper, Action Collapse is the analogue in policy-gradient networks of Neural Collapse and is presented as an optimal geometric structure of last-layer activations and action-head weights (Zhou et al., 2 Sep 2025). These are not the same phenomenon, even though the acronym ACPG is used in both settings.

2. Diversity-promoting ACPG for entropy collapse in personalization

In the personalization formulation, the starting point is the standard surrogate objective

$$L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].$$

The method augments this objective with regularizers constructed from various $\varphi$-divergences and Maximum Mean Discrepancy, encouraging the current policy to follow a different state visitation and/or action choice distribution than previously computed policies (Starnes et al., 2023).

The full ACPG objective is

$$L_{\mathrm{ACPG}}(\theta) = L_{\mathrm{pg}}(\theta) -\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))] -\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).$$

The reference policy is $\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$, or a reference policy such as uniform. The paper enumerates several admissible policy divergences, including

$$D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)), \quad D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),$$

the Jensen–Shannon divergence

$$D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m), \qquad m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},$$

the Hellinger distance

$$D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,$$

and total variation

$\varphi$0

The MMD term is defined on the state–action distribution embedded in an RKHS $\varphi$1 with kernel $\varphi$2. If $\varphi$3, then

$\varphi$4

The paper’s interpretation is operational: each regularizer adds gradient components that are nonzero for every action, and in the MMD case for every state–action pair, so probability mass is reallocated toward less-selected actions (Starnes et al., 2023).

The training routine retains the usual policy-gradient term

$\varphi$5

adds a divergence gradient

$\varphi$6

and an MMD gradient

$\varphi$7

then updates

$\varphi$8

with periodic reference-policy refresh $\varphi$9 every $\pi_\theta(a\mid s)$0 iterations. The paper states that it does not derive a formal convergence proof, but under standard assumptions on smoothness and bounded gradients, gradient ascent on $\pi_\theta(a\mid s)$1 converges to a stationary point (Starnes et al., 2023).

3. ETF-based ACPG and the geometry of Action Collapse

The 2025 paper defines Action Collapse as the policy-gradient analogue of Neural Collapse. When a policy DNN has reached optimal performance under certain ideal conditions, its last-layer state–action activations exhibit within-action collapse, global simplex structure, and alignment of the policy head (Zhou et al., 2 Sep 2025).

Within-action collapse is formulated by partitioning states according to their true optimal action. If

$\pi_\theta(a\mid s)$2

are activations for states in $\pi_\theta(a\mid s)$3, then

$\pi_\theta(a\mid s)$4

and the within-class covariance

$\pi_\theta(a\mid s)$5

converges to zero. The class means, after centering by the global mean $\pi_\theta(a\mid s)$6, form the vertices of a simplex equiangular tight frame:

$\pi_\theta(a\mid s)$7

The action-selection weights collapse to the same ETF:

$\pi_\theta(a\mid s)$8

The paper’s ACPG method turns this geometry into an architectural constraint. A simplex ETF is defined by vectors $\pi_\theta(a\mid s)$9 with $$L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].$$0 such that there exists an orthonormal $$L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].$$1 satisfying

$$L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].$$2

which implies

$$L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].$$3

ACPG fixes the action-head weight matrix

$$L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].$$4

to be exactly such an ETF, optionally scaled by $$L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].$$5 so that $$L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].$$6, and sets the target class means to

$$L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].$$7

The analytical core is Theorem 1. In a Layer-Peeled Model with feature map $$L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].$$8 and fixed final softmax weights $$L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].$$9, the objective is

$\varphi$0

subject to $\varphi$1 for every $\varphi$2. The theorem states informally that, regardless of whether $\varphi$3 is fully explored, balanced, or uniform, any global maximizer $\varphi$4 satisfies

$\varphi$5

The proof sketch uses KKT conditions, the ETF symmetry $\varphi$6, and convexity (Zhou et al., 2 Sep 2025). A plausible implication is that the paper treats the ETF not merely as an initialization heuristic but as a target geometry encoded directly into the action-selection layer.

4. Algorithmic realizations

The ETF-based ACPG algorithm is intentionally minimal in how it alters discrete policy-gradient pipelines. One pre-computes a random simplex ETF

$\varphi$7

scales it so that $\varphi$8, freezes $\varphi$9, initializes backbone parameters $$L_{\mathrm{ACPG}}(\theta) = L_{\mathrm{pg}}(\theta) -\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))] -\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).$$0 at random, and then runs a standard discrete policy-gradient routine with policy

$$L_{\mathrm{ACPG}}(\theta) = L_{\mathrm{pg}}(\theta) -\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))] -\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).$$1

The loss is

$$L_{\mathrm{ACPG}}(\theta) = L_{\mathrm{pg}}(\theta) -\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))] -\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).$$2

possibly with value- or entropy-regularizers, but gradients are back-propagated only through $$L_{\mathrm{ACPG}}(\theta) = L_{\mathrm{pg}}(\theta) -\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))] -\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).$$3, not through $$L_{\mathrm{ACPG}}(\theta) = L_{\mathrm{pg}}(\theta) -\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))] -\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).$$4. The paper states that the method can be integrated into any discrete PG methods, explicitly listing REINFORCE, PPO, TRPO, and A3C, with the hyperparameter constraint $$L_{\mathrm{ACPG}}(\theta) = L_{\mathrm{pg}}(\theta) -\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))] -\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).$$5 so the ETF exists (Zhou et al., 2 Sep 2025).

The personalization ACPG algorithm modifies optimization rather than architecture. Its pseudocode samples a batch of contexts from data $$L_{\mathrm{ACPG}}(\theta) = L_{\mathrm{pg}}(\theta) -\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))] -\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).$$6, samples actions from $$L_{\mathrm{ACPG}}(\theta) = L_{\mathrm{pg}}(\theta) -\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))] -\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).$$7, observes rewards, computes $$L_{\mathrm{ACPG}}(\theta) = L_{\mathrm{pg}}(\theta) -\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))] -\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).$$8, $$L_{\mathrm{ACPG}}(\theta) = L_{\mathrm{pg}}(\theta) -\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))] -\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).$$9, and $\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$0, combines them by

$\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$1

updates $\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$2, and periodically updates the reference policy when $\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$3 (Starnes et al., 2023). Its practical recommendations are to start with $\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$4, monitor the test reward versus entropy curve, increase $\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$5 by $\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$6 if reward plateaus low and entropy is near zero, reduce $\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$7 if the policy becomes too uniform, anneal $\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$8 down to zero over training, and update $\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$9 every $$D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)), \quad D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),$$0–$$D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)), \quad D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),$$1 iterations (Starnes et al., 2023).

Quantum ACPG changes yet another component: action realization from quantum measurement. The underlying variational circuit uses an encoding layer $$D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)), \quad D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),$$2, a variational layer $$D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)), \quad D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),$$3, and an entangling layer $$D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)), \quad D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),$$4, repeated $$D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)), \quad D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),$$5 times so that

$$D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)), \quad D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),$$6

A projective measurement in the computational basis yields a bitstring $$D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)), \quad D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),$$7, and one shot suffices to sample an action through a classical post-processing map

$$D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)), \quad D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),$$8

The policy-gradient loss is

$$D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)), \quad D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),$$9

and gradients are evaluated by the parameter-shift rule on $$D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m), \qquad m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},$$0 or by SPSA (Meyer et al., 2022). The paper’s optimal decoding lemma constructs $$D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m), \qquad m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},$$1 so that, for $$D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m), \qquad m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},$$2,

$$D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m), \qquad m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},$$3

achieving $$D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m), \qquad m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},$$4.

5. Empirical behavior across domains

The personalization paper evaluates ACPG on MNIST classification as a 10-armed bandit, CIFAR-10 classification under a lightweight 2-layer net, and a Spotify synthetic music recommendation task with 50 tracks and user preferences (Starnes et al., 2023). The policy network uses two fully-connected hidden layers of 32 units, ReLU, and a softmax over $$D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m), \qquad m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},$$5 actions; batch size is $$D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m), \qquad m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},$$6; the optimizer is Adam with default learning rate $$D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m), \qquad m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},$$7; and $$D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m), \qquad m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},$$8 and $$D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m), \qquad m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},$$9 are chosen via grid search from $$D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,$$0. Typical settings are $$D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,$$1, $$D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,$$2, and $$D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,$$3 with Gaussian kernel $$D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,$$4. The reported metrics are average reward on the test set, policy entropy

$$D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,$$5

and the action-selection histogram over test contexts. Numerically, MNIST gives baseline PG reward $$D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,$$6 with collapsed entropy, while all ACPG variants reach reward $$D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,$$7 and maintain entropy near $$D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,$$8; CIFAR-10 improves from $$D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}}) = \sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,$$9 to $\varphi$00, described as a $\varphi$01 relative gain with entropy up $\varphi$02; and on Spotify the baseline selects only $\varphi$03 tracks, or $\varphi$04 coverage, while the MMD variant covers $\varphi$05 of tracks with reward $\varphi$06 (Starnes et al., 2023).

The ETF-based paper reports experiments on 10+ Gym tasks, naming CartPole, CarRacing, Breakout, Pong, Enduro, and Q*bert, with four PG algorithms—REINFORCE, PPO, TRPO, and A3C—and both MLP and CNN backbones (Zhou et al., 2 Sep 2025). In CartPole with REINFORCE+MLP, ACPG reaches max reward $\varphi$07 by epoch $\varphi$08 versus epoch $\varphi$09, a $\varphi$10 speedup. In CarRacing with PPO+CNN, ACPG achieves $\varphi$11 higher final reward in the same $\varphi$12 epochs. In Atari Enduro with TRPO+CNN, the baseline best is approximately $\varphi$13 and ACPG best is approximately $\varphi$14, or $\varphi$15. Q*bert with PPO+CNN gives $\varphi$16 best and $\varphi$17 final reward, while Pong with PPO+CNN improves the final score from approximately $\varphi$18 to approximately $\varphi$19, or $\varphi$20. The paper also reports that ACPG’s standard deviation across 20 seeds is uniformly lower than the baselines, and that in Pong+PPO+$\varphi$21-greedy it outperforms baseline across a broad range of $\varphi$22, peaking near $\varphi$23–$\varphi$24 (Zhou et al., 2 Sep 2025).

The quantum paper evaluates ACPG on CartPole-v0, FrozenLake, and ContextualBandits (Meyer et al., 2022). The reported metrics are average return per episode and convergence speed measured as the number of episodes to reach at least $\varphi$25 of optimal. On CartPole-v0, policies with larger $\varphi$26 learn in approximately $\varphi$27 episodes, with optimal return $\varphi$28, whereas $\varphi$29 never converges even after $\varphi$30 episodes; a SOFTMAX-VQC baseline lies in between. On FrozenLake, $\varphi$31 with $\varphi$32 converges in approximately $\varphi$33 episodes, whereas $\varphi$34 needs approximately $\varphi$35. On bandits with $\varphi$36, $\varphi$37 reaches $\varphi$38 average reward in approximately $\varphi$39 episodes, while sub-optimal partitions plateau near $\varphi$40. The hardware implementation on ibmq_manila uses 5 qubits, CX-only transpilation, a ContextualBandits task with $\varphi$41 and $\varphi$42 on $\varphi$43 qubits, CpU depth $\varphi$44, and batch size $\varphi$45 trajectories. Simulator performance reaches expected reward $\varphi$46; hardware without mitigation plateaus at approximately $\varphi$47; mitigated hardware plateaus at approximately $\varphi$48; and the trained test policy picks the optimal action approximately $\varphi$49 across all 8 contexts (Meyer et al., 2022).

6. Limitations, misconceptions, and research directions

The three ACPG variants solve different problems and therefore carry different limitations. The personalization ACPG has no formal monotonic improvement guarantee, relies on careful $\varphi$50 tuning, incurs $\varphi$51 cost per batch for the MMD term, was tested only in contextual bandits, and leaves extension to full MDPs with temporal credit assignment as future work (Starnes et al., 2023). Its ablations further indicate that KL and reverse KL both raise entropy, JS yields a smoother trade-off, Hellinger performs similarly to JS, TV tends to underperform in high-action domains, and MMD gives the highest final entropy and action coverage at the cost of slightly higher computation per update. Gaussian kernels with Hamming distance on action indices work best, while polynomial kernels give lower improvements (Starnes et al., 2023).

ETF-based ACPG is built around a stronger structural claim. Its theorem is stated for a frozen-head setting and an activation norm constraint, and the algorithm requires $\varphi$52 so the ETF exists (Zhou et al., 2 Sep 2025). The paper reports that even on Breakout and Pong, where the unmodified PG head never collapsed, the fixed ETF immediately forces the feature means to become equiangular. A plausible implication is that the method trades representational flexibility in the final layer for a geometry the paper identifies as optimal.

The quantum ACPG introduces negligible classical overhead relative to quantum-circuit runtime, with $\varphi$53 computable in $\varphi$54 per shot and one-time partition precomputation in $\varphi$55, but its hardware results also show the practical sensitivity of quantum reinforcement learning to noise and mitigation quality (Meyer et al., 2022). The paper states that overall classical overhead is less than $\varphi$56 of total runtime and that the single-measurement approach avoids $\varphi$57 expectation-value estimates per shot.

Taken together, the literature suggests that ACPG should be treated as an overloaded acronym rather than a single algorithmic lineage. One usage counteracts peaked policies by explicitly increasing diversity (Starnes et al., 2023); another induces a simplex ETF geometry in classical policy networks (Zhou et al., 2 Sep 2025); and another optimizes action decoding in variational quantum circuits (Meyer et al., 2022). The shared denominator is a modification of policy-gradient training to reshape either the action distribution, the action-head geometry, or the action-readout mechanism.