---
title: Action Collapse Policy Gradient (ACPG)
url: https://www.emergentmind.com/topics/action-collapse-policy-gradient-acpg
type: topic
---

# Action Collapse Policy Gradient (ACPG)

Searching arXiv for the cited ACPG-related papers and any closely related usages of the acronym.
Action Collapse Policy Gradient (ACPG) is not a single uniformly defined algorithm in the arXiv literature, but a shared acronym used for multiple policy-gradient modifications that target different failure modes and architectural constraints. In one line of work, ACPG augments standard policy-gradient optimization with $\varphi$-divergence and Maximum Mean Discrepancy penalties to prevent entropy collapse in personalization tasks [2310.05324]. In a second line, ACPG fixes the policy head to a simplex equiangular tight frame (ETF) so that policy-network features provably exhibit an Action Collapse geometry analogous to Neural Collapse [2509.02737]. In a third, quantum ACPG denotes a variational quantum policy-gradient algorithm with optimized action decoding, in which a single computational-basis measurement and a classical decoding map define the policy [2212.06663]. The commonality across these uses is a modification of policy-gradient structure rather than a shared mathematical objective.

## 1. Terminological scope and conceptual distinctions

A concise way to organize the literature is to distinguish three ACPG meanings by the mechanism each paper changes: regularization of the policy objective, geometric constraints on the action head, or quantum action decoding. The acronym therefore names different interventions at different layers of the policy-gradient pipeline.

| Usage of ACPG | Core mechanism | Reported setting |
|---|---|---|
| Diversity-promoting ACPG [2310.05324] | $\varphi$-divergence and/or MMD regularization | MNIST, CIFAR-10, Spotify personalization |
| ETF-based ACPG [2509.02737] | Freeze the final action head as a simplex ETF | OpenAI Gym with REINFORCE, PPO, TRPO, A3C |
| Quantum ACPG [2212.06663] | Optimized classical decoding of quantum measurements | CartPole-v0, FrozenLake, ContextualBandits, 5-qubit hardware |

A recurrent source of confusion is the phrase *action collapse*. In the personalization paper, action collapse is also called entropy collapse and is a pathology: a learned policy $\pi_\theta(a\mid s)$ assigns near-zero probability to many actions, reducing exploration and coverage [2310.05324]. In the ETF-based paper, Action Collapse is the analogue in policy-gradient networks of Neural Collapse and is presented as an optimal geometric structure of last-layer activations and action-head weights [2509.02737]. These are not the same phenomenon, even though the acronym ACPG is used in both settings.

## 2. Diversity-promoting ACPG for entropy collapse in personalization

In the personalization formulation, the starting point is the standard surrogate objective
$$
L_{\mathrm{pg}}(\theta)=\mathbb{E}_{s\sim D,\;a\sim \pi_\theta(\cdot\mid s)}[\,r(s,a)\cdot \log \pi_\theta(a\mid s)\,].
$$
The method augments this objective with regularizers constructed from various $\varphi$-divergences and Maximum Mean Discrepancy, encouraging the current policy to follow a different state visitation and/or action choice distribution than previously computed policies [2310.05324].

The full ACPG objective is
$$
L_{\mathrm{ACPG}}(\theta)
=
L_{\mathrm{pg}}(\theta)
-\lambda_\phi\,\mathbb{E}_{s}[D_\phi(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s))]
-\lambda_{\mathrm{MMD}}\,\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}).
$$
The reference policy is $\pi_{\mathrm{old}}=\pi_{\theta_{\mathrm{prev}}}$, or a reference policy such as uniform. The paper enumerates several admissible policy divergences, including
$$
D_{\mathrm{KL}}(\pi_\theta(\cdot\mid s)\Vert \pi_{\mathrm{old}}(\cdot\mid s)),
\quad
D_{\mathrm{KL\text{-}rev}}(\pi_\theta\Vert\pi_{\mathrm{old}})
=
D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert\pi_\theta),
$$
the Jensen–Shannon divergence
$$
D_{\mathrm{JS}}(\pi_\theta\Vert\pi_{\mathrm{old}})
=
\frac12 D_{\mathrm{KL}}(\pi_\theta\Vert m)+\frac12 D_{\mathrm{KL}}(\pi_{\mathrm{old}}\Vert m),
\qquad
m=\frac{\pi_\theta+\pi_{\mathrm{old}}}{2},
$$
the Hellinger distance
$$
D_{H^2}(\pi_\theta\Vert\pi_{\mathrm{old}})
=
\sum_a (\sqrt{\pi_\theta(a\mid s)}-\sqrt{\pi_{\mathrm{old}}(a\mid s)})^2,
$$
and total variation
$$
D_{\mathrm{TV}}(\pi_\theta,\pi_{\mathrm{old}})
=
\frac12\sum_a |\pi_\theta(a\mid s)-\pi_{\mathrm{old}}(a\mid s)|.
$$

The MMD term is defined on the state–action distribution embedded in an RKHS $\mathcal H$ with kernel $k((s,a),(s',a'))$. If $\mu_\theta=\mathbb{E}_{s\sim D,a\sim\pi_\theta}[\phi(s,a)]$, then
$$
\mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}})
=
\|\mu_\theta-\mu_{\mathrm{old}}\|_{\mathcal H}^2.
$$
The paper’s interpretation is operational: each regularizer adds gradient components that are nonzero for every action, and in the MMD case for every state–action pair, so probability mass is reallocated toward less-selected actions [2310.05324].

The training routine retains the usual policy-gradient term
$$
g_{\mathrm{pg}} \leftarrow \frac1N \sum_i r_i \cdot \nabla_\theta \log \pi_\theta(a_i\mid s_i),
$$
adds a divergence gradient
$$
g_\phi \leftarrow \frac1N \sum_i \nabla_\theta D_\phi(\pi_\theta(\cdot\mid s_i)\Vert\pi_{\mathrm{old}}(\cdot\mid s_i)),
$$
and an MMD gradient
$$
g_{\mathrm{MMD}}\leftarrow \nabla_\theta \mathrm{MMD}^2(\pi_\theta,\pi_{\mathrm{old}}),
$$
then updates
$$
\theta \leftarrow \theta + \alpha \cdot (g_{\mathrm{pg}}-\lambda_\phi g_\phi-\lambda_{\mathrm{MMD}} g_{\mathrm{MMD}}),
$$
with periodic reference-policy refresh $\,\pi_{\mathrm{old}}\leftarrow\pi_\theta$ every $K$ iterations. The paper states that it does not derive a formal convergence proof, but under standard assumptions on smoothness and bounded gradients, gradient ascent on $L_{\mathrm{ACPG}}(\theta)$ converges to a stationary point [2310.05324].

## 3. ETF-based ACPG and the geometry of Action Collapse

The 2025 paper defines Action Collapse as the policy-gradient analogue of Neural Collapse. When a policy DNN has reached optimal performance under certain ideal conditions, its last-layer state–action activations exhibit within-action collapse, global simplex structure, and alignment of the policy head [2509.02737].

Within-action collapse is formulated by partitioning states according to their true optimal action. If
$$
\{h_{i,k}\}_{i=1}^{n_k}\subset \mathbb{R}^d
$$
are activations for states in $\mathcal S_k=\{s:\mathrm{opt}(s)=a_k\}$, then
$$
h_{i,k}\to \mu_k:=\frac1{n_k}\sum_{i=1}^{n_k} h_{i,k},
$$
and the within-class covariance
$$
\Sigma_k=\frac1{n_k}\sum_i (h_{i,k}-\mu_k)(h_{i,k}-\mu_k)^T
$$
converges to zero. The class means, after centering by the global mean $\mu_G$, form the vertices of a simplex equiangular tight frame:
$$
\langle \mu_k-\mu_G,\;\mu_j-\mu_G\rangle=-\frac1{K-1}\quad (k\neq j),
\qquad
\|\mu_k-\mu_G\|=1.
$$
The action-selection weights collapse to the same ETF:
$$
\langle w_k,w_j\rangle=
\begin{cases}
1, & k=j,\\[6pt]
-\dfrac1{K-1}, & k\neq j.
\end{cases}
$$

The paper’s ACPG method turns this geometry into an architectural constraint. A simplex ETF is defined by vectors $\{m_k\}_{k=1}^K\subset\mathbb R^d$ with $d\ge K-1$ such that there exists an orthonormal $U\in\mathbb R^{d\times K}$ satisfying
$$
M=[m_1,\dots,m_K]
=
\sqrt{\frac{K}{K-1}}\;
U\Big(I_K-\frac1K\mathbf 1_K\mathbf 1_K^T\Big),
$$
which implies
$$
\langle m_i,m_j\rangle
=
\frac{K}{K-1}\delta_{ij}-\frac1{K-1}.
$$
ACPG fixes the action-head weight matrix
$$
W^*=[w_1^*,\dots,w_K^*]\in\mathbb R^{d\times K}
$$
to be exactly such an ETF, optionally scaled by $\sqrt{E_W}$ so that $\|w_k^*\|^2=E_W$, and sets the target class means to
$$
\mu_k=w_k^*,\qquad k=1,\dots,K.
$$

The analytical core is Theorem 1. In a Layer-Peeled Model with feature map $h(s;\theta)\in\mathbb R^d$ and fixed final softmax weights $W^*$, the objective is
$$
\max_{\mathcal H}\quad
J(\mathcal H)
=
\sum_{k=1}^K\sum_{s\in \mathcal S_k} d^\pi(s)
\log
\frac{e^{h_s^T w_k^*}}
{\sum_{j=1}^K e^{h_s^T w_j^*}}
$$
subject to $\|h_s\|^2\le E_H$ for every $s$. The theorem states informally that, regardless of whether $\mathcal S$ is fully explored, balanced, or uniform, any global maximizer $\mathcal H^*$ satisfies
$$
h_s^*=\sqrt{E_H}\;w_{\mathrm{opt}(s)}^*,
\qquad s\in\mathcal S.
$$
The proof sketch uses KKT conditions, the ETF symmetry $\sum_{k=1}^K w_k^*=0$, and convexity [2509.02737]. A plausible implication is that the paper treats the ETF not merely as an initialization heuristic but as a target geometry encoded directly into the action-selection layer.

## 4. Algorithmic realizations

The ETF-based ACPG algorithm is intentionally minimal in how it alters discrete policy-gradient pipelines. One pre-computes a random simplex ETF
$$
W^*
=
\sqrt{\frac{K}{K-1}}\;
U\Big(I_K-\frac1K\mathbf 1\mathbf 1^T\Big)\in\mathbb R^{d\times K},
$$
scales it so that $\|w_k^*\|^2=E_W$, freezes $W\leftarrow W^*$, initializes backbone parameters $\theta$ at random, and then runs a standard discrete policy-gradient routine with policy
$$
\pi_\theta(a\mid s)=\mathrm{softmax}(h(s;\theta)^T W^*).
$$
The loss is
$$
L(\theta)=-\sum_t \hat A_t \log \pi_\theta(a_t\mid s_t),
$$
possibly with value- or entropy-regularizers, but gradients are back-propagated only through $\theta$, not through $W^*$. The paper states that the method can be integrated into any discrete PG methods, explicitly listing REINFORCE, PPO, TRPO, and A3C, with the hyperparameter constraint $d\ge K-1$ so the ETF exists [2509.02737].

The personalization ACPG algorithm modifies optimization rather than architecture. Its pseudocode samples a batch of contexts from data $D$, samples actions from $\pi_\theta$, observes rewards, computes $g_{\mathrm{pg}}$, $g_\phi$, and $g_{\mathrm{MMD}}$, combines them by
$$
\text{gradient}\leftarrow g_{\mathrm{pg}}-\lambda_\phi g_\phi-\lambda_{\mathrm{MMD}} g_{\mathrm{MMD}},
$$
updates $\theta\leftarrow \theta+\alpha\cdot\text{gradient}$, and periodically updates the reference policy when $t\bmod K=0$ [2310.05324]. Its practical recommendations are to start with $\lambda_\phi,\lambda_{\mathrm{MMD}}=10^{-2}$, monitor the test reward versus entropy curve, increase $\lambda$ by $\times2$ if reward plateaus low and entropy is near zero, reduce $\lambda$ if the policy becomes too uniform, anneal $\lambda_\phi$ down to zero over training, and update $\pi_{\mathrm{old}}$ every $K=5$–$10$ iterations [2310.05324].

Quantum ACPG changes yet another component: action realization from quantum measurement. The underlying variational circuit uses an encoding layer $U_{s,\lambda}$, a variational layer $U_\theta$, and an entangling layer $U_{\mathrm{ent}}$, repeated $d$ times so that
$$
U_{s,\lambda,\theta}=(U_{\mathrm{ent}}U_\theta U_{s,\lambda})^d,
\qquad
|\psi_{s,\Theta}\rangle=U_{s,\lambda,\theta}|0\rangle^{\otimes n}.
$$
A projective measurement in the computational basis yields a bitstring $b\in\{0,1\}^n$, and one shot suffices to sample an action through a classical post-processing map
$$
a=f_C(b).
$$
The policy-gradient loss is
$$
L(\Theta)=-\mathbb E_t[\ln \pi_\Theta(a_t\mid s_t)\,G_t],
$$
and gradients are evaluated by the parameter-shift rule on $\langle P_a\rangle$ or by SPSA [2212.06663]. The paper’s optimal decoding lemma constructs $f_C$ so that, for $|A|=M=2^{m+1}$,
$$
f_C(b)=\big[b_0\cdots b_{m-1}\oplus_{i=m}^{n-1} b_i\big]_{10},
$$
achieving $G_{f_C}=n$.

## 5. Empirical behavior across domains

The personalization paper evaluates ACPG on MNIST classification as a 10-armed bandit, CIFAR-10 classification under a lightweight 2-layer net, and a Spotify synthetic music recommendation task with 50 tracks and user preferences [2310.05324]. The policy network uses two fully-connected hidden layers of 32 units, ReLU, and a softmax over $n$ actions; batch size is $100$; the optimizer is Adam with default learning rate $10^{-3}$; and $\lambda_\phi$ and $\lambda_{\mathrm{MMD}}$ are chosen via grid search from $\{10^{-4},10^{-3},10^{-2},10^{-1},1\}$. Typical settings are $\lambda_{\mathrm{ent}}\approx 0.1$, $\lambda_{\mathrm{js}}\approx 0.01$, and $\lambda_{\mathrm{mmd}}\approx 0.01$ with Gaussian kernel $\sigma=1$. The reported metrics are average reward on the test set, policy entropy
$$
H(\pi_\theta)=\mathbb E_s\Big[-\sum_a \pi_\theta(a\mid s)\log \pi_\theta(a\mid s)\Big],
$$
and the action-selection histogram over test contexts. Numerically, MNIST gives baseline PG reward $\sim 0.6$ with collapsed entropy, while all ACPG variants reach reward $\to 1.0$ and maintain entropy near $\log 10$; CIFAR-10 improves from $\sim 0.25$ to $\sim 0.30$, described as a $\sim 20\%$ relative gain with entropy up $30\%$; and on Spotify the baseline selects only $3/50$ tracks, or $\sim 6\%$ coverage, while the MMD variant covers $\sim 40\%$ of tracks with reward $+10\%$ [2310.05324].

The ETF-based paper reports experiments on 10+ Gym tasks, naming CartPole, CarRacing, Breakout, Pong, Enduro, and Q*bert, with four PG algorithms—REINFORCE, PPO, TRPO, and A3C—and both MLP and CNN backbones [2509.02737]. In CartPole with REINFORCE+MLP, ACPG reaches max reward $500$ by epoch $20$ versus epoch $33$, a $33\%$ speedup. In CarRacing with PPO+CNN, ACPG achieves $+50\%$ higher final reward in the same $100$ epochs. In Atari Enduro with TRPO+CNN, the baseline best is approximately $1243$ and ACPG best is approximately $1473$, or $+19\%$. Q*bert with PPO+CNN gives $+7\%$ best and $+6\%$ final reward, while Pong with PPO+CNN improves the final score from approximately $19.8$ to approximately $20.9$, or $+5\%$. The paper also reports that ACPG’s standard deviation across 20 seeds is uniformly lower than the baselines, and that in Pong+PPO+$\epsilon$-greedy it outperforms baseline across a broad range of $\epsilon$, peaking near $\epsilon=0.01$–$0.1$ [2509.02737].

The quantum paper evaluates ACPG on CartPole-v0, FrozenLake, and ContextualBandits [2212.06663]. The reported metrics are average return per episode and convergence speed measured as the number of episodes to reach at least $95\%$ of optimal. On CartPole-v0, policies with larger $G_{f_C}$ learn in approximately $100$ episodes, with optimal return $200$, whereas $G_{f_C}=1.0$ never converges even after $5{,}000$ episodes; a SOFTMAX-VQC baseline lies in between. On FrozenLake, $G=4.0$ with $d=2$ converges in approximately $1{,}000$ episodes, whereas $G=2.0$ needs approximately $2{,}500$. On bandits with $|A|=8$, $G=5.0$ reaches $-0.1$ average reward in approximately $5{,}000$ episodes, while sub-optimal partitions plateau near $-0.75$. The hardware implementation on ibmq_manila uses 5 qubits, CX-only transpilation, a ContextualBandits task with $|S|=8$ and $|A|=2$ on $n=3$ qubits, CpU depth $d=1$, and batch size $B=50$ trajectories. Simulator performance reaches expected reward $1.0$; hardware without mitigation plateaus at approximately $0.1$; mitigated hardware plateaus at approximately $0.3$; and the trained test policy picks the optimal action approximately $91\%$ across all 8 contexts [2212.06663].

## 6. Limitations, misconceptions, and research directions

The three ACPG variants solve different problems and therefore carry different limitations. The personalization ACPG has no formal monotonic improvement guarantee, relies on careful $\lambda$ tuning, incurs $O(N^2)$ cost per batch for the MMD term, was tested only in contextual bandits, and leaves extension to full MDPs with temporal credit assignment as future work [2310.05324]. Its ablations further indicate that KL and reverse KL both raise entropy, JS yields a smoother trade-off, Hellinger performs similarly to JS, TV tends to underperform in high-action domains, and MMD gives the highest final entropy and action coverage at the cost of slightly higher computation per update. Gaussian kernels with Hamming distance on action indices work best, while polynomial kernels give lower improvements [2310.05324].

ETF-based ACPG is built around a stronger structural claim. Its theorem is stated for a frozen-head setting and an activation norm constraint, and the algorithm requires $d\ge K-1$ so the ETF exists [2509.02737]. The paper reports that even on Breakout and Pong, where the unmodified PG head never collapsed, the fixed ETF immediately forces the feature means to become equiangular. A plausible implication is that the method trades representational flexibility in the final layer for a geometry the paper identifies as optimal.

The quantum ACPG introduces negligible classical overhead relative to quantum-circuit runtime, with $f_C(b)$ computable in $O(n)$ per shot and one-time partition precomputation in $O(n\cdot M)$, but its hardware results also show the practical sensitivity of quantum reinforcement learning to noise and mitigation quality [2212.06663]. The paper states that overall classical overhead is less than $1\%$ of total runtime and that the single-measurement approach avoids $O(|A|)$ expectation-value estimates per shot.

Taken together, the literature suggests that ACPG should be treated as an overloaded acronym rather than a single algorithmic lineage. One usage counteracts peaked policies by explicitly increasing diversity [2310.05324]; another induces a simplex ETF geometry in classical policy networks [2509.02737]; and another optimizes action decoding in variational quantum circuits [2212.06663]. The shared denominator is a modification of policy-gradient training to reshape either the action distribution, the action-head geometry, or the action-readout mechanism.

Source: https://www.emergentmind.com/topics/action-collapse-policy-gradient-acpg