Parameterized Action Greedy Actor-Critic

• PAGAC is a reinforcement learning method for parameterized-action spaces that integrates discrete choices with continuous parameters.
• It uses a greedy cross-entropy policy improvement strategy to ensure faster convergence and stable training compared to entropy-based methods.
• Empirical results highlight PAGAC’s runtime efficiency and reliability in fully observable tasks, making it suitable for applications like match-plan generation and robotics.

Parameterized Action Greedy Actor-Critic (PAGAC) is the parameterized-action instantiation of Greedy Actor-Critic (GAC) for reinforcement-learning problems in which each action comprises both a discrete choice and an associated continuous parameter vector. In the comparative study that explicitly names PAGAC, it is not introduced as a separate reinforcement-learning family, but as the greedy actor-critic approach applied to parameterized action (PA) spaces; within that study, PAGAC is presented as the fastest and most reliable among the evaluated PA actor-critic variants for the reported benchmarks and the motivating match-plan generation setting (Bapoo et al., 3 Oct 2025).

1. Parameterized-action setting

PAGAC operates in the standard parameterized-action setting, also formalized in the parameterized-action Markov decision process (PAMDP) literature. A parameterized action space consists of a finite discrete action set

$\mathcal{A}_d = \{a_1, a_2, \dots, a_k\},$

together with, for each discrete action $a \in \mathcal{A}_d$, a continuous parameter space

$\mathcal{X}_a \subseteq \mathbb{R}^{m_a}.$

A complete action is therefore a tuple $(a, x)$, where $a$ is the discrete action and $x \in \mathcal{X}_a$ is the corresponding parameter vector, yielding the full action space

$$\mathcal{A} = \bigcup_{a \in \mathcal{A}_d} \{(a, x) \mid x \in \mathcal{X}_a\}.$$

This hierarchical discrete-then-continuous structure is standard in PAMDP formulations and provides a precise formal substrate for PAGAC-style methods (Fan et al., 2019).

In the comparative PAGAC study, this PA formalism is tied to web search match plan generation. A match plan is described as a sequence of rules and stopping conditions that determine how the index is scanned and which documents are matched. The corresponding reinforcement-learning problem is characterized by hybrid discrete-continuous actions, high-dimensional state representations, sparse rewards, and delayed feedback based on retrieval quality and response time (Bapoo et al., 3 Oct 2025).

2. Core algorithmic structure

Within the paper’s PA actor-critic framing, the actor must produce both a discrete action index and a continuous parameter vector for that action. The general parameterized-action actor-critic architecture is described as having dual actor networks or branches, shared state encoding, and a unified critic. PAGAC inherits this parameterized-action structure from the broader PA actor-critic template, but replaces SAC’s entropy-maximizing policy improvement with GAC’s greedy cross-entropy-based improvement (Bapoo et al., 3 Oct 2025).

The actor-critic backbone is given by standard return, value, and policy-gradient quantities: $G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k},$

$$V^\pi(s) = \mathbb{E}[G_t \mid s_t = s], \qquad Q^\pi(s, a) = \mathbb{E}[G_t \mid s_t = s, a_t = a],$$

$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t),$

$a \in \mathcal{A}_d$0

$a \in \mathcal{A}_d$1

The paper presents these equations as the learning backbone in which PAGAC operates (Bapoo et al., 3 Oct 2025).

The defining mechanism behind the “greedy” designation is summarized as a cross-entropy method for policy improvement: sample $a \in \mathcal{A}_d$2 actions from a proposal policy, evaluate them, select the top $a \in \mathcal{A}_d$3 actions with $a \in \mathcal{A}_d$4, and update the policy to increase the probability of those elite actions. The paper explains that this greedy improvement focuses learning on high-value actions, guarantees policy improvement in the original MDP per the cited GAC work, and avoids policy collapse by using entropy in the proposal policy rather than the final actor policy (Bapoo et al., 3 Oct 2025).

3. Relation to other parameterized-action actor-critic methods

The principal comparisons in the PAGAC study are with PASAC and PATQC. PASAC is the parameterized-action version of SAC and retains entropy regularization while handling discrete-plus-continuous actions. PATQC is the parameterized-action version of TQC and uses distributional critics with truncation to reduce overestimation bias. PAGAC differs from both by adopting greedy policy improvement rather than SAC’s maximum-entropy objective or TQC’s truncation-based critic design (Bapoo et al., 3 Oct 2025).

A useful contrast is provided by the hybrid actor-critic architecture instantiated as H-PPO. H-PPO addresses the same structured action class by decomposing the policy into multiple parallel sub-actor networks and training them with a shared critic. In the parameterized-action case, this yields a discrete actor $a \in \mathcal{A}_d$5, a continuous actor $a \in \mathcal{A}_d$6, and a state-value critic $a \in \mathcal{A}_d$7. Action selection is hierarchical and stochastic: the discrete actor samples an action from a softmax over the discrete choices, the continuous actor outputs parameter distributions, and the executed action is $a \in \mathcal{A}_d$8. Separate clipped PPO losses are applied to the discrete and continuous actors, both driven by the same advantage estimator (Fan et al., 2019). Relative to PAGAC, the salient distinction is that H-PPO uses stochastic hierarchical sampling and separate PPO objectives, whereas PAGAC is positioned around greedy cross-entropy-style policy improvement.

The relation to other actor-critic lines is therefore not merely one of implementation detail. PAGAC is a PA adaptation of GAC; H-PPO is a factorized PPO architecture for parameterized actions; and deep primal-dual actor-critic methods derive policy updates from Bellman duality rather than greedy PA control. The deep primal-dual formulation is conceptually adjacent because its dual policy update can coincide with an actor-critic policy gradient under stationary sampling, but it samples actions from a softmax policy and does not implement parameterized-action greedy control (Cho et al., 2017).

4. Benchmark environments and experimental protocol

The reported PAGAC experiments were conducted in fully observable environments, and the authors state that this lets them remove recurrent networks. The stated consequences are reduced training complexity, computational overhead, risk of optimization instability, and inference latency. This architectural simplification is presented as part of the paper’s emphasis on speed and efficiency (Bapoo et al., 3 Oct 2025).

Two standard PA benchmarks from Bester et al. (2021) are used. Platform-v0 is a platform-style task with three discrete actions—run, hop, and leap—where each action has a continuous parameter controlling horizontal displacement. Its state is 9-dimensional and includes the agent’s and nearby opponent’s positions and velocities together with platform characteristics. Goal-v0 is a simplified RoboCup-style robot soccer task with three discrete actions—kick-to, shoot-goal-left, and shoot-goal-right—each with continuous parameters; its state is 14-dimensional and encodes positions, velocities, orientations, and ball- and goal-related distances (Bapoo et al., 3 Oct 2025).

The comparative protocol evaluates SAC/PASAC, GAC/PAGAC, and TQC/PATQC on those two environments. Hyperparameter optimization was performed with Microsoft NNI, the codebase was modified for GAC and TQC to ensure reproducibility, 10 tests were conducted, and the reported results were based on mean training incentives, final evaluation scores averaged over repetitions, and wall-clock time to complete 5,000 episodes (Bapoo et al., 3 Oct 2025).

5. Reported empirical performance

The strongest reported result is runtime efficiency. On Platform-v0, PAGAC is both fastest and best-performing in return among the compared PA variants. On Goal-v0, PAGAC is again the fastest method, while the return values are described in the source as awkwardly presented: as written in the table, PATQC and PASAC have slightly better returns than PAGAC, even though the paper’s narrative emphasizes PAGAC’s speed and stability (Bapoo et al., 3 Oct 2025).

Benchmark	PAGAC	PASAC / PATQC
Platform-v0	Return 0.170; time 41:24	PASAC 0.143; 56:35. PATQC 0.145; 51:38
Goal-v0	Return -8.552; time 24:04	PASAC -8.200; 26:55. PATQC -8.155; 27:19

The paper repeatedly claims that PAGAC converges faster, trains more quickly over 5,000 episodes, is more stable, and yields reliable decision-making in high-dimensional PA spaces. It also states that PAGAC is consistently the fastest method on both benchmarks, that PATQC is generally second fastest, and that PASAC is slowest (Bapoo et al., 3 Oct 2025).

These results place PAGAC within a specific empirical niche. The paper’s central takeaway is not that greedy actor-critic dominates all PA criteria on every benchmark statistic, but that the combination of training speed, convergence behavior, and practical reliability favors PAGAC under the reported fully observable protocol. This suggests that, in the authors’ framing, greedy policy improvement is especially attractive when rapid convergence is operationally important.

6. Scope, limitations, and prospective extensions

The comparative study’s future-work discussion proposes hybrid strategies combining entropy regularization from SAC with truncation-based ideas from TQC, with the aim of improving stability and performance further. It also calls for broader exploration of generalizability across more tasks and settings (Bapoo et al., 3 Oct 2025).

Several limitations are either explicit or directly inferable from the reported setup. The empirical evaluation is narrow: two benchmark tasks, 5,000 episodes, no extensive ablation study, and limited detail on the exact hyperparameter search and architecture choices are all identified in the supplied summary as limitations that can be inferred from the paper (Bapoo et al., 3 Oct 2025). A plausible implication is that the reported runtime and stability advantages should be interpreted as benchmark-specific evidence rather than as a fully general characterization of PA actor-critic behavior.

A second boundary concerns architectural assumptions. The paper’s removal of recurrent networks depends on full observability. This suggests that the reported simplicity and efficiency gains are tied to settings in which the current state already contains the relevant information; the comparative summary does not establish that the same conclusions would hold under partial observability or in environments requiring recurrence.

Within the broader PA literature, PAGAC therefore occupies a defined position: it is a greedy actor-critic method for hybrid discrete-continuous action spaces, empirically emphasized for speed, stability, and efficiency under a fully observable evaluation protocol, and conceptually distinct from stochastic factorization methods such as H-PPO and from Bellman-duality-based actor-critic variants (Fan et al., 2019).