---
title: Expanded Action Space (ExpA) Overview
url: https://www.emergentmind.com/topics/expanded-action-space-expa
type: topic
---

# Expanded Action Space (ExpA) Overview

Searching arXiv for the cited papers to ground the article in the current record.
Expanded Action Space (ExpA) denotes a family of technical constructions in which the operative space of alternatives is enlarged beyond a flat primitive action set. In current arXiv usage, that enlargement takes markedly different forms depending on domain: in the Information Theory of Action (ITA), ExpA is the joint space of action values and endpoints endowed with a density of action states; in reinforcement learning, it appears as parameterized discrete–continuous actions, macro actions with multiple execution times, or nested curricula of progressively larger subspaces; in large-action-space optimization, it motivates representative subsets and dynamic compaction; and in large language model systems, it extends the output space beyond vocabulary tokens to routing actions and environment-specific primitives [2602.09984] [1511.04143] [1903.01344] [2212.03540] [1906.12266] [2511.08043] [2510.07581] [2511.22104].

## 1. Conceptual scope and formal variants

The unifying feature of ExpA is not a single canonical mathematical object, but a structural shift in what counts as an action or dynamical alternative. In ITA, the primitive object is not a probability over trajectories but a nonnegative multiplicity density $g(A; b, T \mid a)$ over action values $A$ and endpoints $b$, with ExpA defined formally as $(\mathcal{A}\times\mathcal{Q}, g, +)$. In parameterized-action RL, the action becomes a tuple $(a,x)$ combining a discrete action type and continuous parameters. In EASpace, primitive actions are augmented with macro actions $m_e^{(\tau)}$ derived from expert policies and indexed by execution duration. In Growing Action Spaces (GAS), the “expansion” is a sequence of nested subspaces $\mathcal{A}^{(1)} \subset \mathcal{A}^{(2)} \subset \cdots \subset \mathcal{A}^{(K)}=\mathcal{A}$. In LLM-centered formulations, the action space becomes the union of vocabulary tokens and non-linguistic environment actions, $A=A_{\text{lang}}\cup A_{\text{env}}$. In DynaAct, the full proxy space $A$ is large and general-purpose, while the actual per-step candidate set is a dynamically selected compact subset $\mathcal{A}_t\subset A$ [2602.09984] [1511.04143] [1903.01344] [2212.03540] [1906.12266] [2510.07581] [2511.08043].

| Setting | Formal expanded space | Operational role |
|---|---|---|
| ITA | $(\mathcal{A}\times\mathcal{Q}, g, +)$ | Encodes multiplicity of action-valued dynamical alternatives |
| Parameterized RL | $\mathcal{A}=\{(a,x)\mid a\in A_d,\ x\in X_a\}$ | Couples discrete skills with continuous control parameters |
| Macro/curriculum RL | $\tilde A=A\cup M$ or $\mathcal{A}^{(1)}\subset\cdots\subset\mathcal{A}^{(K)}$ | Adds temporal abstraction or staged enlargement |
| LLM/tool use | $A=A_{\text{lang}}\cup A_{\text{env}}$ | Separates reasoning tokens from control actions |
| Dynamic compaction | $\mathcal{A}_t\subset A$ or $\mathcal{A}_r(s)\subseteq \mathcal{A}_{\mathrm{full}}(s)$ | Restricts search to compact, representative candidates |

A recurring consequence is that expansion is typically paired with structure. The enlarged space is not left unregulated: ITA imposes additivity and finite action resolution; PAMDP methods impose architectural factorization; EASpace imposes semi-MDP semantics and duration-indexed macros; GAS imposes nestedness; DynaAct imposes submodular selection; EARL imposes environment-conditioned action masks. This suggests that “expanded” in ExpA rarely means unrestricted proliferation of choices. More often, it means introducing additional latent or explicit degrees of freedom while preserving tractable composition, credit assignment, or inference.

## 2. Expanded Action Space in the Information Theory of Action

In ITA, a system is specified by a configuration space $\mathcal{Q}$ and an action space $\mathcal{A}\subset\mathbb{R}$. For any $a,b\in\mathcal{Q}$ and duration $T>0$, the multiplicity of dynamical alternatives connecting $a\to b$ with total action $A$ is encoded in a nonnegative integrable density $g(A; b,T\mid a)$ with finite variance and a convolution-composition law induced by action additivity:
$$
g(A; c,T_1+T_2\mid a)=\int_{\mathcal{Q}} db\int dA' \, g(A'; b,T_1\mid a)\, g(A-A'; c,T_2\mid b).
$$
ExpA generalizes classical Hamiltonian stationarity in two ways: it replaces the single stationary value $S_{\mathrm{cl}}$ by the full joint space $(A,b)$ with multiplicity $g$, and, under finite action resolution, it requires sufficiently close action contributions to be combined coherently rather than treated as distinguishable alternatives [2602.09984].

Probabilities are introduced only after inference. ITA assigns a joint density $P(A,b\mid a)$ by maximizing relative entropy with normalization and fixed mean action, yielding the exponential family
$$
P(A,b\mid a)=\frac{g(A; b,T\mid a)e^{-\eta A}}{Z(\eta;a)}.
$$
For this family, the Fisher information in $\eta$ equals $\mathrm{Var}(A)$, so the Cramér–Rao bound gives $\Delta A\,\Delta \eta \ge 1$, and naturalness implies a minimal action resolution $\Delta A_{\min}\sim 1/\eta$. The infinitesimal-time argument is then decisive: action increments scale as $\delta S\sim L\,dt$, the standard deviation scales as $\sqrt{dt}$, and $|\delta S_1-\delta S_2|/\Delta(\delta S)_{\min}\to 0$ as $dt\to 0$. All infinitesimal alternatives are therefore operationally indistinguishable and must combine coherently [2602.09984].

The representation problem is resolved by requiring associative and commutative combination of indistinguishable alternatives, multiplicativity under sequential composition, nonnegative probability extraction, classical additivity for distinguishable alternatives, and preservation of probability normalization under nontrivial dynamics. Real nonnegative weights fail because continuous characters $\chi(A)=e^{\lambda A}$ cannot remain bounded except in the trivial case. The minimal faithful real-linear setting is two-dimensional and isomorphic to $\mathbb{C}$; commutativity excludes quaternions; and classical additivity together with continuity yields a positive semidefinite quadratic form, so $P=|\phi|^2$. The unique continuous unit-modulus character is then
$$
\chi(A)=e^{i\eta A}.
$$
From this follow the propagator
$$
K(b,T\mid a)=\int dA\, g(A; b,T\mid a)e^{i\eta A},
$$
the semigroup composition law, unitarity, Hilbert-space structure, Stone’s theorem, the short-time kernel, the Schrödinger equation, and the classical limit via stationary phase. The action scale is fixed empirically by interference and identified with $\hbar$, so $\eta=1/\hbar$ and the coherence parameter is $A/\hbar$ [2602.09984].

## 3. Hybrid and parameterized action spaces in reinforcement learning

In parameterized-action Markov decision processes, the action space is expanded from a flat discrete or continuous set to a hybrid object
$$
\mathcal{A}=\{(a,x)\mid a\in A_d,\ x\in X_a\},
$$
where each discrete action type has its own continuous parameter space. The 2015 RoboCup HFO formulation uses four discrete actions—Dash, Turn, Tackle, Kick—with bounded continuous parameters such as power and direction. The actor network outputs four discrete-action scores and six continuous parameters, while the critic consumes the state together with all actor outputs and returns a scalar $Q(s,a,x)$. Training uses deterministic backpropagation through $Q$ for both the discrete head and the continuous parameters, replay memory, target networks with $\tau=10^{-4}$, and an “inverting gradients” rule to enforce parameter bounds; among inverting, zeroing, and squashing, only inverting yielded robust learning. In evaluation over 100 episodes, three trained DDPG agents scored more reliably than the Helios 2012 RoboCup-2D champion, although they took more timesteps on average [1511.04143].

The later hybrid actor-critic formulation generalizes the same structured view through multiple parallel sub-actors and one shared critic. In H-PPO, a discrete actor learns $\pi_{\theta_d}(a\mid s)$, a continuous actor learns parameter distributions, and a shared state-value critic $V(s)$ supplies common advantages. The method uses separate PPO clipped objectives,
$$
L_d^{\mathrm{CLIP}}(\theta_d),\qquad L_c^{\mathrm{CLIP}}(\theta_c),
$$
with an A3C-style truncated-return advantage estimator. The key architectural rationale is to avoid over-parameterized critics of the form $Q(s,a,x_{a_1},\ldots,x_{a_k})$ when the true dependence is only $Q(s,a,x_a)$. Empirically, H-PPO outperformed DQN, discretization, and P-DQN on Catching Point, Moving, Chase and Attack, and Half Field Football, reaching 95.39% success rate and mean episode reward 9.849 on Half Field Football [1903.01344].

Across these RL formulations, ExpA serves two functions simultaneously. First, it increases expressivity by allowing action semantics to depend on internal structure rather than a monolithic identifier. Second, it changes the credit-assignment problem: learning must propagate signal across discrete choice, parameter selection, and often bounds or hierarchy. The data indicate that successful methods therefore rely on decomposition—actor heads, sub-actors, or critics that respect the factorized semantics of the action space—rather than treating the expanded space as a flat enumeration [1511.04143] [1903.01344].

## 4. Temporal abstraction and internal curricula

A distinct RL interpretation expands the action space temporally. EASpace augments the primitive action set with macro actions derived from sub-optimal expert policies and indexed by duration, yielding $\tilde A=A\cup M$ with $M=\{m_e^{(\tau)}: e\in E,\ \tau\in T\}$. A macro executes expert policy $\pi_e$ for exactly $\tau$ steps, inducing semi-MDP semantics with duration function $\tau(a)$, cumulative shaped return, and next decision state $s_{t+\tau}$. To encourage exploitation of useful long-duration macros, EASpace adds an intrinsic reward bonus for macro actions and learns with semi-MDP Q-learning plus an Intra-Macro-Action Learning Rule (IMALR), which performs tail-return updates at intermediate states visited during macro execution. Under bounded shaped rewards, $\gamma\in(0,1)$, infinite visitation, Robbins–Monro step sizes, and bounded macro bonus, IMALR converges almost surely to the optimal action-value function for the shaped-reward semi-MDP [2212.03540].

GAS expands action spaces through an internal curriculum rather than temporal extension. It defines nested subspaces
$$
\mathcal{A}^{(1)}\subset \mathcal{A}^{(2)}\subset \cdots \subset \mathcal{A}^{(K)}=\mathcal{A},
$$
samples a level $\ell$ per episode, rolls out under $\mathcal{A}^{(\ell)}$, and updates all value heads $Q_k$ for $k\ge \ell$ because actions taken in smaller spaces remain valid in larger ones. The method uses shared encoders, head-specific decoders, and, when a parent–child structure exists, hierarchical delta initialization
$$
\hat Q^*_{\ell+1}(s,a)=\hat Q^*_\ell(s,\mathrm{parent}_\ell(a))+\Delta_\ell(s,a).
$$
The theoretical backbone is monotonicity: if $\mathcal{A}^{(i)}\subset \mathcal{A}^{(j)}$, then $V_i^*(s)\le V_j^*(s)$ for all states. Empirically, GAS improved exploration and asymptotic performance in discretized continuous control and large-scale StarCraft micromanagement; disabling off-action-space transfer or removing hierarchical composition degraded results [1906.12266].

These two lines treat expansion differently. EASpace enlarges the available action alphabet by adding expert-derived temporally extended choices; GAS keeps the full action set fixed in principle but exposes it progressively through nested masks. The shared implication is that expanded action spaces can be used as a vehicle for long-horizon credit assignment and exploration control, not merely as a static representational choice [2212.03540] [1906.12266].

## 5. Dynamic compaction and representative subsets

Another line of work starts from very large action spaces and asks how to construct compact subsets without sacrificing utility. DynaAct does this for LLM reasoning by first estimating a large proxy action space $A$ from “observation sketches” extracted from a diverse corpus, then selecting a small state-dependent candidate set $\mathcal{A}_t$ at each reasoning step by maximizing a submodular objective combining utility and diversity:
$$
F(\mathcal{A}_t; s_t)=\alpha\, f_{\mathrm{util}}(\mathcal{A}_t; s_t)+\beta\, f_{\mathrm{div}}(\mathcal{A}_t).
$$
The utility term is a state-conditioned log-sum-exp over learned embeddings, and the diversity term penalizes redundancy through minimal pairwise dissimilarity. A greedy algorithm builds $\mathcal{A}_t$ with reported complexity $O(m^2|A|)$, after which Monte Carlo Tree Search estimates $Q(s_t,a)$ over the selected candidates. Using Open-Platypus, $k=2{,}500$ groups, and Llama-3.1-70B-Instruct for sketch extraction, the method produced 40,822 observations, deduplicated them into $A$, and then achieved 70.22 / 51.40 / 39.39 / 88.31 / 89.16 / 61.00 exact-match accuracies across MMLU, MMLU-Pro, GPQA, ARC-C, GSM8K, and MATH-500, outperforming rStar on all six benchmarks [2511.08043].

A theoretically different but operationally related construction is representative action selection for extremely large shared action spaces across a family of MDPs. The goal is to build a fixed subset $\mathcal{A}_r(s)\subseteq \mathcal{A}_{\mathrm{full}}(s)$ such that each environment retains a near-optimal action. The algorithm samples environments, computes $Q^*(s,a)$, and adds the optimal action for each sampled environment to the representative set. The performance guarantee is phrased through per-state regret,
$$
\regret(s)=\max_{a\in\mathcal{A}_{\mathrm{full}}(s)}Q^*(s,a)-\max_{b\in\mathcal{A}_r(s)}Q^*(s,b),
$$
with value loss bounded by
$$
V^*(s_0)-V^{\pi_r}(s_0)\le \frac{1}{1-\gamma}\max_s \regret(s).
$$
Under a non-centered sub-Gaussian increment model, expected cluster regret is controlled by Gaussian width plus a diameter-dependent logarithmic term, and the additional coverage error decays exponentially in the sample budget $K$ through factors of the form $(1-p_\ell)^{2K}$ [2511.22104].

Taken together, these methods show that expansion and reduction are not opposites in practice. A plausible implication is that many ExpA systems are best viewed as two-stage constructions: an offline or global enlargement that maximizes expressive coverage, followed by an online or task-specific restriction that maximizes tractability.

## 6. LLMs reasoning beyond language

The most explicit recent LLM formulation expands the model’s action space beyond vocabulary tokens. The setup defines a language environment $e=0$ with vocabulary $V$ and external environments $e\in\{1,\dots,K\}$ with environment-specific action sets $E_i$ and routing actions $g_i$. The global POMDP state is $s_t=(h_t,e_t,z_t)$, and the action space is
$$
A=A_{\mathrm{lang}}\cup A_{\mathrm{env}},\qquad
A_{\mathrm{lang}}=V,\qquad
A_{\mathrm{env}}=\bigcup_{i=1}^K (\{g_i\}\cup E_i).
$$
Action availability is masked by the active environment: in language, the model may emit vocabulary tokens or route to an external environment; inside external environment $i$, it may emit only actions in $E_i$. Environment interactions are mediated by interfaces of the form
$$
(o_t,z_{t+1},\mathrm{exit})=\mathrm{step}_i(h_t,z_t,a_t),
$$
with tokenized observations appended to history and $\mathrm{exit}$ controlling return to language [2510.07581].

Training uses ExpA Reinforcement Learning (EARL) with counterfactual policy optimization (CPO). The LLM retains its standard representation stack and extends only the output head to logits over the expanded action set; weights for new actions are initialized from pretrained token weights corresponding to natural-language descriptions $\mathrm{desc}(a)$. CPO pairs factual rollouts with counterfactual trajectories in which a routing action is forcibly inserted at a linguistically plausible point sampled in proportion to $\pi_\theta(\mathrm{desc}(g_i)\mid h_t,e_t=0)$. Updates use a PPO-style clipped objective with KL regularization against a reference model, with KL computed over the original vocabulary to avoid penalizing adoption of environment actions [2510.07581].

On calculator-based Calc-Bench, EARL outperformed vocabulary-constrained baselines. For 3B-instruct, overall exact match was 71.65 versus 59.75 for Prompt+GRPO and 60.92 for SFT+GRPO; for 7B-instruct, overall exact match was 78.69 versus 68.23 and 62.48. On the partially observed sorting task, EARL achieved perfect Sort-4 accuracy and, after pruning redundant comparisons, EARL* matched the optimal average number of swaps at 1.917 while using 4.833 average comparisons, close to the optimal 4.667 and below GCC built-in’s 5.000 [2510.07581].

Within LLM reasoning, DynaAct occupies a different point in the design space. DynaAct does not extend the model’s output alphabet beyond language; instead, it constructs a large reusable action pool from sketch extraction and compacts it dynamically into a state-tailored set for search. EARL, by contrast, internalizes routing and non-linguistic control actions as first-class outputs of the policy. A common misconception is therefore to equate ExpA with merely “having more actions.” The surveyed formulations indicate a broader pattern: ExpA may mean enlarging the ontology of alternatives, factorizing action semantics, introducing temporal scales, exposing tools as discrete outputs, or constructing compact representative subsets from an otherwise intractable superset [2511.08043] [2510.07581].

Source: https://www.emergentmind.com/topics/expanded-action-space-expa