Soft State-Action Abstractions

Updated 6 July 2026

Soft state-action abstractions are decision process models that relax precise state-action matching, enabling approximate grouping and adaptive planning.
p-ASAP and IPA frameworks apply action-pruning functions to focus on decision-relevant (often optimal) actions, thereby balancing search efficiency with accuracy.
Rate-distortion approaches use stochastic encoders/decoders and an information-theoretic objective to dynamically refine abstraction granularity while controlling value error.

Searching arXiv for recent and foundational papers on soft state-action abstractions, approximate abstractions, and related frameworks. arXiv search query: "soft state-action abstractions reinforcement learning approximate homomorphisms bisimulation rate-distortion" Soft state-action abstractions are abstractions of decision processes that relax exact correspondence between concrete states, actions, or state-action pairs. In recent arXiv work, the term covers at least two distinct technical constructions. One line weakens exact matching conditions inside UCT search trees by requiring agreement only on a selected subset of actions, yielding the p-ASAP and IPA frameworks and the IPA-UCT algorithm (Schmöcker et al., 30 Oct 2025). Another line defines abstractions through stochastic encoders and decoders over abstract states and abstract actions, and chooses their granularity by solving a rate-distortion problem coupled to a Bellman-error certificate (Rosas, 4 Jun 2026). A related but contrasting use of state-action abstraction appears in generalized task and motion planning (TAMP), where continuous geometric information enters through numeric features that are then projected into qualitative bins; that work explicitly does not employ soft or fuzzy memberships in the classical sense (Curtis et al., 2021).

1. Formal scope and definitions

In the UCT-based formulation, the underlying object is a finite MDP $M=(S,\mu_0,\A,\P,R,T)$ with finite legal action sets $\A(s)\subseteq A$. The classical ASAP state-action-pair abstraction induces a state equivalence by demanding that every action in one state has a matching abstract-equivalent action in the other, and conversely. The weakened p-ASAP condition introduces an action-pruning function $J\colon S\to\PP(A)$ with $J(s)\subseteq \A(s)$, and defines

$s_1\sim_J s_2 \iff \Bigl[ \forall\,a_1\in J(s_1)\,\exists\,a_2\in\A(s_2)\;:\;((s_1,a_1),(s_2,a_2))\in\H \;\wedge\; \forall\,a_2\in J(s_2)\,\exists\,a_1\in\A(s_1)\;:\;((s_1,a_1),(s_2,a_2))\in\H \Bigr].$

If $\sim_J$ is an equivalence, it is called a p-ASAP abstraction. ASAP is recovered by taking $J(s)=\A(s)$, while IPA uses the optimal-action set

$J^*(s)=\{\,a\mid Q^*(s,a)=\max_{a'}Q^*(s,a')\,\}.$

This makes “softness” a matter of relaxing which actions must be matched, rather than replacing deterministic abstraction classes by probabilistic memberships (Schmöcker et al., 30 Oct 2025).

In the rate-distortion formulation, a soft abstraction is specified by stochastic mappings on states and state-action pairs,

$\nu_S: S\to\Delta(\hat S),\qquad \nu_A: S\times A\to\Delta(\hat A),$

$\eta_S: \hat S\to\Delta(S),\qquad \eta_A: \hat S\times \hat A\to\Delta(A),$

with factorized encoder and decoder

$\A(s)\subseteq A$0

These mappings induce an abstract MDP $\A(s)\subseteq A$1, and abstract values are grounded back to concrete states by

$\A(s)\subseteq A$2

Here “softness” is literal stochasticity: a concrete state or state-action pair may map to multiple abstract codes with nonzero probability (Rosas, 4 Jun 2026).

The TAMP formulation provides a useful terminological contrast. There, abstract states are induced by Boolean features $\A(s)\subseteq A$3 and qualitative projections of numeric features $\A(s)\subseteq A$4, so that

$\A(s)\subseteq A$5

The paper states that there are no iterative “soft” membership updates and that each feature is either selected or not. Its relevance to the topic lies in the incorporation of continuous geometric information before projection to qualitative bins, not in probabilistic abstraction itself (Curtis et al., 2021).

2. Action-pruned softness in UCT search trees

The central motivation for p-ASAP and IPA is that exact ASAP state abstractions become rare when action spaces are noisy or large. The paper formalizes this by analyzing the probability that two states with $\A(s)\subseteq A$6 and $\A(s)\subseteq A$7 actions, randomly assigned to $\A(s)\subseteq A$8 abstract Q-nodes, satisfy the strict ASAP matching condition. Under uniform assignment, the probability of an ASAP abstraction is

$\A(s)\subseteq A$9

where $J\colon S\to\PP(A)$0 counts surjections of $J\colon S\to\PP(A)$1 items onto $J\colon S\to\PP(A)$2 bins. The upper bound falls exponentially in $J\colon S\to\PP(A)$3, which explains why exact state abstractions become scarce as the number of actions grows (Schmöcker et al., 30 Oct 2025).

The weakened condition replaces “all actions must match” by “actions in $J\colon S\to\PP(A)$4 must match.” In IPA, $J\colon S\to\PP(A)$5 contains only actions that are optimal under $J\colon S\to\PP(A)$6. This is a deliberately lossy condition: two states may be grouped even if they disagree on suboptimal actions, provided their optimal-action structure aligns. A plausible implication is that the abstraction is biased toward decision-relevant distinctions rather than full transition-graph identity.

This construction is embedded in a hierarchy. ASAP is p-ASAP with trivial pruning $J\colon S\to\PP(A)$7. IPA is p-ASAP with $J\colon S\to\PP(A)$8 equal to the optimal-action set. p-ASAP itself is presented as a special case of the broader ASASAP framework, described as “Alternating State And State-Action Pair Abstractions.” The hierarchy is explicitly

$J\colon S\to\PP(A)$9

while ASAP appears as another branch of p-ASAP with no pruning (Schmöcker et al., 30 Oct 2025).

3. IPA-UCT: algorithmic mechanism and asymptotic correctness

IPA-UCT instantiates p-ASAP inside Monte Carlo Tree Search by approximating $J(s)\subseteq \A(s)$0 on the fly from current search statistics. At a state $J(s)\subseteq \A(s)$1, it forms

$J(s)\subseteq \A(s)$2

The same exploration constant $J(s)\subseteq \A(s)$3 used in the main tree policy is used for pruning. The limiting regimes are explicit: $J(s)\subseteq \A(s)$4 yields aggressive pruning by keeping only current best actions, whereas $J(s)\subseteq \A(s)$5 yields no pruning and reduces to ASAP (Schmöcker et al., 30 Oct 2025).

The implementation maintains two recency counters $J(s)\subseteq \A(s)$6, one per Q-node and one per state node. When a node’s counter reaches $J(s)\subseteq \A(s)$7, its abstraction is recomputed. The procedure compute_state_abstraction(s) searches existing abstract state classes at the same depth, selects the largest class compatible with the p-ASAP condition, and otherwise creates a new singleton. During backup, update_state_abstraction is invoked along the path, and when a state’s abstraction changes the update is propagated upward. Statistics are shared across the current abstract group: each tree node carries aggregated $J(s)\subseteq \A(s)$8 over its group, for both Q-nodes and states, and UCB is computed from those group totals (Schmöcker et al., 30 Oct 2025).

The main theoretical claim is asymptotic soundness despite early lossy merges. The paper states that IPA-UCT’s abstraction on the search tree becomes sound almost surely in the limit of infinite iterations, meaning that only true $J(s)\subseteq \A(s)$9-equivalent states and $s_1\sim_J s_2 \iff \Bigl[ \forall\,a_1\in J(s_1)\,\exists\,a_2\in\A(s_2)\;:\;((s_1,a_1),(s_2,a_2))\in\H \;\wedge\; \forall\,a_2\in J(s_2)\,\exists\,a_1\in\A(s_1)\;:\;((s_1,a_1),(s_2,a_2))\in\H \Bigr].$0-equivalent state-action pairs remain. The proof sketch relies on two components. First, UCB exploration ensures that every edge is sampled infinitely often, so Q-statistics converge to $s_1\sim_J s_2 \iff \Bigl[ \forall\,a_1\in J(s_1)\,\exists\,a_2\in\A(s_2)\;:\;((s_1,a_1),(s_2,a_2))\in\H \;\wedge\; \forall\,a_2\in J(s_2)\,\exists\,a_1\in\A(s_1)\;:\;((s_1,a_1),(s_2,a_2))\in\H \Bigr].$1. Second, an induction over tree depth shows that once deeper-layer abstractions are sound, UCB-based pruning almost surely removes only suboptimal actions at the current layer, leaving $s_1\sim_J s_2 \iff \Bigl[ \forall\,a_1\in J(s_1)\,\exists\,a_2\in\A(s_2)\;:\;((s_1,a_1),(s_2,a_2))\in\H \;\wedge\; \forall\,a_2\in J(s_2)\,\exists\,a_1\in\A(s_1)\;:\;((s_1,a_1),(s_2,a_2))\in\H \Bigr].$2 intact and turning $s_1\sim_J s_2 \iff \Bigl[ \forall\,a_1\in J(s_1)\,\exists\,a_2\in\A(s_2)\;:\;((s_1,a_1),(s_2,a_2))\in\H \;\wedge\; \forall\,a_2\in J(s_2)\,\exists\,a_1\in\A(s_1)\;:\;((s_1,a_1),(s_2,a_2))\in\H \Bigr].$3 into an exact $s_1\sim_J s_2 \iff \Bigl[ \forall\,a_1\in J(s_1)\,\exists\,a_2\in\A(s_2)\;:\;((s_1,a_1),(s_2,a_2))\in\H \;\wedge\; \forall\,a_2\in J(s_2)\,\exists\,a_1\in\A(s_1)\;:\;((s_1,a_1),(s_2,a_2))\in\H \Bigr].$4-equivalence (Schmöcker et al., 30 Oct 2025).

A recurrent misconception is that action-pruned abstractions necessarily abandon correctness. In this formulation, the approximation is transient rather than terminal: inaccuracies may occur early, but the limiting abstraction is claimed to recover exact value equivalence.

4. Rate-distortion abstractions and adaptive refinement

The rate-distortion approach starts from a finite discounted MDP $s_1\sim_J s_2 \iff \Bigl[ \forall\,a_1\in J(s_1)\,\exists\,a_2\in\A(s_2)\;:\;((s_1,a_1),(s_2,a_2))\in\H \;\wedge\; \forall\,a_2\in J(s_2)\,\exists\,a_1\in\A(s_1)\;:\;((s_1,a_1),(s_2,a_2))\in\H \Bigr].$5 and induces an abstract MDP through stochastic encoders and decoders. The induced reward and transition model are

The abstract Bellman operator is

This makes abstraction resolution a continuous design variable rather than a discrete choice among hard partitions (Rosas, 4 Jun 2026).

Abstraction quality is controlled by a Bellman-compatible distortion $s_1\sim_J s_2 \iff \Bigl[ \forall\,a_1\in J(s_1)\,\exists\,a_2\in\A(s_2)\;:\;((s_1,a_1),(s_2,a_2))\in\H \;\wedge\; \forall\,a_2\in J(s_2)\,\exists\,a_1\in\A(s_1)\;:\;((s_1,a_1),(s_2,a_2))\in\H \Bigr].$9, with canonical choice

$\sim_J$ 0

and by the expected round-trip distortion

$\sim_J$ 1

The information rates are

$\sim_J$ 2

and the Lagrangian objective is

$\sim_J$ 3

Here $\sim_J$ 4 trades off compression against distortion, while $\sim_J$ 5 sets the relative cost of action versus state bits (Rosas, 4 Jun 2026).

The framework’s central theoretical object is a performance certificate that decomposes value error into a learning term and an abstraction term. With $\sim_J$ 6 the concrete Bellman optimality operator, $\sim_J$ 7 its fixed point, and $\sim_J$ 8, the main bound is

$\sim_J$ 9

The first term is the Bellman residual; the second is an abstraction error expressed through a Hausdorff mismatch $J(s)=\A(s)$0 and a state round-trip kernel $J(s)=\A(s)$1. This gives a precise criterion for when an abstraction has become too coarse relative to the current level of planning error (Rosas, 4 Jun 2026).

The adaptive rule follows directly: begin with a coarse abstraction, continue planning while the Bellman residual dominates the abstraction error, and refine once

$J(s)=\A(s)$2

In practice, the paper monitors the empirical residual against $J(s)=\A(s)$3 as a proxy and increases $J(s)=\A(s)$4 along a precomputed schedule. The solver is described as a structured Blahut-Arimoto procedure that alternates between recomputing abstract marginals, updating $J(s)=\A(s)$5, updating $J(s)=\A(s)$6, selecting blockwise medoid decoders $J(s)=\A(s)$7, and repeating until convergence in the rate-distortion objective. A simpler flat update is also given: $J(s)=\A(s)$8 This suggests a notion of soft abstraction in which refinement is triggered not by a fixed schedule alone but by an explicit balance between optimization error and representation error (Rosas, 4 Jun 2026).

5. Empirical regimes and benchmark behavior

The empirical profile of IPA-UCT is tied to search efficiency in tree-based planning. The evaluation covers more than 20 single- and two-player MDPs drawn from IPPC, navigation, strategy games, and board games, with horizons 50–100 and iteration budgets of 100, 200, 500, and 1000. Two metrics are reported: “Pairings score,” which counts the fraction of tasks where one agent outranks another, and “Relative improvement,” which averages percentage gain over competitors. Against the best OGA variants, including pruned OGA and $J(s)=\A(s)$9-OGA, IPA-UCT shows an approximately $J^*(s)=\{\,a\mid Q^*(s,a)=\max_{a'}Q^*(s,a')\,\}.$ 0 average relative gain at 500 iterations and outperforms in 3 of 4 budget settings. Against random-state abstractions, it still leads by large margins. The same study reports that OGA builds virtually no non-trivial state abstractions in most domains, whereas IPA-UCT with $J^*(s)=\{\,a\mid Q^*(s,a)=\max_{a'}Q^*(s,a')\,\}.$ 1 yields up to $J^*(s)=\{\,a\mid Q^*(s,a)=\max_{a'}Q^*(s,a')\,\}.$ 2– $J^*(s)=\{\,a\mid Q^*(s,a)=\max_{a'}Q^*(s,a')\,\}.$ 3 fewer singleton state nodes. Runtime overhead is reported as median $J^*(s)=\{\,a\mid Q^*(s,a)=\max_{a'}Q^*(s,a')\,\}.$ 4 at 100 iterations and $J^*(s)=\{\,a\mid Q^*(s,a)=\max_{a'}Q^*(s,a')\,\}.$ 5 at 2000 iterations, with decision times per call remaining in low milliseconds (Schmöcker et al., 30 Oct 2025).

The rate-distortion study evaluates FourRooms, Taxi, DoorKey, and a ring-structured SysAdmin with $J^*(s)=\{\,a\mid Q^*(s,a)=\max_{a'}Q^*(s,a')\,\}.$ 6 machines. It reports a smooth compression-distortion frontier: as $J^*(s)=\{\,a\mid Q^*(s,a)=\max_{a'}Q^*(s,a')\,\}.$ 7 grows, average distortion falls while the effective number of abstract $J^*(s)=\{\,a\mid Q^*(s,a)=\max_{a'}Q^*(s,a')\,\}.$ 8 pairs grows. The adaptive rule is said to recover near-optimal performance, quantified as at least $J^*(s)=\{\,a\mid Q^*(s,a)=\max_{a'}Q^*(s,a')\,\}.$ 9 of $\nu_S: S\to\Delta(\hat S),\qquad \nu_A: S\times A\to\Delta(\hat A),$ 0, often under substantial lossy compression (Rosas, 4 Jun 2026).

Domain	Adaptive $\nu_S: S\to\Delta(\hat S),\qquad \nu_A: S\times A\to\Delta(\hat A),$ 1	Decomposition
FourRooms	0.318	$\nu_S: S\to\Delta(\hat S),\qquad \nu_A: S\times A\to\Delta(\hat A),$ 2
Taxi	0.159	$\nu_S: S\to\Delta(\hat S),\qquad \nu_A: S\times A\to\Delta(\hat A),$ 3
DoorKey	0.133	$\nu_S: S\to\Delta(\hat S),\qquad \nu_A: S\times A\to\Delta(\hat A),$ 4

The same paper distinguishes where compression occurs. In FourRooms, most compression comes from states; in Taxi, from actions; in DoorKey, from both. In SysAdmin, as $\nu_S: S\to\Delta(\hat S),\qquad \nu_A: S\times A\to\Delta(\hat A),$ 5 grows, the selected abstraction maintains constant normalized distortion while retaining a vanishing fraction of state-action information, which the paper interprets as capturing local ring structure (Rosas, 4 Jun 2026).

The TAMP study offers a third empirical regime. It learns feature sets and abstract actions from a handful of example plans, then uses a FOND planner to obtain an abstract policy $\nu_S: S\to\Delta(\hat S),\qquad \nu_A: S\times A\to\Delta(\hat A),$ 6, which is embedded back into the continuous TAMP domain. On Unstack, Sort, Clutter, and Transport, the constrained-search approach is reported as $\nu_S: S\to\Delta(\hat S),\qquad \nu_A: S\times A\to\Delta(\hat A),$ 7– $\nu_S: S\to\Delta(\hat S),\qquad \nu_A: S\times A\to\Delta(\hat A),$ 8 faster than an unguided TAMP solver as the number of objects grows, and it scales to dozens of objects where standard TAMP times out. Although this is not a soft abstraction in the probabilistic sense, it shows how geometry-informed state and action abstractions can materially change search behavior (Curtis et al., 2021).

6. Relation to hard abstractions, terminology, and extensions

A persistent terminological issue is that “soft” does not denote a single formalism. In the rate-distortion framework, softness means stochastic encoding and decoding, continuous control over abstraction capacity, and a lossy compression objective. In IPA-UCT, softness means approximate matching induced by pruning attention to a decision-relevant subset of actions. In the TAMP setting, the paper explicitly states that there is no fuzzy membership update; the only soft aspect is the use of continuous geometric information before qualitative projection (Rosas, 4 Jun 2026, Schmöcker et al., 30 Oct 2025, Curtis et al., 2021).

Another common misunderstanding is that soft abstractions are merely heuristic compression schemes without recovery paths to exactness. The cited work draws a more differentiated picture. In IPA-UCT, increasing $\nu_S: S\to\Delta(\hat S),\qquad \nu_A: S\times A\to\Delta(\hat A),$ 9 suppresses pruning and recovers ASAP in the limit $\eta_S: \hat S\to\Delta(S),\qquad \eta_A: \hat S\times \hat A\to\Delta(A),$ 0; in the opposite direction, $\eta_S: \hat S\to\Delta(S),\qquad \eta_A: \hat S\times \hat A\to\Delta(A),$ 1 keeps only current best actions (Schmöcker et al., 30 Oct 2025). In the rate-distortion framework, taking $\eta_S: \hat S\to\Delta(S),\qquad \eta_A: \hat S\times \hat A\to\Delta(A),$ 2 or pruning small-mass codes yields hard abstractions with deterministic blocks of the desired capacity (Rosas, 4 Jun 2026). These constructions suggest that softness can function as a controlled relaxation rather than as an unrelated alternative to exact abstraction.

The proposed extensions point toward broader integration with planning and RL systems. For IPA-UCT, the paper suggests applying the same principle to RAVE, progressive widening, and PUCT in AlphaZero, guiding $\eta_S: \hat S\to\Delta(S),\qquad \eta_A: \hat S\times \hat A\to\Delta(A),$ 3 by learned value or policy networks, and combining abstraction with intra-abstraction policies (Schmöcker et al., 30 Oct 2025). For rate-distortion abstractions, suggested deep-RL extensions replace Blahut-Arimoto by variational encoders and decoders, estimate distortion through learned transition or representation models, and use sample-based TD errors together with learned bisimulation proxies such as MICo or DeepMDP losses (Rosas, 4 Jun 2026). This suggests that the current literature treats soft state-action abstraction less as a single algorithmic family than as a unifying principle for trading representational compression against control fidelity.