---
title: Afterstate-Evaluating Actor Network
url: https://www.emergentmind.com/topics/afterstate-evaluating-actor-network
type: topic
---

# Afterstate-Evaluating Actor Network

Searching arXiv for the cited paper and closely related context.
In the Tetris reinforcement-learning framework introduced in "Bitboard version of Tetris AI" [2603.26765], the **Afterstate-Evaluating Actor Network** is a policy parameterization that assigns probabilities to legal placements by evaluating the **afterstates** induced by those placements rather than directly evaluating action indices or generic state-action pairs. In this usage, an afterstate is the board configuration **after the current piece has been placed and line clears have been applied, but before the next random tetromino is generated**. The method exploits a structural property of Tetris: the geometric consequence of a placement is deterministic, whereas stochasticity enters only through the next-piece draw. The resulting actor is therefore organized around deterministic successor boards, represented through **9 DT features** per candidate afterstate, combined with masking and a softmax policy over up to **34** legal actions [2603.26765].

## 1. Definition of afterstate and induced Markov structure

The paper defines three distinct objects: the current state \(s\), the action \(a\), and the afterstate \(as\). The current state includes the board configuration, the current falling piece, and the random next-piece information available in the state representation. An action is a legal placement choice for the current tetromino, represented as a combination of **rotation** and **horizontal position**. The afterstate, denoted either in prose as “afterstate” or symbolically through \(as\) and \(f(s,a)\), is the deterministic board obtained once action \(a\) is executed in state \(s\), including any line clearing, but **before** the environment samples the next tetromino [2603.26765].

This induces a two-stage transition structure. First, \((s,a)\) deterministically produces \(as\). Second, the environment samples the next piece, converting \(as\) into the next full state \(s'\). The paper states this directly: “Given a state \(s\) and an action \(a\), the state where the next random piece has not yet been generated after executing action \(a\) is called an afterstate,” and “The afterstate combined with the next randomly generated piece constitutes the next state \(s'\).” In the paper’s notation, the value of an afterstate is expressed as an expectation over subsequent full states,
\[
V(as) = \sum_{s' \in \mathcal{S}} V(s') P(s'|as),
\]
with the qualification that the printed equation is typeset imperfectly but its intended semantics are clear [2603.26765].

The technical significance of this decomposition is that it separates the agent’s direct control over board topology from the environment’s stochastic next-piece process. This suggests that the policy-learning problem can be reformulated around the quality of resulting board configurations rather than the quality of abstract action labels.

## 2. Policy parameterization by afterstate evaluation

The core departure from conventional actor parameterization is that the actor does **not** map a state directly to logits over a fixed action list. Instead, for a given state \(s\), the environment enumerates all feasible placements for the current tetromino, computes the afterstate associated with each legal action, extracts features for each afterstate, and then applies a shared evaluator to those candidate afterstates. The policy is defined as
\[
\pi(a|s) = \frac{\exp(V(f(s, a)))}{\sum_b \exp(V(f(s, b)))}.
\]
Here, \(f(s,a)\) denotes the feature representation of the afterstate produced by action \(a\), and \(V(\cdot)\) is a scalar evaluator applied to that representation [2603.26765].

The paper contrasts this with a conventional Boltzmann policy derived from action-values,
\[
\pi(a|s) = \frac{\exp(Q(s,a))}{\sum_b \exp(Q(s,b))},
\]
and with ordinary PPO parameterizations of \(\pi_\theta(a|s)\) from state input alone. The distinction is not merely cosmetic. In the proposed construction, policy logits are **derived from afterstate evaluations**, the action set is generated dynamically from legal placements, and invalid actions are suppressed by **masking** rather than by a permanent fixed-output head [2603.26765].

The paper characterizes this shift as moving the learning target from “evaluating the quality of action-block combinations” to “evaluating the quality of board configurations.” A plausible implication is that the network is asked to model an object whose geometry is more directly coupled to long-term return in Tetris than the raw symbolic action representation.

## 3. Input representation, masking, and network form

For each candidate afterstate, the actor uses **9 DT features**. The paper names them explicitly: **Landing height**, **Eroded piece cells**, **Row transitions**, **Column transitions**, **Holes**, **Board well**, **Hole depth**, **Rows with holes**, and **Pattern diversity** [2603.26765]. Because up to **34 actions** are possible for a piece, the actor input is formed by concatenating all afterstate feature vectors:
\[
34 \times 9 = 306.
\]
The resulting actor input is therefore a **306-dimensional vector** denoted \(f(s,a)_{\text{all}}\) in the paper’s discussion.

Since not every tetromino has 34 legal placements, the actor also receives a **mask** with value 1 for feasible actions and 0 for infeasible actions. The mask is used to force illegal actions to zero probability by assigning them negative infinity before the softmax. The paper’s Figure 4 description states that “Through reshape operations, a shared linear layer evaluates all afterstates, masks out infeasible actions, and applies softmax to yield action probabilities” [2603.26765].

The architecture details that are actually specified are narrow and important. The paper provides the input dimension (**306**), the per-afterstate feature dimension (**9**), support for batch size \(n \ge 1\), a **shared linear layer** over afterstates, masking, and softmax. It does **not** specify hidden-layer widths, nonlinear activations, normalization, embeddings, residual modules, or convolutional or transformer components. In the REINFORCE ablation, the authors explicitly state that “only linear layers with dimensionality identical to the input vector are utilized” to avoid architecture bias. The afterstate actor is therefore best described, within the paper’s own level of detail, as a **shared linear afterstate evaluator plus masked softmax** [2603.26765].

A common misconception would be to interpret the method as a generic deep actor over raw board tensors. That is not the implementation described here. The actor consumes hand-crafted DT descriptors of candidate afterstates rather than raw bitmaps.

## 4. Legal-action enumeration and afterstate generation

The network depends on explicit enumeration of legal placements. Each action is represented as a combination of **rotation state** and **horizontal translation**. The number of legal actions is piece-dependent because the number of unique rotations and admissible translations depends on tetromino geometry. The paper gives the following examples:

| Piece type | Rotations | Total actions |
|---|---:|---:|
| O-piece | 1 | up to 9 |
| I-piece | 2 | 17 |
| S/Z | 2 | 17 |
| L/J/T | 4 | 34 |

The implementation stores metadata such as `rotationNum`, `actionSize`, and `maxHeight` in the `Sq` object [2603.26765].

Move generation proceeds by reading the current tetromino type, enumerating its rotations, enumerating all horizontal placements permitted by piece width and board width, computing the landing row using bitboard collision logic, applying the placement to produce a new board, clearing completed lines, storing metadata such as reward, drop height, and delete-line mask, and finally computing DT features for the resulting afterstate. The environment method `Tetris.get_9feature()` returns “the DT features and mask for up to 34 possible successor states based on the given state,” which is exactly the interface consumed by the actor [2603.26765].

Collision detection is performed through bitwise AND operations between shifted piece columns and board columns. Landing is found by starting near the relevant column heights and descending until overlap. Rotation handling is simplified by explicitly predefining every rotated shape, along with its width and height, in bitboard form. This means that the full policy pipeline is tightly integrated with efficient afterstate construction: the environment enumerates legal actions and afterstates, computes DT features, builds a 34-slot feature structure plus legal-action mask, the actor scores all slots with shared weights, the mask removes invalid actions, and softmax produces \(\pi(a|s)\) [2603.26765].

This dependence on exact move enumeration is central. The approach is tractable here because the action set is small, finite, and piece-structured.

## 5. Training within PPO and comparison with action-value actors

The final system trains the actor with **PPO**, specifically the paper’s **buffer-based PPO** variant. **REINFORCE** is used only for the ablation comparing afterstate and action-value actors, in order to isolate the actor and avoid critic effects [2603.26765]. The paper gives the REINFORCE return
\[
G_t = \sum_{t'=t}^{T} \gamma^{t'-t} r_{t'},
\]
with policy update
\[
\theta \gets \theta + \alpha \sum_{t=0}^{T} G_t \nabla_{\theta} \log \pi_{\theta}(a_t|s_t).
\]
For PPO, it presents the TD error and GAE expressions
\[
\delta_t = r + \gamma \cdot \text{Critic}(f(s_t, a_t)) - \text{Critic}(f(s_t)),
\]
\[
A_t = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l},
\]
and the clipped surrogate objective
\[
L_a = \min\left[ \frac{\pi_{\theta}(a|s)}{\pi_{\theta_k}(a|s)} A_t,\ 
\text{clip}\left( \frac{\pi_{\theta}(a|s)}{\pi_{\theta_k}(a|s)}, 1-\epsilon, 1+\epsilon \right) A_t \right].
\]
The paper notes that some notation is inconsistent, but these equations capture the intended training logic [2603.26765].

The buffer-based PPO algorithm collects transitions continuously during play, stores them in a buffer, and performs PPO updates whenever `Buffer.size == batchSize`, after which the buffer is reset. The paper presents the workflow as initializing Actor, Critic, and Replay Buffer; rolling out gameplay; computing \(f(s,a)_{\text{all}}\) and mask for each state; selecting actions with the actor; executing `env.step(a)`; storing transitions; and then, once the buffer threshold is reached, repeating `epoch` training epochs with `miniBatchSize` samples to compute TD error, GAE, and actor and critic updates [2603.26765].

The hyperparameters reported for buffer-based PPO are \(\gamma = 0.99\), \(\lambda = 0.99\), \(\epsilon = 0.2\), \(\alpha_\theta = 3\times 10^{-4}\), \(\alpha_\omega = 3\times 10^{-4}\), `batchSize = 2048`, `miniBatchSize = 256`, `epoch = 10`, and `totalSteps = 61,440`. The trajectory-based PPO baseline uses \(\gamma = 0.99\), \(\lambda = 0.99\), \(\tau_0 = 0.5\), \(\tau_k = 0.00025\), \(\alpha_\theta = 3\times 10^{-4}\), \(\alpha_\omega = 3\times 10^{-4}\), `epoch = 10`, \(\epsilon = 0.1\), and `episode = 12,500`. The reward is described repeatedly as the number of lines cleared by a placement [2603.26765].

An important clarification is that the “buffer” is **not** presented as off-policy replay memory. The text does not explicitly mention entropy bonus, value-loss coefficient tuning, observation normalization, reward normalization, or replay in the off-policy sense. The buffer is a batching mechanism for PPO updates rather than a separate replay-learning regime.

## 6. Empirical findings, learned coefficients, and limitations

The paper’s most direct evidence for the Afterstate-Evaluating Actor Network is the ablation against an **action-value Actor** with **48-dimensional input**. The afterstate actor uses **9-dimensional input** in that REINFORCE comparison, with only linear layers used on both sides to avoid architectural confounds. The reported conclusion is explicit: “The experimental results indicate that the Actor evaluating afterstates utilizes fewer weights and achieves significantly better performance than the Actor evaluating the action-value function” [2603.26765]. The text does not provide exact numeric entries for that ablation beyond what appears graphically in Figure 11, so no more precise quantitative claim is available.

For PPO using the afterstate actor, the paper reports a trajectory-based PPO average of **3840.30** over the last 150 episodes across 5 runs, a buffer-based PPO average of **3829.04** over the last 20 test episodes across 5 runs, and a **10,000-game test average** of **4124.47** for the best trained model [2603.26765]. It also reports a large difference in training interactions and elapsed time for the overall system: buffer PPO uses **61,440 total steps**, compared with **69,046,726 total steps** for trajectory PPO, while one run yields **166 s** total training time for buffer PPO versus **10,972 s** for trajectory PPO. These gains are attributed to the whole framework rather than to the actor alone, but the afterstate formulation is presented as one of its central lightweight design choices.

The actor remains tied to **DT features** rather than end-to-end raw observation learning. The paper lists the learned coefficients for one best-performing PPO result as follows: Landing height \(-0.51\), Eroded piece cells \(0.16\), Row transitions \(-0.40\), Column transitions \(-0.75\), Holes \(-0.18\), Board well \(-0.39\), Hole depth \(-0.17\), Rows with holes \(-0.83\), and Diversity \(0.36\) [2603.26765]. This supports the interpretation that the actor is effectively a learned linear scorer over hand-crafted afterstate descriptors.

The method’s limitations are also explicit. It depends on a domain in which the chosen action deterministically creates a post-decision board and randomness enters afterward; on the feasibility of enumerating all candidate actions and computing all afterstates; and on the availability of informative hand-crafted features. The paper further notes that transfer from **10x10** training to **10x20** is only partial because the standard board creates a larger state space, longer horizons, more difficult long-term value estimation, and different layout strategy demands. It also reports that all methods degrade severely under adversarial Z/S sequences [2603.26765]. These observations indicate that the afterstate actor is not presented as a universal architecture; rather, it is a Tetris-specialized actor that exploits a particular decomposition between deterministic placement effects and stochastic next-piece generation.

Source: https://www.emergentmind.com/topics/afterstate-evaluating-actor-network