Afterstate-Evaluating Actor Network

• The paper’s main contribution is the afterstate evaluation method that scores board configurations to separate deterministic placement outcomes from subsequent stochastic events.
• It uses 9 DT features per candidate afterstate and a shared linear layer with masking to efficiently process up to 34 legal actions.
• Empirical results show the afterstate actor outperforms traditional action-value actors, achieving significant efficiency and performance gains with buffer-based PPO training.

Searching arXiv for the cited paper and closely related context. In the Tetris reinforcement-learning framework introduced in "Bitboard version of Tetris AI" (Chen et al., 24 Mar 2026), the Afterstate-Evaluating Actor Network is a policy parameterization that assigns probabilities to legal placements by evaluating the afterstates induced by those placements rather than directly evaluating action indices or generic state-action pairs. In this usage, an afterstate is the board configuration after the current piece has been placed and line clears have been applied, but before the next random tetromino is generated. The method exploits a structural property of Tetris: the geometric consequence of a placement is deterministic, whereas stochasticity enters only through the next-piece draw. The resulting actor is therefore organized around deterministic successor boards, represented through 9 DT features per candidate afterstate, combined with masking and a softmax policy over up to 34 legal actions (Chen et al., 24 Mar 2026).

1. Definition of afterstate and induced Markov structure

The paper defines three distinct objects: the current state $s$, the action $a$, and the afterstate $as$. The current state includes the board configuration, the current falling piece, and the random next-piece information available in the state representation. An action is a legal placement choice for the current tetromino, represented as a combination of rotation and horizontal position. The afterstate, denoted either in prose as “afterstate” or symbolically through $as$ and $f(s,a)$, is the deterministic board obtained once action $a$ is executed in state $s$, including any line clearing, but before the environment samples the next tetromino (Chen et al., 24 Mar 2026).

This induces a two-stage transition structure. First, $(s,a)$ deterministically produces $as$. Second, the environment samples the next piece, converting $as$ into the next full state $a$0. The paper states this directly: “Given a state $a$1 and an action $a$2, the state where the next random piece has not yet been generated after executing action $a$3 is called an afterstate,” and “The afterstate combined with the next randomly generated piece constitutes the next state $a$4.” In the paper’s notation, the value of an afterstate is expressed as an expectation over subsequent full states,

$a$5

with the qualification that the printed equation is typeset imperfectly but its intended semantics are clear (Chen et al., 24 Mar 2026).

The technical significance of this decomposition is that it separates the agent’s direct control over board topology from the environment’s stochastic next-piece process. This suggests that the policy-learning problem can be reformulated around the quality of resulting board configurations rather than the quality of abstract action labels.

2. Policy parameterization by afterstate evaluation

The core departure from conventional actor parameterization is that the actor does not map a state directly to logits over a fixed action list. Instead, for a given state $a$6, the environment enumerates all feasible placements for the current tetromino, computes the afterstate associated with each legal action, extracts features for each afterstate, and then applies a shared evaluator to those candidate afterstates. The policy is defined as

$a$7

Here, $a$8 denotes the feature representation of the afterstate produced by action $a$9, and $as$0 is a scalar evaluator applied to that representation (Chen et al., 24 Mar 2026).

The paper contrasts this with a conventional Boltzmann policy derived from action-values,

$as$1

and with ordinary PPO parameterizations of $as$2 from state input alone. The distinction is not merely cosmetic. In the proposed construction, policy logits are derived from afterstate evaluations, the action set is generated dynamically from legal placements, and invalid actions are suppressed by masking rather than by a permanent fixed-output head (Chen et al., 24 Mar 2026).

The paper characterizes this shift as moving the learning target from “evaluating the quality of action-block combinations” to “evaluating the quality of board configurations.” A plausible implication is that the network is asked to model an object whose geometry is more directly coupled to long-term return in Tetris than the raw symbolic action representation.

3. Input representation, masking, and network form

For each candidate afterstate, the actor uses 9 DT features. The paper names them explicitly: Landing height, Eroded piece cells, Row transitions, Column transitions, Holes, Board well, Hole depth, Rows with holes, and Pattern diversity (Chen et al., 24 Mar 2026). Because up to 34 actions are possible for a piece, the actor input is formed by concatenating all afterstate feature vectors: $as$3 The resulting actor input is therefore a 306-dimensional vector denoted $as$4 in the paper’s discussion.

Since not every tetromino has 34 legal placements, the actor also receives a mask with value 1 for feasible actions and 0 for infeasible actions. The mask is used to force illegal actions to zero probability by assigning them negative infinity before the softmax. The paper’s Figure 1 description states that “Through reshape operations, a shared linear layer evaluates all afterstates, masks out infeasible actions, and applies softmax to yield action probabilities” (Chen et al., 24 Mar 2026).

The architecture details that are actually specified are narrow and important. The paper provides the input dimension (306), the per-afterstate feature dimension (9), support for batch size $as$5, a shared linear layer over afterstates, masking, and softmax. It does not specify hidden-layer widths, nonlinear activations, normalization, embeddings, residual modules, or convolutional or transformer components. In the REINFORCE ablation, the authors explicitly state that “only linear layers with dimensionality identical to the input vector are utilized” to avoid architecture bias. The afterstate actor is therefore best described, within the paper’s own level of detail, as a shared linear afterstate evaluator plus masked softmax (Chen et al., 24 Mar 2026).

A common misconception would be to interpret the method as a generic deep actor over raw board tensors. That is not the implementation described here. The actor consumes hand-crafted DT descriptors of candidate afterstates rather than raw bitmaps.

4. Legal-action enumeration and afterstate generation

The network depends on explicit enumeration of legal placements. Each action is represented as a combination of rotation state and horizontal translation. The number of legal actions is piece-dependent because the number of unique rotations and admissible translations depends on tetromino geometry. The paper gives the following examples:

Piece type	Rotations	Total actions
O-piece	1	up to 9
I-piece	2	17
S/Z	2	17
L/J/T	4	34

The implementation stores metadata such as rotationNum, actionSize, and maxHeight in the Sq object (Chen et al., 24 Mar 2026).

Move generation proceeds by reading the current tetromino type, enumerating its rotations, enumerating all horizontal placements permitted by piece width and board width, computing the landing row using bitboard collision logic, applying the placement to produce a new board, clearing completed lines, storing metadata such as reward, drop height, and delete-line mask, and finally computing DT features for the resulting afterstate. The environment method Tetris.get_9feature() returns “the DT features and mask for up to 34 possible successor states based on the given state,” which is exactly the interface consumed by the actor (Chen et al., 24 Mar 2026).

Collision detection is performed through bitwise AND operations between shifted piece columns and board columns. Landing is found by starting near the relevant column heights and descending until overlap. Rotation handling is simplified by explicitly predefining every rotated shape, along with its width and height, in bitboard form. This means that the full policy pipeline is tightly integrated with efficient afterstate construction: the environment enumerates legal actions and afterstates, computes DT features, builds a 34-slot feature structure plus legal-action mask, the actor scores all slots with shared weights, the mask removes invalid actions, and softmax produces $as$6 (Chen et al., 24 Mar 2026).

This dependence on exact move enumeration is central. The approach is tractable here because the action set is small, finite, and piece-structured.

5. Training within PPO and comparison with action-value actors

The final system trains the actor with PPO, specifically the paper’s buffer-based PPO variant. REINFORCE is used only for the ablation comparing afterstate and action-value actors, in order to isolate the actor and avoid critic effects (Chen et al., 24 Mar 2026). The paper gives the REINFORCE return

$as$7

with policy update

$as$8

For PPO, it presents the TD error and GAE expressions

$as$9

$as$0

and the clipped surrogate objective

$as$1

The paper notes that some notation is inconsistent, but these equations capture the intended training logic (Chen et al., 24 Mar 2026).

The buffer-based PPO algorithm collects transitions continuously during play, stores them in a buffer, and performs PPO updates whenever Buffer.size == batchSize, after which the buffer is reset. The paper presents the workflow as initializing Actor, Critic, and Replay Buffer; rolling out gameplay; computing $as$2 and mask for each state; selecting actions with the actor; executing env.step(a); storing transitions; and then, once the buffer threshold is reached, repeating epoch training epochs with miniBatchSize samples to compute TD error, GAE, and actor and critic updates (Chen et al., 24 Mar 2026).

The hyperparameters reported for buffer-based PPO are $as$3, $as$4, $as$5, $as$6, $as$7, batchSize = 2048, miniBatchSize = 256, epoch = 10, and totalSteps = 61,440. The trajectory-based PPO baseline uses $as$8, $as$9, $f(s,a)$0, $f(s,a)$1, $f(s,a)$2, $f(s,a)$3, epoch = 10, $f(s,a)$4, and episode = 12,500. The reward is described repeatedly as the number of lines cleared by a placement (Chen et al., 24 Mar 2026).

An important clarification is that the “buffer” is not presented as off-policy replay memory. The text does not explicitly mention entropy bonus, value-loss coefficient tuning, observation normalization, reward normalization, or replay in the off-policy sense. The buffer is a batching mechanism for PPO updates rather than a separate replay-learning regime.

6. Empirical findings, learned coefficients, and limitations

The paper’s most direct evidence for the Afterstate-Evaluating Actor Network is the ablation against an action-value Actor with 48-dimensional input. The afterstate actor uses 9-dimensional input in that REINFORCE comparison, with only linear layers used on both sides to avoid architectural confounds. The reported conclusion is explicit: “The experimental results indicate that the Actor evaluating afterstates utilizes fewer weights and achieves significantly better performance than the Actor evaluating the action-value function” (Chen et al., 24 Mar 2026). The text does not provide exact numeric entries for that ablation beyond what appears graphically in Figure 2, so no more precise quantitative claim is available.

For PPO using the afterstate actor, the paper reports a trajectory-based PPO average of 3840.30 over the last 150 episodes across 5 runs, a buffer-based PPO average of 3829.04 over the last 20 test episodes across 5 runs, and a 10,000-game test average of 4124.47 for the best trained model (Chen et al., 24 Mar 2026). It also reports a large difference in training interactions and elapsed time for the overall system: buffer PPO uses 61,440 total steps, compared with 69,046,726 total steps for trajectory PPO, while one run yields 166 s total training time for buffer PPO versus 10,972 s for trajectory PPO. These gains are attributed to the whole framework rather than to the actor alone, but the afterstate formulation is presented as one of its central lightweight design choices.

The actor remains tied to DT features rather than end-to-end raw observation learning. The paper lists the learned coefficients for one best-performing PPO result as follows: Landing height $f(s,a)$5, Eroded piece cells $f(s,a)$6, Row transitions $f(s,a)$7, Column transitions $f(s,a)$8, Holes $f(s,a)$9, Board well $a$0, Hole depth $a$1, Rows with holes $a$2, and Diversity $a$3 (Chen et al., 24 Mar 2026). This supports the interpretation that the actor is effectively a learned linear scorer over hand-crafted afterstate descriptors.

The method’s limitations are also explicit. It depends on a domain in which the chosen action deterministically creates a post-decision board and randomness enters afterward; on the feasibility of enumerating all candidate actions and computing all afterstates; and on the availability of informative hand-crafted features. The paper further notes that transfer from 10x10 training to 10x20 is only partial because the standard board creates a larger state space, longer horizons, more difficult long-term value estimation, and different layout strategy demands. It also reports that all methods degrade severely under adversarial Z/S sequences (Chen et al., 24 Mar 2026). These observations indicate that the afterstate actor is not presented as a universal architecture; rather, it is a Tetris-specialized actor that exploits a particular decomposition between deterministic placement effects and stochastic next-piece generation.