Forward-Sampled Action Selection in RL

• Forward-sampled action selection is a family of reinforcement learning techniques that generate and refine action samples in a forward pass for immediate decision making.
• These methods filter actions using value estimates, density corrections, local search, or statistical criteria to effectively balance exploration and exploitation.
• They are applied across various settings such as continuous Q-learning, bounded SAC, and online planning, improving performance and stability in high-dimensional control tasks.

Forward-sampled action selection denotes a family of reinforcement-learning and planning procedures in which the decision mechanism uses samples generated in the forward pass itself to determine what action is executed, recommended, or further refined. Across the cited literature, the sampled object varies: a candidate action can be accepted or rejected according to an approximate value function, a stochastic-policy sample can be locally refined before environment interaction, a bounded-action policy can select the mode of its post-transform density rather than the transformed Gaussian mean, the available action set can itself be a random variable, and online planners can sample heuristic values or future action sequences to decide what to expand next (Kumar et al., 2021, Chang et al., 2024, Chen et al., 2024, Boutilier et al., 2018, Bonet et al., 2019, Hay et al., 2014, Zhang et al., 5 Jul 2025).

1. Conceptual frame and recurrent design pattern

The surveyed literature suggests a common operational template. First, some object relevant to control is sampled in the forward decision process: a feasible action, a perturbation around a policy sample, a post-squash action density, a realized action subset, a rollout-based heuristic, or a knockoff copy of the action vector. Second, that sampled object is filtered by a value estimate, a density correction, a local search rule, an availability constraint, or a statistical selection criterion. Third, the resulting decision is used immediately for environment interaction or for current-step planning, rather than being treated only as offline analysis.

Setting	Sampled object	Decision mechanism
Continuous-action Q-learning	Candidate action $\xi$	Accept-reject via $\hat q(\xi)$
Policy optimization with ESA	Policy sample $a_t$	Local ESC refinement before execution
Bounded-action SAC	Post-squash action grid	Select mode of transformed density
SAS-MDPs	Realized action set $A_s^{(t)}$	Choose best available action
Anytime planning	Sampled heuristics / rollouts	Expand IN or OUT tip nodes
Knockoff action selection	Action and knockoff copies	Threshold anti-symmetric statistics

A recurring distinction is between sampling as undirected exploration noise and sampling as a decision rule. QIS explicitly uses the approximate Q-function to define the sampling policy itself rather than an externally imposed $\epsilon$-greedy schedule; ESA does not execute the raw stochastic action, but refines it before execution; the SAC correction argues that $\tanh(\mu)$ need not be the most probable bounded action; SAS-MDPs require action choice to condition on the realized available set; and metalevel planning methods treat simulation selection as a decision problem in its own right (Kumar et al., 2021, Chang et al., 2024, Chen et al., 2024, Boutilier et al., 2018, Hay et al., 2014).

2. Value-shaped accept-reject sampling in continuous action spaces

In "Importance Sampling based Exploration in Q Learning" (Kumar et al., 2021), forward-sampled action selection is implemented as a behavior policy for continuous-action Q-learning. The state-action value function is approximated by a parametric model,

$\hat q_t(s,a,\theta)\approx q_t(s,a),$

with temporal-difference updates of the form

$$\hat q_t(s,a,\theta) \leftarrow (1-\lambda)\hat q_t(s,a,\theta) + \lambda\big(r_t(s_t,a_t)+\gamma \min_{a_{t+1}}\hat q_{t+1}(s_{t+1},a_{t+1},\theta)\big).$$

The paper’s novel step is to use the current approximation not only for backup targets but also for action generation.

The proposed method, Q-learning Importance Sampling (QIS), draws a proposed action $\xi$ from a simple proposal over the feasible action set, evaluates $\hat q(\xi)$, and accepts the proposal according to

$\hat q(\xi)$0

followed by acceptance if

$\hat q(\xi)$1

Because the problem is formulated as a minimization problem, smaller $\hat q(\xi)$2 values have higher acceptance probability. If $\hat q(\xi)$3 is near the current best value $\hat q(\xi)$4, then $\hat q(\xi)$5 is near $\hat q(\xi)$6; if it is near $\hat q(\xi)$7, acceptance becomes unlikely. The paper emphasizes that no action has zero probability of being accepted, so the scheme retains exploration even when it strongly biases toward promising actions.

This accept-reject rule is explicitly contrasted with $\hat q(\xi)$8-greedy. QIS does not use a fixed exploration probability, and it never performs a “pure exploit” sample in the $\hat q(\xi)$9-greedy sense. Early in learning, because $a_t$0 and $a_t$1 are crude, the acceptance probabilities are relatively broad and exploration is high. Later, as the approximation sharpens, the proposal distribution becomes more concentrated near low-cost actions, which gives an explicit exploration–exploitation tradeoff without a tunable $a_t$2. The paper also states that in this one-step Q-learning setting, the forward-sampling step does not require importance-weight correction, because the sampled action is the actual action being evaluated and updated (Kumar et al., 2021).

Algorithmically, the method performs a forward pass over stages and samples, then a backward pass that updates the Q-function approximation and recomputes $a_t$3 and $a_t$4. A computationally cheaper variant, QIS-RE, recomputes the historical extrema only every $a_t$5 iterations. In the numerical comparison on stochastic generation expansion planning, the authors report that QIS and QIS-RE produce more stable and higher-quality policies than epsilon-greedy and epsilon-decay, with less variability across replications (Kumar et al., 2021).

3. Sample-then-refine action execution via extremum-seeking control

"Extremum-Seeking Action Selection for Accelerating Policy Optimization" (Chang et al., 2024) formulates forward-sampled action selection as a sample-then-refine procedure inserted directly into model-free policy optimization. The setting is a continuous-control MDP

$a_t$6

with a Gaussian stochastic policy

$a_t$7

and return objective

$a_t$8

The paper argues that standard Gaussian action sampling is often inefficient in unstable robotic control tasks, because most random samples fall into regions that are dynamically disastrous or low-value.

The proposed mechanism, ESA, keeps the policy sample but refines it online using extremum-seeking control (ESC). Standard ESC is written as

$a_t$9

$A_s^{(t)}$0

In the RL adaptation, the procedure is: sample

$A_s^{(t)}$1

construct a probing input

$A_s^{(t)}$2

query the critic at

$A_s^{(t)}$3

update

$A_s^{(t)}$4

and execute the refined action

$A_s^{(t)}$5

The pseudocode given in the paper stores $A_s^{(t)}$6 in the replay buffer and then performs otherwise standard PPO- or SAC-style optimization.

The significance of ESA lies in the fact that the correction occurs before environment interaction. The sampled action is only the starting point of a local search, not the final decision. The paper emphasizes that no additional models or external oracles are needed beyond the critic or Q-network already present in actor-critic methods. Empirically, adding ESA accelerates learning, improves final performance, outperforms parameter noise and Ornstein–Uhlenbeck-noise baselines, and helps especially in the quadrotor task where vanilla PPO can diverge. The reported overhead is at most about 50% longer runtime per episode for 2048 steps. The stated limitations are equally specific: ESC is local, depends on frequency separation, assumes exploitable local Q-structure, requires careful tuning of $A_s^{(t)}$7, $A_s^{(t)}$8, and $A_s^{(t)}$9, and depends on critic quality (Chang et al., 2024).

4. Transformed policy densities and mode-aware action choice in bounded SAC

"Rethinking Soft Actor-Critic in High-Dimensional Action Spaces: The Cost of Ignoring Distribution Shift" (Chen et al., 2024) identifies a distinct forward-sampling issue in bounded-action actor-critic methods. SAC produces an unbounded Gaussian pre-squash action,

$\epsilon$0

and maps it to the bounded action space by

$\epsilon$1

The inverse transform is

$\epsilon$2

The paper’s central claim is that the nonlinear $\epsilon$3 transformation induces a distribution shift: the post-squash distribution is not simply a Gaussian clipped to $\epsilon$4, because the Jacobian term reshapes the density. In the general transformed-policy form,

$\epsilon$5

the determinant is

$\epsilon$6

This Jacobian factor implies that the deterministic action commonly used at evaluation time,

$\epsilon$7

need not coincide with the mode of the executed policy in post-squash space,

$\epsilon$8

The paper frames this as a forward-sampling mismatch: the policy network produces $\epsilon$9, SAC’s usual deterministic output uses $\tanh(\mu)$0, but the most probable post-squash action may be different. In multiple dimensions, the product Jacobian means that small per-dimension deviations can accumulate into a substantial mismatch in the joint action distribution, and the paper emphasizes that the effect becomes increasingly pronounced in high-dimensional action spaces such as Humanoid and HumanoidStandup.

The proposed correction separates inference and training. During inference, the method discretizes the action interval with

$\tanh(\mu)$1

evaluates $\tanh(\mu)$2 on the grid, and returns the maximizing $\tanh(\mu)$3. During training, it preserves stochasticity through inverse transform sampling with

$\tanh(\mu)$4

sampling $\tanh(\mu)$5 and selecting the smallest discretized $\tanh(\mu)$6 such that $\tanh(\mu)$7. The empirical comparison uses six MuJoCo tasks—HalfCheetah-v4, Hopper-v4, Walker2d-v4, Humanoid-v4, HumanoidStandup-v4, and Reacher-v4—and contrasts Original-deterministic, Original-refineSampling, and RefineT-refineSampling. On Humanoid-v4, refined methods improve cumulative reward by about 10% to 20%, with RefineT-refineSampling showing the best stability and narrowest variance bands; on HumanoidStandup-v4, RefineT-refineSampling improves performance by about 5% to 10%; on Reacher-v4, baseline and Original-refineSampling can be slightly better, suggesting that the correction is less beneficial in low-dimensional or simple tasks. The paper also defines convergence speed as

$\tanh(\mu)$8

and reports that RefineT-refineSampling is best on Humanoid-v4, while baseline converges fastest on Reacher-v4 (Chen et al., 2024).

5. Sampled action availability and compact optimal action ranking

"Planning and Learning with Stochastic Action Sets" (Boutilier et al., 2018) generalizes forward-sampled action selection to the case where the set of feasible actions is itself random. Starting from a discounted base MDP

$\tanh(\mu)$9

the available action set at state $\hat q_t(s,a,\theta)\approx q_t(s,a),$0 and time $\hat q_t(s,a,\theta)\approx q_t(s,a),$1 is a random subset

$\hat q_t(s,a,\theta)\approx q_t(s,a),$2

drawn from a state-dependent distribution $\hat q_t(s,a,\theta)\approx q_t(s,a),$3 under the history-independence assumption

$\hat q_t(s,a,\theta)\approx q_t(s,a),$4

The paper assumes $\hat q_t(s,a,\theta)\approx q_t(s,a),$5, so some action is always available.

To recover standard MDP semantics, the paper defines an embedded MDP whose states are $\hat q_t(s,a,\theta)\approx q_t(s,a),$6, and a compressed MDP over the original state space whose “actions” are state policies

$\hat q_t(s,a,\theta)\approx q_t(s,a),$7

The compressed optimal Bellman operator is written as

$\hat q_t(s,a,\theta)\approx q_t(s,a),$8

A central structural result is that optimal policies admit a compact decision-list representation: at each state, one stores a permutation of the base actions and, when the realized set is $\hat q_t(s,a,\theta)\approx q_t(s,a),$9, executes the first available action in that ordering. The key intuition given by the paper is that the Q-value of an action at a state does not depend on what other actions are present. Once actions are sorted by Q-value, the optimal action is simply the highest-ranked available one.

This structure yields exact and approximate planning methods. Under the product distribution assumption, each action $$\hat q_t(s,a,\theta) \leftarrow (1-\lambda)\hat q_t(s,a,\theta) + \lambda\big(r_t(s_t,a_t)+\gamma \min_{a_{t+1}}\hat q_{t+1}(s_{t+1},a_{t+1},\theta)\big).$$0 is independently available with probability $$\hat q_t(s,a,\theta) \leftarrow (1-\lambda)\hat q_t(s,a,\theta) + \lambda\big(r_t(s_t,a_t)+\gamma \min_{a_{t+1}}\hat q_{t+1}(s_{t+1},a_{t+1},\theta)\big).$$1, and if the actions are sorted by descending Q-value, the expected maximum can be computed efficiently. The paper states that the compressed Bellman operator can be computed in

$$\hat q_t(s,a,\theta) \leftarrow (1-\lambda)\hat q_t(s,a,\theta) + \lambda\big(r_t(s_t,a_t)+\gamma \min_{a_{t+1}}\hat q_{t+1}(s_{t+1},a_{t+1},\theta)\big).$$2

time per iteration. For arbitrary sampleable action-set distributions, it uses sampled realizations

$$\hat q_t(s,a,\theta) \leftarrow (1-\lambda)\hat q_t(s,a,\theta) + \lambda\big(r_t(s_t,a_t)+\gamma \min_{a_{t+1}}\hat q_{t+1}(s_{t+1},a_{t+1},\theta)\big).$$3

to approximate the expectation over subsets.

From the learning perspective, the paper shows that standard Q-learning remains sound provided that the next-state maximization is taken over the realized available action set rather than the full base action set: $$\hat q_t(s,a,\theta) \leftarrow (1-\lambda)\hat q_t(s,a,\theta) + \lambda\big(r_t(s_t,a_t)+\gamma \min_{a_{t+1}}\hat q_{t+1}(s_{t+1},a_{t+1},\theta)\big).$$4 To execute the learned policy, one simply chooses

$$\hat q_t(s,a,\theta) \leftarrow (1-\lambda)\hat q_t(s,a,\theta) + \lambda\big(r_t(s_t,a_t)+\gamma \min_{a_{t+1}}\hat q_{t+1}(s_{t+1},a_{t+1},\theta)\big).$$5

The paper therefore provides a principled formulation in which forward-sampled action selection means: the environment samples which actions are available, the agent observes the realized set, and the agent selects the best available action under a Q-ranking (Boutilier et al., 2018).

6. Forward planning with sampled heuristics and metalevel computation selection

In large MDPs, forward-sampled action selection also appears as online planning under limited computation. "Action Selection for MDPs: Anytime AO* vs. UCT" (Bonet et al., 2019) studies the task of choosing the current action by solving a truncated-horizon problem from the current state, executing one action, and replanning. A finite-horizon MDP is converted into an implicit AND/OR graph whose OR nodes are $$\hat q_t(s,a,\theta) \leftarrow (1-\lambda)\hat q_t(s,a,\theta) + \lambda\big(r_t(s_t,a_t)+\gamma \min_{a_{t+1}}\hat q_{t+1}(s_{t+1},a_{t+1},\theta)\big).$$6 and AND nodes are $$\hat q_t(s,a,\theta) \leftarrow (1-\lambda)\hat q_t(s,a,\theta) + \lambda\big(r_t(s_t,a_t)+\gamma \min_{a_{t+1}}\hat q_{t+1}(s_{t+1},a_{t+1},\theta)\big).$$7, with Bellman-style recurrences

$$\hat q_t(s,a,\theta) \leftarrow (1-\lambda)\hat q_t(s,a,\theta) + \lambda\big(r_t(s_t,a_t)+\gamma \min_{a_{t+1}}\hat q_{t+1}(s_{t+1},a_{t+1},\theta)\big).$$8

The proposed Anytime AO* algorithm, AOT, departs from classical AO* by explicitly balancing exploration and exploitation when heuristics are non-admissible and even randomly sampled. At each iteration it selects, with probability $$\hat q_t(s,a,\theta) \leftarrow (1-\lambda)\hat q_t(s,a,\theta) + \lambda\big(r_t(s_t,a_t)+\gamma \min_{a_{t+1}}\hat q_{t+1}(s_{t+1},a_{t+1},\theta)\big).$$9, a non-terminal tip node IN the best partial graph and, with probability $\xi$0, a non-terminal tip node OUT of the best partial graph; the default in experiments is $\xi$1. It also permits heuristic values to be random samples, such as rollouts of a base policy, and averages them incrementally until expansion. The paper further defines a $\xi$2 quantity that measures how much a node’s value would need to change to alter the best partial solution, and chooses tip nodes with minimal $\xi$3. Performance is measured by the average cost to the goal as a function of the time window per selected action, averaging 1,000 episodes, and the results show that AOT often outperforms domain-independent UCT, while typically starting more slowly because each expansion is more expensive (Bonet et al., 2019).

"Selecting Computations: Theory and Applications" (Hay et al., 2014) places this planning view inside a broader metalevel decision-theoretic framework. A metalevel probability model consists of utilities $\xi$4 and a countable set of computations $\xi$5 whose outcomes can be sampled. Given computation cost $\xi$6, the metalevel decision problem is an MDP in which actions are either to perform a computation or to stop. Its value function is

$\xi$7

where $\xi$8 is the total number of computations performed and

$\xi$9

The paper proves that the optimal policy’s expected number of computations is bounded by the value of perfect information times $\hat q(\xi)$0, gives a counterexample showing that stopping with probability one does not imply bounded computation almost surely, and shows that selection problems are not indexable in general. It introduces the myopic policy, the blinkered policy for independent actions, and distribution-free value-of-information bounds that lead to the heuristic rule: sample the arm with the largest estimated VOI and stop if all estimated VOIs are below the computation cost threshold. In Monte Carlo tree search, the paper proposes a hybrid scheme that uses VOI-based selection at the root and UCT at non-root nodes; in 9x9 Go, for threshold $\hat q(\xi)$1, the VOI-aware policy reaches a winning rate of about 64% for 10000 samples per ply (Hay et al., 2014).

Taken together, these two papers show that forward-sampled action selection need not mean direct sampling of physical actions only. It can also mean forward sampling of heuristic values, rollouts, or computations whose purpose is to improve the current action choice before acting.

7. Knockoff-based online action-coordinate selection in deep RL

"Where to Intervene: Action Selection in Deep Reinforcement Learning" (Zhang et al., 5 Jul 2025) studies a different but related problem: selecting which action coordinates matter in high-dimensional deep RL. The paper defines a sufficient action set $\hat q(\xi)$2 by the conditional independence conditions

$\hat q(\xi)$3

for all $\hat q(\xi)$4, and seeks a minimal sufficient action set. Its forward-sampling mechanism is model-free knockoff generation from the current policy. At each step,

$\hat q(\xi)$5

so the action and its knockoff are sampled directly and exactly from the same conditional distribution.

After collecting data, the response is

$\hat q(\xi)$6

and a generic ML selector computes original and knockoff importances $\hat q(\xi)$7 and $\hat q(\xi)$8, aggregates them via

$\hat q(\xi)$9

forms an anti-symmetric statistic

$\hat q(\xi)$00

for example $\hat q(\xi)$01, and selects variables by thresholding: $\hat q(\xi)$02 The threshold is

$\hat q(\xi)$03

To stabilize selection under dependent online RL data, the paper uses sample splitting and majority vote, with

$\hat q(\xi)$04

and proves the modified false discovery rate guarantee

$\hat q(\xi)$05

under stationarity and exponential $\hat q(\xi)$06-mixing.

Integration into deep RL is through a hard mask. If $\hat q(\xi)$07 is the selected mask, then critic masking is

$\hat q(\xi)$08

and policy masking is

$\hat q(\xi)$09

Empirically, the method is evaluated on Ant, HalfCheetah, and Hopper with artificially enlarged action dimensions $\hat q(\xi)$10 and $\hat q(\xi)$11, using PPO and SAC, and comparing All, True, and KS. The reported findings are that KS achieves much better reward performance than All, often performs close to True, and achieves high TPR with very low FDR/FPR. Examples given in the paper include Ant/PPO with $\hat q(\xi)$12, where KS obtains FDR $\hat q(\xi)$13, FPR $\hat q(\xi)$14, reward $\hat q(\xi)$15, while All has FDR $\hat q(\xi)$16, FPR $\hat q(\xi)$17, reward $\hat q(\xi)$18; HalfCheetah/SAC with $\hat q(\xi)$19, where KS reward is $\hat q(\xi)$20 versus $\hat q(\xi)$21 for All; and Hopper/PPO with $\hat q(\xi)$22, where KS reward is $\hat q(\xi)$23 versus $\hat q(\xi)$24 for All. The paper further states that the selection step can finish in under 20 seconds with a few thousand samples, and reports a clinical control application on MIMIC-III in which clinically relevant treatments such as vaso_dose and iv_input are identified while many less relevant treatments are avoided (Zhang et al., 5 Jul 2025).