Targeted Rollout Algorithm in Sequential Decisions

• Targeted Rollout Algorithm is a family of methods that selectively allocates simulation or correction resources to the most decision-relevant parts of sequential decision problems.
• It employs techniques such as biased aggregation, OCBA-guided simulation allocation, and uncertainty-aware termination to enhance evaluation efficiency and policy improvement.
• The approach is applicable in various domains including discounted MDPs, multiagent control, RL post-training, and even experimental deployment and energy management.

As an umbrella term, Targeted-Rollout Algorithm is best understood as referring to rollout-based methods that concentrate lookahead, correction capacity, or rollout budget on the parts of a sequential decision problem that are most decision-relevant. The literature uses this idea in several non-identical ways: biased aggregation that corrects rollout errors locally in discounted MDPs, OCBA-guided simulation allocation across candidate actions, point-based rollout on reachable belief states in DEC-POMDPs, agent-by-agent rollout in multiagent control, and rollout scheduling or replay for RL post-training of LLMs. This suggests a family of related constructions rather than a single standardized algorithm (Bertsekas, 2019, Sarkale et al., 2018, Wu et al., 2012, Lu et al., 9 Feb 2026).

1. Formal rollout basis and the meaning of targeting

In its classical form, rollout is a one-step policy-improvement method. For a discounted MDP with Bellman operator

$$(TJ)(x)=\min_{a\in A(x)}\Big\{g(x,a)+\alpha\sum_{x'}p(x'\mid x,a)J(x')\Big\},$$

the optimal cost function satisfies $J^*=TJ^*$, while a rollout policy uses a base policy cost-to-go or surrogate value in a one-step lookahead. In the discounted control notation used for infinite-horizon rollout, improvement from a base policy $\mu$ is obtained by

$$u^*(x)\in \arg\min_{u\in U(x)} \sum_{y} p_{xy}(u)\,[g(x,u,y)+\alpha J_\mu(y)].$$

The standard improvement guarantee is that rollout with a base policy’s exact cost-to-go is no worse than the base policy; finite-horizon and infinite-horizon versions of this monotonicity are treated explicitly in the dynamic programming and approximate dynamic programming formulations (Bertsekas, 2019, Bertsekas, 2019, Bertsekas, 2022).

The “targeted” aspect enters when the surrogate value, the state subset on which it is defined, the action components optimized, or the rollouts themselves are selected to match the structure of the objective. In Bayesian optimization and sequential estimation, targeting means choosing the terminal surrogate cost $G(b_N)$ so that the one-step lookahead explicitly aims at the final optimization or estimation objective rather than a generic information measure. In deterministic optimal control, targeting can mean restricting terminal feasibility and data fitting to a forward-invariant sampled set generated by a base policy. In model-based RL, the target can be regions where the learned model is locally trustworthy rather than a uniform rollout depth everywhere (Bertsekas, 2022, Li et al., 2021, Frauenknecht et al., 2024).

A common misconception is that targeted rollout is synonymous with deeper lookahead. The literature shows a different pattern: many targeted-rollout methods keep one-step or shallow-horizon lookahead, but direct computational effort toward selected aggregates, selected belief points, selected action components, or selected rollout samples. This suggests that “targeting” is principally about where rollout effort is spent, not only how far the lookahead extends.

2. Biased aggregation and targeted policy improvement in discounted MDPs

A canonical formalization appears in biased aggregation for approximate dynamic programming. The central construct is a bias function $V\in\mathbb{R}^n$, together with aggregate states, disaggregation probabilities, and aggregation probabilities arranged in a cyclic transition structure $A\to I_0\to I_1\to A$. The induced shaped per-stage cost is

$g'(i,u,j)=g(i,u,j)-V(i)+\alpha V(j),$

and the optimal costs of the original and shaped problems satisfy $J^*(i)=J(i)+V(i)$. This preserves optimal policies, but the approximation quality of the aggregate problem depends crucially on the choice of $V$ (Bertsekas, 2019).

Eliminating the intermediate variables yields a fixed-point equation over a low-dimensional aggregate correction vector $J^*=TJ^*$0:

$J^*=TJ^*$1

The mapping $J^*=TJ^*$2 is a sup-norm contraction and has a unique fixed point $J^*=TJ^*$3. Once $J^*=TJ^*$4 is computed, the corrected value used for improvement is

$J^*=TJ^*$5

and the improved policy is obtained by one-step lookahead with $J^*=TJ^*$6 rather than with $J^*=TJ^*$7 alone (Bertsekas, 2019).

The special case that most directly motivates the phrase targeted rollout is $J^*=TJ^*$8. With a single aggregate state, biased aggregation reduces exactly to standard rollout based on $J^*=TJ^*$9. With multiple aggregate states, the method becomes a more powerful targeted rollout in which the correction is constant within each aggregate, so that

$\mu$0

The additional term $\mu$1 focuses correction capacity on the aggregates where rollout residuals are largest; the paper recommends grouping states so that $\mu$2 varies little within each aggregate, for small $\mu$3 (Bertsekas, 2019).

Approach	Specification	Characteristic
Classical aggregation	$\mu$4	Coarse piecewise-constant approximation with few aggregates
Standard rollout	One aggregate, $\mu$5	One-step improvement with no local correction
Targeted rollout	Multiple aggregates, $\mu$6	Local correction where rollout is most in error

The associated error and convergence properties are explicit. In hard aggregation, if the variation of $\mu$7 within each aggregate is at most $\mu$8, then

$\mu$9

If $$u^*(x)\in \arg\min_{u\in U(x)} \sum_{y} p_{xy}(u)\,[g(x,u,y)+\alpha J_\mu(y)].$$0 is close to $$u^*(x)\in \arg\min_{u\in U(x)} \sum_{y} p_{xy}(u)\,[g(x,u,y)+\alpha J_\mu(y)].$$1, then $$u^*(x)\in \arg\min_{u\in U(x)} \sum_{y} p_{xy}(u)\,[g(x,u,y)+\alpha J_\mu(y)].$$2, so the aggregate correction is small; if $$u^*(x)\in \arg\min_{u\in U(x)} \sum_{y} p_{xy}(u)\,[g(x,u,y)+\alpha J_\mu(y)].$$3, then $$u^*(x)\in \arg\min_{u\in U(x)} \sum_{y} p_{xy}(u)\,[g(x,u,y)+\alpha J_\mu(y)].$$4 and the scheme is exact. The same framework yields an enhanced approximate policy iteration in which $$u^*(x)\in \arg\min_{u\in U(x)} \sum_{y} p_{xy}(u)\,[g(x,u,y)+\alpha J_\mu(y)].$$5 is obtained by Monte Carlo, TD/LSTD/LSPE, Q-learning/SARSA, neural networks, or aggregation-based policy evaluation, and the aggregate design may be changed at each iteration (Bertsekas, 2019).

3. Budget-aware and uncertainty-aware targeting

A distinct meaning of targeted rollout appears in simulation optimization, where the target is not an aggregate state but the allocation of rollout samples across candidate actions. In network-level post-hazard recovery, rollout uses a base policy $$u^*(x)\in \arg\min_{u\in U(x)} \sum_{y} p_{xy}(u)\,[g(x,u,y)+\alpha J_\mu(y)].$$6 and a depth-$$u^*(x)\in \arg\min_{u\in U(x)} \sum_{y} p_{xy}(u)\,[g(x,u,y)+\alpha J_\mu(y)].$$7 Monte Carlo estimate

$$u^*(x)\in \arg\min_{u\in U(x)} \sum_{y} p_{xy}(u)\,[g(x,u,y)+\alpha J_\mu(y)].$$8

Instead of allocating equal simulation effort to every action, OCBA solves a budget-allocation problem that maximizes the probability of correct selection under a fixed budget, with the asymptotic rule

$$u^*(x)\in \arg\min_{u\in U(x)} \sum_{y} p_{xy}(u)\,[g(x,u,y)+\alpha J_\mu(y)].$$9

where $G(b_N)$0 is the current sample variance and $G(b_N)$1 is the gap from the current best action. On the water-network recovery problem, rollout fused with OCBA performed competitively with rollout under total equal allocation at a simulation budget of about $G(b_N)$2–$G(b_N)$3 of rollout with total equal allocation, while retaining the non-myopic character of rollout (Sarkale et al., 2018).

A different budget-and-depth targeting mechanism appears in model-based actor-critic with uncertainty-aware rollout adaption. There the question is framed as “Where to trust your model?” rather than “When to trust your model?”. The method uses an ensemble-based geometric Jensen–Shannon divergence $G(b_N)$4 as a proxy for local epistemic uncertainty, and terminates a model rollout as soon as

$G(b_N)$5

or $G(b_N)$6. The threshold $G(b_N)$7 is adapted online from a quantile of first-step uncertainties, scaled by a single factor $G(b_N)$8. This produces longer model rollouts in locally well-modeled regions and shorter rollouts elsewhere, and the paper reports substantial improvements in data efficiency and performance over MBPO and M2AC on MuJoCo, often matching or surpassing SAC (Frauenknecht et al., 2024).

These examples show that targeting may act across candidate actions or along rollout depth. A plausible implication is that rollout quality depends at least as much on selective allocation of finite simulation effort as on the nominal horizon of the lookahead.

4. Multiagent, decentralized, and partially observed variants

In multiagent control, targeted rollout often means optimizing only a subset of decision components at a time while the remaining components are fixed to base-policy actions. For a joint control $G(b_N)$9, standard rollout requires evaluating up to $V\in\mathbb{R}^n$0 joint actions when each agent has at most $V\in\mathbb{R}^n$1 actions. Multiagent local rollout instead unfolds the joint decision into sequential subdecisions. Agent $V\in\mathbb{R}^n$2 minimizes a local Q-factor in which earlier agents’ choices are fixed to their already-computed rollout actions and later agents’ choices are fixed to the base policy. This reduces complexity from $V\in\mathbb{R}^n$3 to $V\in\mathbb{R}^n$4 per decision state while preserving the finite-horizon cost improvement property relative to the base policy; in discounted infinite-horizon approximate policy iteration, the resulting sequence converges, under a tie-breaking rule, to an agent-by-agent optimal policy (Bertsekas, 2019).

The same principle is extended to partially observable multiagent problems. In the multiagent POMDP setting, one-agent-at-a-time truncated rollout and an order-optimized variant reduce the per-step computational burden from $V\in\mathbb{R}^n$5 to $V\in\mathbb{R}^n$6, or to $V\in\mathbb{R}^n$7 when the agent order is optimized online. The application to multi-robot repair on graphs under partial information reaches state space sizes of approximately $V\in\mathbb{R}^n$8 and control space sizes of approximately $V\in\mathbb{R}^n$9, and the rollout methods are embedded in an offline approximate policy iteration scheme using neural network classifiers to approximate successive rollout policies (Bhattacharya et al., 2020).

In DEC-POMDPs, targeting is shifted from action components to the belief simplex. DecRSPI samples reachable beliefs under heuristic joint policies, then performs rollout-based one-step lookahead only at those sampled beliefs. The controller representation has memory usage $A\to I_0\to I_1\to A$0, and the algorithmic cost scales linearly with the number of agents when per-agent parameters are fixed. On the stochastic Mars Rover benchmark at horizon $A\to I_0\to I_1\to A$1, the reported runtime is approximately $A\to I_0\to I_1\to A$2 for DecRSPI versus approximately $A\to I_0\to I_1\to A$3 for PBIP-IPG, while learned values remain close to the model-based baseline (Wu et al., 2012).

A constrained deterministic analogue appears in multidimensional assignment and related layered-graph problems. There, targeted multiagent rollout replaces a joint search over up to $A\to I_0\to I_1\to A$4 controls with $A\to I_0\to I_1\to A$5 sequential micro-decisions of at most $A\to I_0\to I_1\to A$6 choices each, so the per-stage evaluation cost drops to $A\to I_0\to I_1\to A$7. The cost-improvement property is preserved through a fortified rollout construction, and the repeated solution of closely related two-dimensional assignments is accelerated by auction-algorithm warm starts through price reuse (Bertsekas, 2020).

5. Targeted rollout in LLM post-training and RLVR

Recent LLM post-training work reinterprets rollout targeting as data selection. Instead of using every generated rollout uniformly, these methods score or filter rollouts, prompt groups, or replayed samples according to their expected contribution to policy improvement.

Method	Targeting unit	Selection signal
Contextual Rollout Bandits	Individual rollout or buffered sample	Training-dynamics context and induced performance gain
Pilot-Commit	Prompt group	Pilot-estimated reward variance or success-rate band
DOTS + rollout replay	Question and recent rollout	Adaptive difficulty near $A\to I_0\to I_1\to A$8 and replay usefulness

One formulation is “Contextual Rollout Bandits” (CBS). The paper explicitly states that it does not use the phrase “Targeted-Rollout Algorithm,” and that CBS instantiates targeted rollout selection: each rollout is an arm in a contextual bandit, described by a $A\to I_0\to I_1\to A$9-dimensional context including reward signals, group statistics, length, truncation, entropy, clip ratio, usage count, and sample age. The scheduler is a neural MLP trained online with an MSE loss on advantage-weighted rewards, and selection combines exploitation with freshness-aware $g'(i,u,j)=g(i,u,j)-V(i)+\alpha V(j),$0-greedy exploration. The method provides a regret bound

$g'(i,u,j)=g(i,u,j)-V(i)+\alpha V(j),$1

and proves that enlarging the rollout buffer improves the achievable performance upper bound. Across six mathematical-reasoning benchmarks and GRPO, DAPO, and GSPO training regimes, intra-group CBS reduced average policy-optimization wall time by $g'(i,u,j)=g(i,u,j)-V(i)+\alpha V(j),$2, while buffer-related overhead remained at most $g'(i,u,j)=g(i,u,j)-V(i)+\alpha V(j),$3 of total time (Lu et al., 9 Feb 2026).

A more explicitly budget-allocation view is given by Pilot-Commit for group-based RL post-training. The pilot stage estimates prompt informativeness from a fraction of the budget using the proxy $g'(i,u,j)=g(i,u,j)-V(i)+\alpha V(j),$4 under binary rewards; the commit stage allocates the remaining rollouts to prompts whose pilot success rate lies within a retained band, with defaults such as $g'(i,u,j)=g(i,u,j)-V(i)+\alpha V(j),$5, $g'(i,u,j)=g(i,u,j)-V(i)+\alpha V(j),$6, and $g'(i,u,j)=g(i,u,j)-V(i)+\alpha V(j),$7. The rationale is that group-based updates are strongest when within-group reward variance is high, which in the binary case occurs near $g'(i,u,j)=g(i,u,j)-V(i)+\alpha V(j),$8. Across multiple math-reasoning benchmarks and models from $g'(i,u,j)=g(i,u,j)-V(i)+\alpha V(j),$9 to $J^*(i)=J(i)+V(i)$0 parameters, Pilot-Commit reached target accuracy up to $J^*(i)=J(i)+V(i)$1 faster than GRPO and $J^*(i)=J(i)+V(i)$2 faster than DAPO in cumulative rollouts (Kim et al., 26 May 2026).

Difficulty-targeted online data selection and rollout replay provide a closely related formulation. Adaptive difficulty is defined as the current failure rate

$J^*(i)=J(i)+V(i)$3

and a soft sampling distribution concentrates fresh rollouts around target difficulty $J^*(i)=J(i)+V(i)$4. The paper proves that for binary rewards the expected squared norm of the unclipped group-relative gradient is proportional to $J^*(i)=J(i)+V(i)$5, which is maximized at $J^*(i)=J(i)+V(i)$6. A rollout replay buffer then reuses recent informative samples through importance-corrected GRPO updates. Reported gains include total RL fine-tuning time reductions of $J^*(i)=J(i)+V(i)$7 to $J^*(i)=J(i)+V(i)$8 to reach the same performance as GRPO, together with a $J^*(i)=J(i)+V(i)$9 increase in the ratio of effective questions selected by DOTS (Sun et al., 5 Jun 2025).

These methods depart sharply from the classical control-theoretic meaning of rollout. Here the “target” is not a state or action sequence but the subset of sampled trajectories used to update a LLM. This suggests a broadened contemporary usage in which rollout is a training resource to be scheduled, filtered, or replayed rather than merely a one-step policy-improvement operator.

6. Related formulations, domain-specific instantiations, and conceptual boundaries

In deterministic optimal control, targeted rollout is formalized through a forward-invariant sampled set $V$0 generated by a base policy $V$1. The terminal value used by the depth-$V$2 rollout is

$V$3

where $V$4 on $V$5 and $V$6 otherwise. Under nonnegative stage costs and forward invariance, the rollout policy satisfies

$V$7

The same construction extends to trajectory constraints by state augmentation, and to multiagent systems through simplified joint-action subsets that still contain the base-policy action (Li et al., 2021).

A business-process instantiation treats targeted rollout as direct per-action Monte Carlo evaluation under a reward that exactly decomposes mean cycle time. With reward

$V$8

the cumulative return satisfies the sample-path identity

$V$9

so maximizing return is equivalent to minimizing total cycle time. The algorithm computes $J^*=TJ^*$00 by $J^*=TJ^*$01 rollouts of horizon $J^*=TJ^*$02 for each feasible action, uses common random numbers across actions, and trains a policy network by supervised classification on the best action labels. The reported result is that the method consistently learns the optimal policy in all six evaluated business processes, whereas the compared state-of-the-art algorithm can only learn the optimal policy in two (Middelhuis et al., 15 Apr 2025).

A further domain-specific example is airport service electric-vehicle energy management. The customized rollout evaluates candidate controls using two base heuristics: a renewable-matching charging heuristic and a greedy-charging heuristic, using the cheaper one-step-plus-heuristic completion whenever feasible. On both typical summer and winter days, the rollout achieves a total cost approximately $J^*=TJ^*$03 less than greedy charging, and the method is shown to be adaptive to short-notice flight-schedule changes (Wei et al., 2019).

The term also has a separate experimental-design and deployment meaning. In staggered experiments, rollout refers irreversible treatment adoption times across units rather than approximate dynamic programming. The non-adaptive precision-optimal schedule has a low–high–low shape in the fraction entering treatment each period, and the adaptive Precision-Guided Adaptive Experiment reduces opportunity cost by over $J^*=TJ^*$04 relative to static design benchmarks while preserving valid post-experiment inference under sample splitting (Xiong et al., 2019). In feature deployment for websites and mobile apps, staged rollout combines continuous monitoring through a variant of the sequential probability ratio test with automated ramp-up rules; the stated benefits are early regression detection for defective features, rapid rollout for healthy features, and reduced manual intervention through automation (Zhao et al., 2019).

A common misconception is therefore that targeted rollout always denotes a reinforcement-learning control law. The surveyed literature supports a narrower but more accurate statement: the phrase consistently signals selective allocation of rollout effort, but the object being rolled out may be a control sequence, a belief-state backup, a model-generated trajectory, a prompt group, a replay sample, a treatment schedule, or a feature flag ramp.