Stochastic Decision Horizons

Updated 5 July 2026

Stochastic decision horizons are modeling devices that define the explicit lookahead, reward claim period, and trajectory influence in decision-making problems.
They are implemented through mechanisms like finite tree depth, random deadlines, state–action continuation, and temporal abstraction, adapting to uncertainty and constraint considerations.
Key applications include multistage stochastic programming, optimal stopping, constrained reinforcement learning, and contextual bandits, addressing computational trade-offs and decision efficacy.

Stochastic decision horizons are modeling devices that determine how far a decision problem explicitly “looks ahead,” how long rewards remain claimable, or how long trajectories continue to contribute to value. Across recent work, the horizon may be a finite scenario-tree depth $T$ , a random deadline $\tau$ , a state–action–dependent continuation probability $\alpha(s,a)$ , a reward-dependence horizon $h$ , or a macro-action length $L$ that compresses many primitive steps into one abstract decision (Gioia et al., 2022, Campbell et al., 10 Mar 2025, Milosevic et al., 4 Feb 2026). The common issue is not merely temporal extent, but how uncertainty, feasibility, and information interact with truncation, continuation, and stopping. In this sense, stochastic decision horizons connect multistage stochastic programming, optimal stopping, constrained reinforcement learning, bandits with delayed temporal dependence, finite-horizon control complexity, and stochastic extensive-form models.

1. Conceptual scope and formal variants

In multistage stochastic programming, the decision horizon is the finite “look-ahead” over which one explicitly models uncertainty via a scenario tree $\mathcal T$ , with periods $t=0,1,\dots,T$ ; beyond $T$ the model ignores further uncertainty (Gioia et al., 2022). In filtering-and-stopping problems, the horizon is a random deadline $\tau$ with survival functions $c_i(t)=P(\tau>t\mid X=r_i)$ , including exponential deadlines, deterministic horizon $\tau$ 0, and general nonincreasing continuous $\tau$ 1 on $\tau$ 2 with $\tau$ 3 (Campbell et al., 10 Mar 2025). In constrained reinforcement learning, a stochastic decision horizon replaces the constant discount interpretation of survival with a state–action–dependent continuation probability $\tau$ 4, so infeasible or unsafe actions can shorten the horizon locally (Milosevic et al., 4 Feb 2026).

Horizon mechanism	Formal device	Representative setting
Finite look-ahead	Tree depth $\tau$ 5	Multistage ATO planning
Random availability	Deadline $\tau$ 6, survival $\tau$ 7	Sequential estimation and stopping
Local continuation	$\tau$ 8	Constrained RL
Long reward dependence	Horizon $\tau$ 9, sparse $\alpha(s,a)$ 0	Contextual bandits
Temporal abstraction	Macro length $\alpha(s,a)$ 1	Offline RL with MCTS

A recurring distinction is between a horizon as an exogenous time limit and a horizon as an endogenous consequence of the chosen action. The former appears in finite-horizon models and stochastic deadlines; the latter appears when continuation itself depends on state, action, or learned abstraction. This suggests that “horizon design” is not a single modeling choice but a family of mechanisms for allocating uncertainty resolution, computational effort, and credit assignment across time.

2. Truncation, myopia, and terminal valuation in stochastic programming

In assemble-to-order planning, longer horizons $\alpha(s,a)$ 2 yield a longer explicit look-ahead but lead to exponentially many scenarios, making the model computationally intractable for $\alpha(s,a)$ 3 large (Gioia et al., 2022). The general multistage model optimizes over production, assembly, inventory, and lost sales on a scenario tree, and truncation creates the well-known end-of-horizon effect: a look-ahead of only a few periods tends to undervalue inventory that will be useful beyond the model’s last stage. The pure two-stage model ( $\alpha(s,a)$ 4) is explicitly described as myopic because it never rewards keeping components for $\alpha(s,a)$ 5.

To offset this, the enriched two-stage model FOSVA adds a terminal value function $\alpha(s,a)$ 6 to the second stage and approximates it as a separable concave, piecewise-linear function,

$\alpha(s,a)$ 7

with breakpoints $\alpha(s,a)$ 8 and slopes $\alpha(s,a)$ 9. The slopes are learned offline by finite differences on the two-stage model, and concavity is enforced by taking running averages. This preserves a MILP structure while introducing a salvage term for inventory at the horizon boundary (Gioia et al., 2022).

The paper compares TS, FOSVA, MP, MS3, MS3+ $h$ 0, and deterministic linear models in rolling-horizon simulation over a 24-month horizon with data characterized by seasonality, bimodality, and correlations in end-item demand. Computational experiments use 35 end-items, 60 components, 5 machines, and capacity tightness $h$ 1. The pure two-stage policies are reported as severely myopic, achieving approximately $h$ 2– $h$ 3 of perfect-information profit and suffering lost sales greater than $h$ 4 above average. FOSVA consistently doubles profit to about $h$ 5– $h$ 6 of perfect information, while small three-stage trees or mean-demand tails achieve profits $h$ 7– $h$ 8 points below FOSVA and keep higher inventories, approximately $h$ 9– $L$ 0 of the perfect-information level. Adding more stages often hurts performance due to overstocking and extra holding cost, and deterministic MRP with safety stocks does not match FOSVA or MS3/MP (Gioia et al., 2022).

A common misconception is that deeper trees necessarily dominate shallower approximations. The reported results do not support that conclusion: in these ATO instances, a two-stage look-ahead with a learned terminal value can outperform deeper but truncated multistage trees.

3. Random deadlines, stopping boundaries, and random-horizon contracts

In the sequential estimation problem with an unknown reward $L$ 1, observations follow

$L$ 2

and the posterior probability $L$ 3 evolves as

$L$ 4

The reward must be claimed before a random deadline $L$ 5, and the objective is expressed through

$L$ 6

There is a threshold $L$ 7 such that continuation occurs for $L$ 8, and stopping occurs as soon as $L$ 9. In the continuation region, $\mathcal T$ 0 solves a linear parabolic PDE; on the boundary, both value matching and smooth fit hold; and the boundary satisfies a Volterra integral equation (Campbell et al., 10 Mar 2025).

The examples clarify how horizon structure changes optimal behavior. With exponential deadlines $\mathcal T$ 1 and $\mathcal T$ 2, the boundary decreases smoothly from $\mathcal T$ 3 toward $\mathcal T$ 4 as $\mathcal T$ 5. With a deterministic horizon $\mathcal T$ 6, one recovers a finite-horizon classical sequential test and $\mathcal T$ 7 (Campbell et al., 10 Mar 2025). The reported interpretation is that stochastic deadlines fundamentally alter the value of information: under infinite horizon one never stops unless $\mathcal T$ 8, whereas under a finite or stochastic deadline the stopper trades off delay in learning against risk of deadline.

Random horizons also arise in continuous-time principal–agent models. In that setting, both the principal and the agent may terminate the contract at a finite $\mathcal T$ 9-stopping time $t=0,1,\dots,T$ 0, and the contract is a triple $t=0,1,\dots,T$ 1 consisting of a designer’s stopping time, a continuous transfer rate, and a lump-sum payment at termination (Lin et al., 2020). The agent’s continuation-utility process $t=0,1,\dots,T$ 2 is used to parameterize revealing contracts, and the second-best problem is reduced to a stochastic-control problem for the principal. In the American version, quitting is endogenous: the agent may choose $t=0,1,\dots,T$ 3. This places stochastic decision horizons directly inside incentive constraints and stopping rules rather than treating them as exogenous deadlines only (Lin et al., 2020).

4. State–action continuation and survival-weighted returns in constrained RL

In constrained reinforcement learning, stochastic decision horizons are introduced through a Control as Inference formulation in which a trajectory continues according to both a discount gate and a continuation gate. If $t=0,1,\dots,T$ 4 and $t=0,1,\dots,T$ 5, then continuation requires $t=0,1,\dots,T$ 6 and $t=0,1,\dots,T$ 7, and the expected survival gate yields the survival-weighted return

$t=0,1,\dots,T$ 8

Here $t=0,1,\dots,T$ 9 attenuates both immediate reward and future rewards, so constraint-violating state-actions effectively truncate long-term credit (Milosevic et al., 4 Feb 2026).

Two violation semantics are distinguished. Under absorbing-state semantics (AS), when $T$ 0 the process truly terminates and the KL term is also gated by prior survival. Under virtual termination (VT), the process continues generating actions after feasibility failure, but future rewards remain gated by $T$ 1, while the KL cost is not gated by $T$ 2. Both share the same survival-weighted reward term but induce different Bellman updates and different off-policy solution methods. AS-SAC uses a single soft $T$ 3-function under the shaped MDP with reward $T$ 4 and discount $T$ 5. VT-MPO uses the unregularized survival $T$ 6 with actor update exactly the standard MPO E-step and M-step after substituting $T$ 7 (Milosevic et al., 4 Feb 2026).

The reported experiments cover Safety Gymnasium and Hyfydy musculoskeletal control. VT-MPO and AS-SAC reduce cost heavily relative to unconstrained MPO while retaining high reward; compared to CPO and C-TRPO, the SDH methods learn faster and achieve equal or lower violations at higher returns; and VT-MPO scales effectively to the high-dimensional Hyfydy setup. The same source also states an important limitation: SDH trades exact constraint satisfaction guarantees for a relaxed termination-style surrogate and is not a true chance-constrained MDP (Milosevic et al., 4 Feb 2026).

This is a central controversy in interpretation. Treating constraints as horizon-shortening continuation is operationally convenient and replay-compatible, but it is not equivalent to exact CMDP feasibility.

5. Long-range dependence, temporal abstraction, and effective horizon reduction

In stochastic contextual bandits with long horizon rewards, the current reward depends on at most $T$ 8 prior actions and contexts, up to a time horizon of $T$ 9. The model uses a filtered context

$\tau$ 0

with $\tau$ 1 $\tau$ 2-sparse, $\tau$ 3, and reward

$\tau$ 4

The main objective is to avoid polynomial dependence on $\tau$ 5. The paper gives regret bounds $\tau$ 6 in the data-poor regime $\tau$ 7 and $\tau$ 8 in the data-rich regime $\tau$ 9, and attributes this to exploiting sparsity rather than naive low-rank recovery (Qin et al., 2023). This formalizes one meaning of stochastic decision horizon: the horizon of reward dependence may be very long, but the effective learnable horizon is governed by sparse support and the accumulation of mass in $c_i(t)=P(\tau>t\mid X=r_i)$ 0.

A distinct but related mechanism appears in learned temporal abstraction for offline RL. L-MAP introduces fixed-length macro-actions $c_i(t)=P(\tau>t\mid X=r_i)$ 1, learns a discrete latent action space with a state-conditional VQ-VAE and a prior model $c_i(t)=P(\tau>t\mid X=r_i)$ 2, and plans with MCTS over latent macro-actions. Because each latent action corresponds to $c_i(t)=P(\tau>t\mid X=r_i)$ 3 primitive steps, a depth- $c_i(t)=P(\tau>t\mid X=r_i)$ 4 search covers $c_i(t)=P(\tau>t\mid X=r_i)$ 5 primitive steps. The source states that choosing $c_i(t)=P(\tau>t\mid X=r_i)$ 6 large effectively shortens the decision-making horizon and reduces branching, enabling deeper lookahead over fewer macro-steps; it also states that progressive widening allows the planner to refine by adding more distinct latent actions as time permits (Luo et al., 28 Feb 2025).

The empirical results are reported in terms of both latency and return. On MuJoCo locomotion with macro length $c_i(t)=P(\tau>t\mid X=r_i)$ 7 and planning steps $c_i(t)=P(\tau>t\mid X=r_i)$ 8 in raw space, L-MAP sustains an approximately $c_i(t)=P(\tau>t\mid X=r_i)$ 9 Hz control loop, whereas Trajectory Transformer takes about $\tau$ 00– $\tau$ 01 Hz for similar horizons. On stochastic MuJoCo, L-MAP mean $\tau$ 02 exceeds TAP $\tau$ 03, TT $\tau$ 04, and CQL $\tau$ 05; on deterministic MuJoCo, L-MAP $\tau$ 06 matches TT $\tau$ 07 and surpasses TAP $\tau$ 08 (Luo et al., 28 Feb 2025). A plausible implication is that temporal abstraction can act as a horizon-selection mechanism in its own right, even when the underlying MDP discount factor is unchanged.

6. Complexity, infinite-horizon formulations, and stochastic extensive forms

Finite-horizon strategy complexity provides a lower-level view of horizon effects. For finite-horizon MDPs and simple stochastic games with reachability objectives, $\tau$ 09-optimal counter-based strategies require $\tau$ 10 memory, and there exist FMDPs whose only optimal strategies are counter-based with period

$\tau$ 11

The upper bound arises because, if

$\tau$ 12

then an infinite-horizon optimal strategy is $\tau$ 13-optimal in the finite-horizon game, so implementation only needs a counter up to $\tau$ 14 (Chatterjee et al., 2012). This shows that even when infinite-horizon policies are memoryless, finite horizons can induce substantial periodic and memory complexity.

At the opposite extreme, infinite-horizon stochastic programs exploit stationarity directly. In CE-Inf-EDDP, the Bellman recursion

$\tau$ 15

is approximated by an explorative dual dynamic programming algorithm that reuses cuts under stationarity and merges forward and backward passes for more frequent cut updates (Ju et al., 2023). The source explicitly contrasts infinite and finite horizon design: an infinite-horizon formulation avoids explicit truncation at $\tau$ 16, while a finite-horizon approximation must choose $\tau$ 17 large enough so $\tau$ 18 tolerance.

Multi-horizon stochastic programming introduces another computational response: replacing recourse maps $\tau$ 19 by neural network surrogates $\tau$ 20 embedded as MILP constraints. In a 15-year UK power-system planning case study with 5-year steps and short-term operational scenarios $\tau$ 21, the surrogate-based approach is reported as up to $\tau$ 22 times faster than the deterministic equivalent, with $\tau$ 23, MAPE $\tau$ 24, comparably low in-sample dispersion, and improved out-of-sample performance relative to the deterministic equivalent (Zhang et al., 2 Dec 2025). This suggests that horizon complexity is not only a modeling issue but also an approximation issue: one may retain a multi-horizon structure while compressing the operational tail.

At the most abstract level, stochastic extensive forms generalize dynamic games under exogenous uncertainty. A stochastic decision forest is built on a measurable scenario space $\tau$ 25, and well-posedness is characterized by order-theoretic properties of the underlying forest. In action-path forms, well-posedness holds if and only if the time axis $\tau$ 26 is essentially well-ordered; to handle instantaneous reaction, the framework introduces vertically extended time

$\tau$ 27

with lexicographic order (Rapsch, 6 Aug 2025). Here the decision horizon is no longer merely a scalar length. It becomes part of the structure of information, reaction, and temporal ordering itself.

Taken together, these strands show that stochastic decision horizons are not a single technique but a cross-cutting formal theme. They govern myopia in truncated stochastic programs, threshold structure under random deadlines, survival-weighted credit assignment in constrained RL, sparse recovery under long reward dependence, macro-level temporal compression, memory complexity in finite-horizon control, and well-posedness in stochastic extensive forms.