Option-level REINFORCE in Hierarchical RL

• Option-level REINFORCE is a policy-gradient framework in hierarchical RL that applies expected intra-option updates over multiple candidate options using call-and-return dynamics.
• The approach improves data efficiency and credit assignment by reusing a single primitive transition to update every plausible option, reducing estimator variance.
• Empirical results with Multi-updates Option Critic (MOC) show enhanced performance and faster convergence relative to standard option-critic models across diverse tasks.

Searching arXiv for recent and foundational papers on option-level policy gradients and related option frameworks. Option-level REINFORCE denotes policy-gradient learning in hierarchical reinforcement learning when temporally extended actions, or options, enter the gradient through option-conditioned action policies, termination functions, or a policy over options. In the literature represented here, the most direct modern instantiation is not a vanilla score-function estimator over sampled options, but an expected intra-option policy-gradient update that reuses a single primitive transition to update every option that is plausible under call-and-return dynamics (Klissarov et al., 2021). Related work places this idea in a broader landscape: adversarial imitation can train a high-level option selector as a stochastic policy over augmented states (Jing et al., 2021), attention-augmented option-critic keeps the same gradient theorems while changing the option-conditioned representation (Chunduru et al., 2022), inverse reinforcement learning can use option-gradient stationarity conditions as reward constraints (Hwang et al., 2019), and finite-horizon SMDP theory provides a non-policy-gradient contrast for joint high-level and low-level option learning (Drappo et al., 2024). This suggests that “option-level REINFORCE” is best understood as a family of policy-gradient constructions over hierarchical decision variables rather than as a single canonical algorithm.

1. Option-level policy gradients in the options framework

The standard setting is a Markov decision process

$$\mathcal{M}=\langle \mathcal{S}, \mathcal{A}, r, P \rangle,$$

with finite state space $\mathcal{S}$, finite action space $\mathcal{A}$, reward distribution $r(s,a)$, and transition kernel $P(s'|s,a)$. For a flat stochastic policy $\pi_\zeta(a|s)$, the ordinary policy-gradient theorem is recalled as

$$\frac{\partial J(\zeta)}{\partial \zeta} = \sum_s d^\gamma(s)\sum_a \frac{\partial \pi_\zeta(a|s)}{\partial \zeta} Q_\pi(s,a),$$

with discounted occupancy

$$d^\gamma(s)=\sum_{s_0} d(s_0)\sum_{t=0}^{\infty}\gamma^t P^\pi(S_t=s\mid S_0=s_0).$$

Option-level policy-gradient methods inherit this structure but replace flat state-action occupancy by state-option or state-option-action occupancy (Klissarov et al., 2021).

An option $o\in\mathcal{O}$ consists of an intra-option policy $\pi(a|s,o)$, a termination function $\mathcal{S}$0, and an initiation set $\mathcal{S}$1. The hierarchy also includes a policy over options

$\mathcal{S}$2

The execution model is call-and-return: at state $\mathcal{S}$3, the agent selects $\mathcal{S}$4, samples a primitive action $\mathcal{S}$5, transitions to $\mathcal{S}$6, samples termination according to $\mathcal{S}$7, and, if termination occurs, draws a new option from $\mathcal{S}$8 (Klissarov et al., 2021).

The key structural object is the upon-arrival option distribution

$\mathcal{S}$9

This quantity states that after arriving in $\mathcal{A}$0, either the current option terminates and a fresh option is drawn from the high-level policy, or the current option persists. In the formulation emphasized by “Flexible Option Learning,” this known distribution is the main analytical lever for replacing single sampled-option updates with expected multi-option updates (Klissarov et al., 2021).

Although initiation sets are part of the formal option definition, the modern option-critic simplification used in that work effectively takes initiation sets to be the full state space, while noting that the construction could be generalized with interest functions. A plausible implication is that most of the technical weight falls on intra-option policies, terminations, and the policy over options rather than on explicit initiation-set learning.

2. Expected multi-option intra-option policy gradient

The central modern interpretation of option-level REINFORCE is an expected intra-option policy-gradient update over all options, not merely the option that happened to execute. Standard option-critic uses the intra-option policy gradient

$\mathcal{A}$1

where

$\mathcal{A}$2

is the discounted occupancy over state-option pairs (Klissarov et al., 2021).

The key step is a structural decomposition of this occupancy: $\mathcal{A}$3 The unknown environment-dependent terms remain sampled from trajectories, but the factor $\mathcal{A}$4 is analytically available from the hierarchy itself. This permits a direct replacement of “update only the currently executing option” by “update all options proportionally to how likely they are to be active upon arrival” (Klissarov et al., 2021).

The resulting update is

$\mathcal{A}$5

This is the paper’s most direct contribution to what may be called option-level REINFORCE (Klissarov et al., 2021). It is policy-gradient-style, but it is not a plain REINFORCE estimator over options in the sense of sampling an option and applying $\mathcal{A}$6. Instead, it is an expected update over all option heads, conditioned on the arrival state and previous option.

The paper explicitly relates this construction to all-action policy gradient and conditional Monte Carlo. Conditioning on the arrival state and previous option makes it possible to average analytically over the option variable, which is the stated reason variance should decrease. The broader credit-assignment implication is also explicit: one primitive action can propagate learning signal simultaneously to many candidate options rather than only to the option that happened to be sampled. This is presented as a way to combat the “rich-get-richer” pathology in which already-favored options keep receiving most of the updates (Klissarov et al., 2021).

A common misconception is that multi-option updates require latent-option inference. In this formulation they do not. The method does use intra-option policy gradients and actor-critic-style gradients, but it does not require an additional inference network, does not require explicit estimation of the stationary state-option distribution, and is not cast as an EM or variational surrogate objective (Klissarov et al., 2021).

3. Option evaluation, off-policy correction, and consistency

The same expected-update idea appears on the evaluation side. For linear TD with $\mathcal{A}$7, the usual one-option update is rearranged as $\mathcal{A}$8. Under the assumptions that each option policy induces an ergodic Markov chain and that $\mathcal{A}$9 and $r(s,a)$0 are strictly positive on all state-option pairs, the limiting stationary distribution exists: $r(s,a)$1 with decomposition

$r(s,a)$2

This matters because the factor $r(s,a)$3 is known even when the full occupancy is not (Klissarov et al., 2021).

The resulting all-option evaluation step is

$r(s,a)$4

Thus a single primitive transition is reused to update all options, weighted by their call-and-return plausibility (Klissarov et al., 2021).

When evaluation considers an arbitrary counterfactual option $r(s,a)$5, the paper introduces an importance-sampled target

$r(s,a)$6

The paper states that this target is unbiased and corrects for the off-policyness created by updating options other than the sampled one (Klissarov et al., 2021). This is one of the sharpest distinctions from plain on-policy REINFORCE: once a primitive action is reused as evidence for multiple options, counterfactual option mismatch appears and requires correction for evaluation.

The notion of “consistency” is therefore softened relative to older deterministic intra-option learning views. In stochastic function-approximation settings, an option is relevant to the observed experience to the extent that $r(s,a)$7 is non-negligible and, for evaluation, to the extent that its intra-option policy assigns nonzero probability to the observed primitive action. This is also why deterministic options are a limitation here: if $r(s,a)$8 for the observed $r(s,a)$9, the importance ratio breaks down. The paper explicitly notes that the use of importance sampling “naturally excludes the possibility that options be defined through deterministic policies” (Klissarov et al., 2021).

Another potential misunderstanding is that importance sampling is intrinsic to the entire approach. The paper is explicit that this is not so: importance sampling is needed for option evaluation in the way the method is formulated, but not for the option-control algorithm itself (Klissarov et al., 2021).

4. Multi-updates Option Critic

The expected-update construction is presented as a drop-in replacement for the single-option update part of option-critic, yielding Multi-updates Option Critic, or MOC. The high-level loop remains the standard call-and-return procedure: choose an option $P(s'|s,a)$0, sample action $P(s'|s,a)$1, observe $P(s'|s,a)$2, sample termination from $P(s'|s,a)$3, and then either continue with the same option or draw a new option $P(s'|s,a)$4 (Klissarov et al., 2021).

What changes is the critic and intra-option actor update. For each option $P(s'|s,a)$5, the evaluation step uses

$P(s'|s,a)$6

followed by

$P(s'|s,a)$7

The improvement step similarly applies a weighted policy-gradient update to each option. The printed pseudocode contains a notational inconsistency because it uses $P(s'|s,a)$8 rather than $P(s'|s,a)$9 inside the log-policy and critic term; the intended update, consistent with the main learning equation, is the multi-option version with $\pi_\zeta(a|s)$0 in those positions (Klissarov et al., 2021).

Termination and master-policy learning are largely left intact. The termination update is

$\pi_\zeta(a|s)$1

with advantage

$\pi_\zeta(a|s)$2

The policy over options update is

$\pi_\zeta(a|s)$3

Accordingly, MOC should be viewed as preserving the standard option-critic termination and master-policy components while replacing single-option critic and intra-option actor updates with expected multi-option updates (Klissarov et al., 2021).

The paper also introduces a hyperparameter $\pi_\zeta(a|s)$4, the probability of performing all-option updates rather than sampled-option-only updates. This creates a continuum between standard option-critic and fully expected MOC. The stated motivation is a variance, bias, and diversity trade-off: expected updates should reduce variance, but aggressively updating all options may make them too similar (Klissarov et al., 2021).

5. Empirical profile and statistical trade-offs

The empirical claim attached to option-level expected updates is that many options can be improved simultaneously from one primitive transition with practical gains in performance and data efficiency. In FourRooms with fixed hallway options, updating all relevant options accelerates learning of option values relative to updating only the sampled option. In FourRooms with learned options, MOC reaches option-critic-level performance in about half the number of episodes and shows lower variance. In MountainCar with linear function approximation, MOC outperforms option-critic in both sample efficiency and transfer performance, and performance generally improves as the probability $\pi_\zeta(a|s)$5 of doing all-option updates increases. In MuJoCo transfer tasks—HalfCheetahDir, AntWalls, and Walker2dStand—MOC outperforms IOC, standard option-critic-style baselines, and flat PPO in sample efficiency and final performance. In MiniWorld transfer tasks, MOC often transfers better and more sample-efficiently than option-critic (Klissarov et al., 2021).

The paper presents these effects as consistent with the conditional Monte Carlo interpretation of the update. Expected multi-option averaging should lower estimator variance, and the simultaneous propagation of learning signal across options should improve credit assignment for temporally extended behaviors. This statistical interpretation is stated directly through the analogy to Expected SARSA and Expected Policy Gradients (Klissarov et al., 2021).

The diversity trade-off is more nuanced. The experiments report that in tabular settings, large $\pi_\zeta(a|s)$6 can reduce option diversity, whereas intermediate $\pi_\zeta(a|s)$7 can preserve or improve diversity. In function approximation, the issue appears less severe, and high $\pi_\zeta(a|s)$8 often helps. The authors suggest a future state-dependent $\pi_\zeta(a|s)$9 to restrict off-policy option updates where importance ratios are too large (Klissarov et al., 2021). This suggests that option-level REINFORCE in expectation is not automatically diversity-preserving; the reuse of one primitive transition across many option heads can improve efficiency while also encouraging homogenization if not modulated.

The limitations stated for this mechanism are likewise structural. The evaluation analysis assumes ergodicity of each option policy and positivity of $$\frac{\partial J(\zeta)}{\partial \zeta} = \sum_s d^\gamma(s)\sum_a \frac{\partial \pi_\zeta(a|s)}{\partial \zeta} Q_\pi(s,a),$$0 and $$\frac{\partial J(\zeta)}{\partial \zeta} = \sum_s d^\gamma(s)\sum_a \frac{\partial \pi_\zeta(a|s)}{\partial \zeta} Q_\pi(s,a),$$1 on all state-option pairs. Evaluation of arbitrary options from one observed action requires stochastic intra-option policies because of the importance-sampling correction. The method is built around call-and-return execution and Markov options, and it does not eliminate all option-discovery pathologies even though it mitigates some of them (Klissarov et al., 2021).

6. Related formulations, contrasts, and scope conditions

Neighboring work shows that option-level REINFORCE can mean different things depending on which component of the hierarchy is treated as the stochastic policy. In Option-GAIL, the hierarchical controller is rewritten into a one-step option model with high-level policy $$\frac{\partial J(\zeta)}{\partial \zeta} = \sum_s d^\gamma(s)\sum_a \frac{\partial \pi_\zeta(a|s)}{\partial \zeta} Q_\pi(s,a),$$2 and low-level policy $$\frac{\partial J(\zeta)}{\partial \zeta} = \sum_s d^\gamma(s)\sum_a \frac{\partial \pi_\zeta(a|s)}{\partial \zeta} Q_\pi(s,a),$$3, so the joint policy factorizes as $$\frac{\partial J(\zeta)}{\partial \zeta} = \sum_s d^\gamma(s)\sum_a \frac{\partial \pi_\zeta(a|s)}{\partial \zeta} Q_\pi(s,a),$$4. The learning objective is option-occupancy matching over tuples $$\frac{\partial J(\zeta)}{\partial \zeta} = \sum_s d^\gamma(s)\sum_a \frac{\partial \pi_\zeta(a|s)}{\partial \zeta} Q_\pi(s,a),$$5, optimized by an EM-style loop in which the E-step infers latent expert options and the M-step updates both levels via adversarial RL. The paper does not write an explicit score-function gradient for $$\frac{\partial J(\zeta)}{\partial \zeta} = \sum_s d^\gamma(s)\sum_a \frac{\partial \pi_\zeta(a|s)}{\partial \zeta} Q_\pi(s,a),$$6, but its reduction to a high-level MDP makes the REINFORCE interpretation direct: the option selector is a stochastic policy over augmented states, trained from discriminator-derived returns (Jing et al., 2021).

Attention Option-Critic stays closer to classical option-critic. It uses the inherited intra-option policy gradient theorem and the inherited termination gradient theorem, while adding option-specific attention masks $$\frac{\partial J(\zeta)}{\partial \zeta} = \sum_s d^\gamma(s)\sum_a \frac{\partial \pi_\zeta(a|s)}{\partial \zeta} Q_\pi(s,a),$$7. The sampled actor update has the explicit log-gradient form

$$\frac{\partial J(\zeta)}{\partial \zeta} = \sum_s d^\gamma(s)\sum_a \frac{\partial \pi_\zeta(a|s)}{\partial \zeta} Q_\pi(s,a),$$8

and the termination update uses the option advantage $$\frac{\partial J(\zeta)}{\partial \zeta} = \sum_s d^\gamma(s)\sum_a \frac{\partial \pi_\zeta(a|s)}{\partial \zeta} Q_\pi(s,a),$$9. However, the policy over options is not learned by REINFORCE; option selection is handled by $$d^\gamma(s)=\sum_{s_0} d(s_0)\sum_{t=0}^{\infty}\gamma^t P^\pi(S_t=s\mid S_0=s_0).$$0-greedy or $$d^\gamma(s)=\sum_{s_0} d(s_0)\sum_{t=0}^{\infty}\gamma^t P^\pi(S_t=s\mid S_0=s_0).$$1-soft improvement over learned option values. This makes AOC an actor-critic variant of option-level policy-gradient learning rather than a direct gradient method over the master policy (Chunduru et al., 2022).

Option Compatible Reward Inverse Reinforcement Learning uses option-level policy-gradient identities in a different way. It writes the intra-option gradient and the termination gradient as first-order optimality conditions, then turns those conditions into linear constraints defining a compatible $$d^\gamma(s)=\sum_{s_0} d(s_0)\sum_{t=0}^{\infty}\gamma^t P^\pi(S_t=s\mid S_0=s_0).$$2-feature space and, through reward shaping, a reward-feature space. It also gives a trajectory-likelihood score-function identity over option trajectories and uses second-order conditions to choose a reward. Here, option-level REINFORCE is not the optimization algorithm being deployed; it is the mathematical substrate used to characterize which reward functions are compatible with demonstrated hierarchical behavior (Hwang et al., 2019).

A final contrast comes from the finite-horizon SMDP analysis of high-level and low-level option learning. “A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning” does not derive policy-gradient estimators at all. It treats the high level as an FH-SMDP with cumulative option reward $$d^\gamma(s)=\sum_{s_0} d(s_0)\sum_{t=0}^{\infty}\gamma^t P^\pi(S_t=s\mid S_0=s_0).$$3 and random duration, alternates Options-UCBVI at the high level with UCBVI for low-level option MDPs, and proves regret bounds $$d^\gamma(s)=\sum_{s_0} d(s_0)\sum_{t=0}^{\infty}\gamma^t P^\pi(S_t=s\mid S_0=s_0).$$4 for high-level learning with fixed low-level policies and $$d^\gamma(s)=\sum_{s_0} d(s_0)\sum_{t=0}^{\infty}\gamma^t P^\pi(S_t=s\mid S_0=s_0).$$5 for joint alternating learning. Its relevance to option-level REINFORCE is conceptual rather than algorithmic: it makes explicit that the high-level decision process over options is an SMDP with cumulative rewards and stochastic durations, which is precisely the semantics a true REINFORCE-over-options derivation would need to respect (Drappo et al., 2024).

Taken together, these results delimit the meaning of the topic. In the narrowest current sense, option-level REINFORCE is best represented by expected intra-option policy-gradient learning over all call-and-return-consistent options, as in MOC (Klissarov et al., 2021). In a broader sense, it includes any score-function or actor-critic construction in which the stochastic object of interest is an option-conditioned policy, a termination function, or a high-level option selector. What it does not denote, at least in the cited work, is a single universally adopted estimator over $$d^\gamma(s)=\sum_{s_0} d(s_0)\sum_{t=0}^{\infty}\gamma^t P^\pi(S_t=s\mid S_0=s_0).$$6.