Hierarchical Importance Sampling Policy Optimization

• The paper introduces HisPO, which leverages a time-conditioned hyper-policy to adapt policy parameters for lifelong, non-stationary reinforcement learning tasks.
• It employs a multiple importance sampling estimator with a differentiable divergence penalty to balance bias, variance, and mitigate overfitting.
• Empirical results show HisPO matches or outperforms baselines in environments like Forex trading and water-reservoir management while reducing catastrophic forgetting.

Searching arXiv for the specified paper and closely related context. {"queries":[{"query":"(Liotet et al., 2021) Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization","limit":5},{"query":"\"Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization\"","limit":5},{"query":"Chandak 2020 Pro-OLS Pro-WLS reinforcement learning continual arXiv","limit":5},{"query":"Al-Shedivat 2018 ONPG arXiv continual reinforcement learning","limit":5}]} Found relevant papers on arXiv, including "Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization" (Liotet et al., 2021). Other related results include work associated with ONPG and continual/non-stationary RL, but the article below is grounded strictly in the provided data and cites the core source (Liotet et al., 2021). Searching specifically for the baseline papers mentioned in the source description so they can be cited precisely. {"queries":[{"query":"Chandak Pro-OLS Pro-WLS arXiv reinforcement learning non-stationary","limit":10},{"query":"Al-Shedivat ONPG arXiv 2018 continual reinforcement learning","limit":10},{"query":"\"Pro-OLS\" reinforcement learning Chandak","limit":10},{"query":"\"ONPG\" reinforcement learning Al-Shedivat","limit":10}]} The targeted search did not return reliable, unambiguous arXiv matches for the baseline acronyms. To avoid introducing potentially incorrect citations, the article cites the primary source (Liotet et al., 2021) and names the baselines exactly as given in the provided data. Hierarchical Importance Sampling Policy Optimization (HisPO) is a framework for lifelong reinforcement learning in settings where the dynamics continually evolve. It is introduced in “Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization” and learns a hyper-policy whose input is time and whose output determines the parameters of the policy to be queried at that time. The method combines a multiple-importance-sampling estimate of future performance with a past-performance term to mitigate catastrophic forgetting, and it embeds a differentiable variance bound as a penalization term to avoid overfitting the collected data. In the implementation-oriented description, HisPO is also denoted POLIS (Liotet et al., 2021).

1. Problem formulation and hierarchical structure

HisPO is formulated in a parameter-based policy-optimization setting. At each discrete decision epoch $t$, or task index $t$, the behavioral policy $\pi_{\theta_t}(a \mid x)$ is determined by a time-conditioned hyper-policy

$\theta_t \sim \nu_\phi(\cdot \mid t),$

where $\phi \in \mathbb{R}^d$ are the hyper-parameters (Liotet et al., 2021). An equivalent representation is a deterministic mapping

$\theta_t = \eta_\phi(t),$

followed by additive exploration noise; in the description, $\nu_\phi$ is taken to be, for example, a Gaussian $N(\eta_\phi(t), \Sigma)$.

The optimization target is to learn $\phi$ so that, at each future time $t$, the induced base policy $t$0 maximizes long-run return under smoothly changing Markov decision process dynamics. This establishes a two-level hierarchy: the upper level produces policy parameters as a function of time, while the lower level is the policy actually executed in the environment.

This construction is specifically tailored to lifelong settings rather than stationary control. The time-conditioned hyper-policy makes temporal variation part of the policy parameterization itself. A plausible implication is that HisPO treats non-stationarity as a first-class modeling assumption rather than as a perturbation around a fixed policy class.

2. Multiple-importance-sampling estimator of future return

Let $t$1 denote the current time, $t$2 the size of the past buffer $t$3, and $t$4 the look-ahead horizon. The target quantity is the $t$5-step-ahead return

$t$6

Because one cannot sample under $t$7 at future times $t$8, HisPO reuses past data

$t$9

through a multiple-importance-sampling estimator with exponentially decaying weights $\pi_{\theta_t}(a \mid x)$0:

$\pi_{\theta_t}(a \mid x)$1

which yields

$\pi_{\theta_t}(a \mid x)$2

Equivalently, each past sample $\pi_{\theta_t}(a \mid x)$3 is identified with an importance weight

$\pi_{\theta_t}(a \mid x)$4

where

$\pi_{\theta_t}(a \mid x)$5

so that

$\pi_{\theta_t}(a \mid x)$6

approximates

$\pi_{\theta_t}(a \mid x)$7

The estimator is designed to reuse old trajectories in a setting where direct sampling from future task distributions is unavailable. The exponential decay term $\pi_{\theta_t}(a \mid x)$8 provides explicit temporal weighting of reused experience. This suggests that the estimator balances recency against sample reuse through $\pi_{\theta_t}(a \mid x)$9, while the look-ahead horizon $\theta_t \sim \nu_\phi(\cdot \mid t),$0 determines how aggressively future performance is extrapolated.

3. Bias, variance, and the differentiable divergence penalty

The description emphasizes that off-policy multiple importance sampling carries high variance when $\theta_t \sim \nu_\phi(\cdot \mid t),$1 diverges from past $\theta_t \sim \nu_\phi(\cdot \mid t),$2. To control this effect, the paper derives a tractable upper bound on

$\theta_t \sim \nu_\phi(\cdot \mid t),$3

$\theta_t \sim \nu_\phi(\cdot \mid t),$4

where

$\theta_t \sim \nu_\phi(\cdot \mid t),$5

$\theta_t \sim \nu_\phi(\cdot \mid t),$6

$\theta_t \sim \nu_\phi(\cdot \mid t),$7

and

$\theta_t \sim \nu_\phi(\cdot \mid t),$8

is the exponential 2-Rényi divergence (Liotet et al., 2021).

In practice, the mixture divergences are not formed exactly. Instead, a variational upper bound is used, for example the two-step “$\theta_t \sim \nu_\phi(\cdot \mid t),$9-first” bound described as Eq. (16), and this bound is fully differentiable in $\phi \in \mathbb{R}^d$0.

The same section of the method also presents a bias bound. Under smoothness Assumptions 3.1–3.2,

$\phi \in \mathbb{R}^d$1

These results formalize the central trade-off in HisPO: future-return estimation is made possible by reweighting past data, but doing so introduces controlled bias and potentially substantial variance. The method addresses this by coupling temporal decay, smoothness assumptions, and a differentiable Rényi-divergence proxy within one optimization framework.

4. Mixed objective and catastrophic-forgetting control

HisPO does not optimize the future-return estimator in isolation. It also tracks the “$\phi \in \mathbb{R}^d$2-step-behind” return on the same buffer:

$\phi \in \mathbb{R}^d$3

The description states that its gradient discourages $\phi \in \mathbb{R}^d$4 from driving denominator terms $\phi \in \mathbb{R}^d$5, which would spuriously inflate $\phi \in \mathbb{R}^d$6.

The combined estimate is

$\phi \in \mathbb{R}^d$7

The final surrogate objective is a lower-confidence bound on the true mixed return:

$\phi \in \mathbb{R}^d$8

with $\phi \in \mathbb{R}^d$9 a tunable exploration/regularization coefficient. In practice, $\theta_t = \eta_\phi(t),$0 is replaced by its differentiable variational bound $\theta_t = \eta_\phi(t),$1 (Liotet et al., 2021).

This objective serves three roles stated explicitly in the source description: it provides a controlled-bias approximation of future return, penalizes policy shifts that blow up off-policy variance, and prevents denominator collapse via the past-return term. The catastrophic-forgetting aspect enters through the inclusion of past performance rather than through a separate replay or consolidation mechanism.

5. End-to-end optimization procedure

The implementation-oriented pseudocode describes HisPO, or POLIS, with the following inputs: buffer-size $\theta_t = \eta_\phi(t),$2, horizon $\theta_t = \eta_\phi(t),$3, decay $\theta_t = \eta_\phi(t),$4, retrain-interval $\theta_t = \eta_\phi(t),$5, inner-epochs $\theta_t = \eta_\phi(t),$6, and regularization parameter $\theta_t = \eta_\phi(t),$7.

At each iteration, the procedure samples $\theta_t = \eta_\phi(t),$8, executes $\theta_t = \eta_\phi(t),$9, observes $\nu_\phi$0, and stores $\nu_\phi$1 in a circular buffer of size $\nu_\phi$2. When $\nu_\phi$3 and $\nu_\phi$4, the method performs $\nu_\phi$5 inner optimization epochs. In each inner epoch it forms the weights $\nu_\phi$6 via the MIS formula, computes $\nu_\phi$7, $\nu_\phi$8, and the divergence-bound term $\nu_\phi$9, evaluates

$N(\eta_\phi(t), \Sigma)$0

and updates

$N(\eta_\phi(t), \Sigma)$1

using, for example, RMSProp or Adam. The loop then increments $N(\eta_\phi(t), \Sigma)$2.

The source description highlights that the surrogate $N(\eta_\phi(t), \Sigma)$3 is fully differentiable with respect to $N(\eta_\phi(t), \Sigma)$4, which permits off-the-shelf gradient optimizers. This places the complexity of the method primarily in estimator construction and regularization design rather than in nonstandard optimization machinery.

6. Theoretical guarantees and empirical behavior

The theoretical guarantees reported for HisPO consist of a bias bound, the variance bound above, and a lower-confidence result derived via Cantelli’s inequality. Specifically, with probability at least $N(\eta_\phi(t), \Sigma)$5,

$N(\eta_\phi(t), \Sigma)$6

The description summarizes the consequence as follows: optimizing $N(\eta_\phi(t), \Sigma)$7 ensures controlled-bias approximation of future return, penalizes policy shifts that blow up off-policy variance, and prevents denominator collapse via the past-return term (Liotet et al., 2021).

The empirical evaluation compares HisPO, denoted POLIS, against three baselines: a stationary hyper-policy with no time-dependence, Pro-OLS and Pro-WLS (Chandak et al. 2020), and ONPG (Al-Shedivat et al. 2018). The environments are Forex trading on EUR–USD historical data from 2009–2020 split into $N(\eta_\phi(t), \Sigma)$8-day sequences, Vasicek-rate trading under smooth non-stationarity, and water-reservoir management with three non-stationary inflow profiles.

Environment	Reported result
Forex-trading (EUR–USD)	HisPO matches or slightly outperforms best baselines in 2013–2016, and closes the gap on 2017–2020 after hyper-parameter re-tuning
Vasicek-rate trading	HisPO substantially outperforms all baselines, with higher mean return and lower variance
Water-reservoir management (“Dam”)	Stationary hyper-policy is already strong; HisPO matches stationary on all profiles, while Pro-OLS/WLS and ONPG degrade

The quantitative highlights given in the description are specific. On the Vasicek task, HisPO achieved mean return approximately $N(\eta_\phi(t), \Sigma)$9 versus $\phi$0 for the stationary hyper-policy and $\phi$1 baseline, with $\phi$2–$\phi$3 lower standard deviation. On the Dam task, HisPO’s average daily cost was within $\phi$4 of stationary, while halving the regret of the other baselines.

These findings also clarify a potential misconception. HisPO is not presented as uniformly superior to every simpler alternative in every environment. In the Dam setting, a stationary hyper-policy is reported to be already strong, and HisPO matches rather than exceeds it. Conversely, under smooth non-stationarity in the Vasicek task, the time-conditioned hyper-policy and MIS-based future-return estimation appear particularly effective. This suggests that the benefit of HisPO is most pronounced when temporal variation is substantial enough to reward explicit time conditioning, yet smooth enough for the smoothness-based bias control to remain meaningful.