Scale-free adaptive planning for deterministic dynamics & discounted rewards

Abstract: We address the problem of planning in an environment with deterministic dynamics and stochastic rewards with discounted returns. The optimal value function is not known, nor are the rewards bounded. We propose Platypoos, a simple scale-free planning algorithm that adapts to the unknown scale and smoothness of the reward function. We provide a sample complexity analysis for Platypoos that improves upon prior work and holds simultaneously over a broad range of discount factors and reward scales, without the algorithm knowing them. We also establish a matching lower bound showing our analysis is optimal up to constants.

• The paper introduces a scale-free adaptive planning algorithm that dynamically adapts to unknown reward and noise ranges without pre-calibrated parameters.
• It exploits discount-induced value function smoothness to achieve exponential simple regret decay in deterministic and low-noise settings.
• Empirical results demonstrate that the approach outperforms OLOP by successfully balancing exploration and efficient evaluation in various MDP environments.

Scale-Free Adaptive Planning for Deterministic Dynamics and Discounted Rewards

Problem Setting and Motivation

The paper "Scale-free adaptive planning for deterministic dynamics & discounted rewards" (2604.18312) addresses the problem of planning in environments with deterministic dynamics and stochastic, discounted rewards, under a fixed numerical budget and unknown reward/noise ranges. The primary goal is to robustly recommend the best initial action in a Markov Decision Process (MDP) given only generative access to environment transitions, with the evaluation metric based on simple regret–the difference between value achieved by the recommended action sequence and the optimal sequence starting from the same state.

Existing open-loop planning algorithms, notably OLOP, require prior knowledge of reward/noise range, which is often unavailable or inaccurate in practice and leads to premature failure (if underestimated) or inefficiency (if overestimated). Furthermore, OLOP and similar UCB-based algorithms do not leverage the inherent smoothness introduced by the discount factor in value functions, resulting in suboptimal exploration/exploitation balancing, especially in low-noise regimes. The paper introduces a novel planning algorithm, denoted in the paper as a scale-free adaptive method, that circumvents these limitations by dynamically adapting to reward/noise ranges and exploiting value-function smoothness, without requiring any a priori parameter calibration.

The Scale-Free Planning Algorithm

The algorithm is conceptually inspired by scale-free function optimization strategies such as StroquOOL [bartlett2019simple], but adapts these principles to planning trees induced by deterministic dynamics and discounted rewards. The method sequentially expands promising nodes in the planning tree, allocating an increasing amount of queries at deeper levels, and adaptively pools empirical returns across action-prefixes, without explicit UCBs or explicit knowledge of reward/noise ranges.

Deterministic Reward Case

In deterministic environments, the algorithm proceeds by opening nodes at growing depths with sample budgets inversely proportional to the depth (i.e., more shallow nodes get more evaluations). This permits exponentially deep exploration relative to the numeric budget, leveraging discount-induced smoothness and thereby attaining exponential simple regret decay with respect to budget for low-branching factor problems.

Stochastic Reward Case

In the presence of noise, the approach incorporates multiple (adaptive) evaluations per node, guided by the observed gap between empirical and optimal values. The allocation of evaluations across depths and nodes is governed by a "scale-free" partitioning strategy, so that uncertainty in noisy returns does not preclude efficient planning. The method asymptotically achieves rates on simple regret scaling like $O\left((n/b^2)^{-\alpha}\right)$, where $\alpha$ depends on the value smoothness and tree branching structure, and recovers the deterministic exponential rate as noise vanishes, which OLOP cannot achieve.

Theoretical Analysis and Strong Claims

The paper makes several theoretical contributions:

• The algorithm dynamically adapts to unknown reward and noise ranges, avoiding failures characteristic of OLOP under misspecification.
• It empirically and theoretically achieves much faster learning than classical UCB planning algorithms, especially when noise is low.
• For problems where the value function is particularly smooth (low effective branching factor $\kappa = 1$), the algorithm achieves exponentially fast simple regret decay in the numeric budget (e.g., $r_n = O(\rho^n)$ for some $\rho<1$), outperforming OLOP's $O(n^{-1/2})$ rate.
• The performance guarantees are robust to additional global regularity parameters $(\nu,\rho)$, and further adapt in settings where these are unknown or smaller than default ($\nu<1/(1-\gamma)$, $\rho<\gamma$).

These results are formalized in a series of theorems and empirical upper bounds which specify the simple regret decay rates with respect to the budget and problem structure, both for deterministic and stochastic settings. Notably, the algorithm exhibits scale-free adaptation behavior, achieving optimal rates without any explicit parameter specification.

Empirical Evaluation

The empirical section validates the algorithm's performance against OLOP across varying noise regimes and reward scales, using a simple binary MDP with reward structures engineered to accentuate the benefits of deep exploration and delayed rewards.

(Figure 1)

Figure 1: Average cumulative discounted return for OLOP and the scale-free adaptive algorithm, for varying noise levels ($b=1,10,20,50$).

The results exhibit that the scale-free algorithm systematically outperforms OLOP, both with correctly specified and misspecified reward/noise ranges. OLOP's sensitivity to range misspecification is explicitly illustrated: when the input range is overestimated, its exploration becomes overly conservative and regret decay slows precipitously.

(Figure 2)

Figure 2: Sensitivity of OLOP to $\alpha$0 and input noise $\alpha$1 compared to the scale-free algorithm’s robust adaptation.

Practical and Theoretical Implications

Practically, the proposed scale-free planning mechanism enables robust, efficient planning in deterministic and stochastic discounted MDPs without hardcoded environment-specific parameters. This is especially pertinent for real-world applications (robotics, games, sequential decision-making) where reward/noise scales are ambiguous or variable across tasks.

Theoretically, the analysis clarifies the efficiency gains permitted by exploiting smoothness induced by discounting, and provides a generic bridge between function optimization and planning. The strong exponential regret reduction in deterministic, low-branching scenarios is particularly notable, suggesting that classical planning hardness assumptions may be too pessimistic when discounting and smoothness are present.

Future Directions

This line of work opens several avenues for further study, including:

• Extending scale-free adaptive planning to settings with partial observability, model mismatch, or continuous state/action spaces.
• Analyzing and integrating reset conditions, sample efficiency under practical exploration constraints, and deeper connections to modern deep RL paradigms.
• Investigating the interplay between discount-driven smoothness adaptation and learned value representations in model-based RL, and the potential for combining these planners with function approximation techniques.

Conclusion

The paper advances planning methodology in deterministic and stochastic discounted reward environments by introducing a scale-free adaptive algorithm that robustly and efficiently selects actions under an unknown reward/noise range and budget constraints. The theoretical and empirical results demonstrate substantial improvements in learning speed and robustness compared to OLOP, especially in low-noise and low-branching settings. This advancement underscores the importance of planning strategies that adapt to environmental regularities and paves the way for more principled and efficient approaches in large-scale and uncertain decision domains.