---
title: 'Platypoos: Scale-Free Planning Alg.'
url: https://www.emergentmind.com/topics/platypoos
type: topic
---

# Platypoos: Scale-Free Planning Alg.

Searching arXiv for the primary paper and closely related prior work to ground the article.
arXiv search: 2604.18312 Platypoos OLOP StroquOOL SequOOL OPD
Platypoos is a scale-free planning algorithm for deterministic-dynamics, discounted-reward problems with a fixed simulation budget when rewards are noisy and the learner does not know the reward range or noise range in advance. It is introduced in "Scale-free adaptive planning for deterministic dynamics & discounted rewards" [2604.18312] as an adaptive replacement for OLOP in the regime where one has access to a generative model and seeks the best first action from a current root state rather than a globally optimal policy. Its central technical feature is simultaneous adaptation to unknown reward and noise scales and to latent smoothness of the value function, while preserving guarantees comparable to or better than prior planning methods and yielding markedly faster rates in favorable low-noise or noiseless regimes [2604.18312].

## 1. Problem class and formal objective

Platypoos is defined for an infinite-horizon discounted MDP with state space \(X\), finite action space \(A\) with branching factor \(K = |A|\), deterministic dynamics,
\[
x_{t+1} = f(x_t,a_t),
\]
and stochastic rewards of the form
\[
r_t = r(x_t,a_t) + \varepsilon_t,
\]
where the noise is independent across rounds and bounded [2604.18312].

Two reward models are considered. In the deterministic-reward case,
\[
\varepsilon_t = 0, \qquad r_t = r(x_t,a_t).
\]
In the stochastic-reward case, the observed reward satisfies
\[
\mathbb{E}[r_t \mid x_t] = r(x_t,a_t), \qquad |r_t - r(x_t,a_t)| \le b,
\]
where the unknown quantity \(b \in \mathbb{R}_+\) is the reward-noise range. Rewards are assumed bounded in \([0,R_{\max}]\), and the discount factor
\[
0 \le \gamma < 1
\]
is known [2604.18312].

The learner has access to a generative model and a total budget of \(n\) node openings or evaluations. The objective is to recommend the best next action from a given root state \(x\), minimizing simple regret rather than cumulative regret. For a policy \(\pi: X \to A\), the discounted value is
\[
V^\pi(x) \triangleq \sum_{t\ge 0} \gamma^t r(x_t,\pi(x_t)),
\]
with \(x_0=x\) and \(x_{t+1}=f(x_t,\pi(x_t))\), and
\[
Q^\pi(x,a) \triangleq r(x,a) + \gamma V^\pi(f(x,a)).
\]
The optimal quantities are
\[
V^\star(x) \triangleq \sup_\pi V^\pi(x), \qquad Q^\star(x,a) \triangleq \sup_\pi Q^\pi(x,a),
\]
with Bellman equations
\[
V^\star(x) = \max_{a\in A} \Bigl(r(x,a) + \gamma V^\star(f(x,a))\Bigr),
\]
\[
Q^\star(x,a) = r(x,a) + \gamma \max_{b\in A} Q^\star(f(x,a),b).
\]
If the algorithm recommends action \(a(n)\) after budget \(n\), the simple regret is
\[
r_n \triangleq \max_{a\in A} Q^\star(x,a) - Q^\star(x,a(n)),
\]
so the formal target is
\[
\min_{a(n)} \; r_n.
\]

This formulation places Platypoos squarely in fixed-budget planning with deterministic transitions and stochastic rewards. A plausible implication is that its guarantees are tailored to simulator-based decision support rather than online reinforcement learning in the cumulative-regret sense.

## 2. Tree-based representation and discounted smoothness

The planning problem is rewritten as optimization over the infinite tree of action sequences. For an infinite sequence \(a=(a_0,a_1,\dots)\in A^\infty\), its value is
\[
v(a) \triangleq \sum_{t\ge 0} \gamma^t r(x_t,a_t), \qquad x_{t+1}=f(x_t,a_t), \; x_0=x.
\]
For a finite sequence \(a\in A^\bullet\),
\[
v(a) \triangleq \sup_{a'\in A^\infty} v(aa'),
\]
namely the value of the best continuation after prefix \(a\). The optimal root value is
\[
v^\star \triangleq v(\emptyset)=V^\star(x).
\]

For a node \(a\) of depth \(h(a)\), the partial discounted return is
\[
u(a) \triangleq \sum_{t=0}^{h(a)-1} \gamma^t r(x_t,a_t),
\]
and the trivial upper bound is
\[
b(a) \triangleq u(a) + \frac{\gamma^{h(a)}R_{\max}}{1-\gamma}.
\]
Hence
\[
u(a)\le v(a)\le b(a).
\]
This discounted structure induces a baseline smoothness: even without extra regularity assumptions, the tail beyond depth \(h\) is controlled by \(\gamma^h/(1-\gamma)\) [2604.18312].

The key structural smoothness proposition states that there exist \(\nu \in (0,R_{\max}/(1-\gamma)]\) and \(\rho\in(0,1]\) such that for all depths \(h\ge 0\) and all nodes \(a\in A^h\),
\[
u(a) \ge v(a) - \nu \rho^h.
\]
This always holds with the trivial choice
\[
\nu = \frac{R_{\max}}{1-\gamma}, \qquad \rho=\gamma,
\]
but some problems may exhibit extra smoothness through smaller \(\nu\) or \(\rho<\gamma\) [2604.18312]. Platypoos does not require \((\nu,\rho)\) as input and is analyzed as adapting to them.

To quantify the effective size of the near-optimal set, the paper defines the near-optimal branching factor. For any \(\nu>0\) and \(\rho\in(0,1)\),
\[
\kappa^v(\nu,\rho) \triangleq \inf\left\{ \kappa\ge 1 : \exists C>1,\ \forall h\ge 0,\ \mathcal N_h^v(3\nu\rho^h)\le C\kappa^h \right\},
\]
where \(\mathcal N_h^v(\varepsilon)\) counts nodes \(a\in A^h\) such that
\[
v(a)\ge v^\star-\varepsilon.
\]
An analogous \(\kappa^u(\nu,\rho)\) is introduced using \(u(a)\) instead of \(v(a)\), and Platypoos’ analysis is phrased in terms of \(\kappa^u(\nu,\rho)\) [2604.18312].

The favorable case \(\kappa=1\) corresponds to essentially a unique near-optimal branch. This suggests that Platypoos’ strongest rates are expected when the planning tree rapidly narrows around a single optimal continuation.

## 3. Algorithmic design and scale-free adaptation

Platypoos is a budget allocation strategy over depths and sample counts. It combines open-loop planning over the action tree, as in OLOP, with a scale-free optimization strategy inspired by StroquOOL [2604.18312]. The core issue is that the appropriate number of samples per node depends on unknown noise level and unknown gap structure: too few samples yield unreliable estimates, while too many prevent deep exploration. Platypoos resolves this by simultaneously trying many sample levels.

At each depth \(h\), and for each sampling level indexed by \(p\), the algorithm opens some of the best currently estimated nodes using roughly \(2^p\)-scaled samples. Large \(p\) assigns many samples per node and explores fewer nodes; small \(p\) allocates fewer samples per node and explores more broadly. Rather than relying on user-specified confidence radii, Platypoos implicitly runs many candidate exploration schedules in parallel and then performs a cross-validation or final evaluation phase among candidate branches corresponding to different \(p\)-values [2604.18312].

The paper characterizes the method as scale-free because it does not require the reward range \(R_{\max}\), the noise bound \(b\), the smoothness parameters \(\nu,\rho\), or the branching factor \(\kappa\). It does require the total budget \(n\), the action set \(A\), the discount factor \(\gamma\), and access to a generative model [2604.18312]. This definition of scale-freeness is precise: the algorithm avoids upper confidence bounds that explicitly plug in reward or noise scales.

The reconstructed main procedure initializes
\[
h_{\max} \gets \left\lfloor \frac{n}{2(\log_2 n + 1)^2}\right\rfloor,
\qquad
p_{\max} \gets \left\lfloor \log_2(h_{\max}) \right\rfloor,
\]
then enters an exploration phase over depths and sample levels, followed by a cross-validation phase over candidate paths, and finally outputs
\[
a^n \gets \arg\max_{\{a^p:\ p\in[0:p_{\max}]\}} \hat u(a^p)
\]
[2604.18312]. The source text is described as partially corrupted, but the supplied formulas retain the intended structure of the schedule.

A common misconception would be to treat Platypoos as merely an untuned UCB variant. The paper’s description is narrower and more specific: it is not based on a single confidence scale, but on a family of sample-allocation scales combined with final validation.

## 4. Technical mechanism: confidence control, opening conditions, and discount dependence

The analysis is organized around a high-probability confidence event \(\xi\) controlling empirical prefix-value estimates. If \(\mathcal C\) is the set of sequences evaluated by Platypoos, then
\[
\xi \triangleq \left\{ \forall a\in\mathcal C,\ \forall h\in[2:h(a)],\ p\in[0:p_{\max}]: \text{ if } T_{a_{[h]}} \ge \left\lceil (h-1)2^p\gamma^{2(h-1)}\right\rceil, \text{ then } \left| \hat u(a)-u(a)\right| \le b\sqrt{\frac{\log(4n/\delta)}{2^{p+1}}} \right\},
\]
and
\[
\mathbb P(\xi)\ge 1-\delta.
\]
The deviation radius therefore scales as
\[
b\sqrt{\frac{\log(4n/\delta)}{2^{p+1}}},
\]
so larger \(p\) yields tighter confidence through heavier sampling [2604.18312].

The key technical quantity \(\perp_{h,p}\) denotes the depth of the deepest opened node on an optimal branch under sample level \(p\). A principal lemma states that, on event \(\xi\), for
\[
h \in [h_{\max}], \qquad p \in \left[0:\left\lfloor \log_2\!\Bigl(\frac{h_{\max}}{h^2\gamma^{2h}}\Bigr)\right\rfloor\right],
\]
one has \(\perp_{h,p}=h\) if two conditions hold simultaneously. The first is discrimination by sampling,
\[
b\sqrt{\frac{\log(4n/\delta)}{2^{p+1}}} \le \nu\rho^h,
\]
meaning estimation error is small enough to resolve \(h\)-depth near-optimality. The second is sufficient opening relative to the near-optimal set, expressed generically as
\[
\forall h'\in[h],\quad
\frac{h_{\max}}{h' \left\lceil h'2^p\gamma^{2h'}\right\rceil}
\ge C(\nu,\rho)\kappa^{h'}.
\]
The paper also gives more detailed case splits depending on whether \(h2^p\gamma^{2h}\le 1\) or \(h2^p\gamma^{2h}\ge 1\), and whether \(\gamma^2\kappa\le 1\) or \(\gamma^2\kappa\ge 1\) [2604.18312].

Discounting enters the algorithm and its proof in three distinct ways. First, it provides the baseline smoothness relation
\[
u(a) \ge v(a) - \frac{\gamma^{h(a)}R_{\max}}{1-\gamma}.
\]
Second, the sampling schedule uses terms involving \(\gamma^{2h}\), reflecting the smaller importance of deeper rewards under discounting. Third, validation cost and confidence accumulation involve
\[
\frac{1}{(1-\gamma^2)^2},
\]
which arises from controlling discounted sums of bounded deviations through \(\sum_t \gamma^{2t}\) [2604.18312].

These ingredients clarify that Platypoos is neither depth-uniform nor noise-uniform. Its schedule is explicitly structured around the interaction among depth, discounting, near-optimal branching, and empirical uncertainty.

## 5. Performance guarantees and regime structure

The guarantees separate deterministic and stochastic reward settings. For noiseless rewards, a simpler SequOOL-style planning method yields the following simple-regret rates for a planning problem with associated \((\nu,\rho)\) and branching factor \(\kappa=\kappa^u(\nu,\rho)\):
- if \(\kappa=1\),
  \[
  r_n \le \nu \rho^{\frac{1}{C}\left\lfloor\frac{n}{\log n}\right\rfloor};
  \]
- if \(\kappa>1\),
  \[
  r_n
  = O\!\left(
  \nu
  \left(
  \frac{\log n}{C}\left\lfloor\frac{n}{\log n}\right\rfloor
  \right)^{-\frac{\log(1/\rho)}{\log \kappa}}
  \right).
  \]
Under a reset constraint, the exponential case weakens to \(\rho^{\sqrt n}\), while the polynomial case remains essentially the same [2604.18312].

For stochastic rewards, the analysis introduces
\[
g_{n,b}^{\delta,\gamma}\triangleq \log\!\left(\frac{n^{3/2}}{b(1-\delta)}\right)
\]
up to notation corruption in the source, and defines special depths \(h_1,h_2\) via balances between noise discrimination and near-optimal coverage:
\[
h_1 = \frac{1}{\log(\gamma^2\kappa/\rho^2)} \log\!\left(\frac{n_1}{\log n_1}\right)+o(1),
\qquad
n_1 \triangleq \frac{\nu^2\log(\gamma^2\kappa/\rho^2)}{C\,b^2\,g_{n,b}^{\delta,\gamma}},
\]
\[
h_2 = \frac{1}{\log(1/\rho^2)} \log\!\left(\frac{n_2}{\log n_2}\right)+o(1),
\qquad
n_2 \triangleq \frac{\nu^2\log(1/\rho^2)}{C\,b^2\,g_{n,b}^{\delta,\gamma}}.
\]
These depths characterize how deeply Platypoos can certify the optimal branch in different regimes [2604.18312].

In the high-noise regime, if the noise is large enough to satisfy the paper’s two high-noise conditions, Platypoos satisfies
\[
r_n = \begin{cases}
\tilde O\!\left(\left(\frac{n}{b^2}\right)^{-1/2}\right), & \text{if }\gamma^2\kappa\le 1,\\[1ex]
\tilde O\!\left(\left(\frac{n}{b^2}\right)^{-\frac{\log(1/\rho)}{\log(\gamma^2\kappa/\rho^2)}}\right), & \text{if }\gamma^2\kappa>1.
\end{cases}
\]
In the low-noise regime, if the paper’s low-noise conditions hold, then
\[
r_n = \begin{cases}
O(\nu\rho^n), & \text{if }\kappa=1,\\[1ex]
O\!\left( \nu\left(\frac{n}{b^2}\right)^{-\frac{\log(1/\rho)}{\log(\kappa)}} \right), & \text{if }\kappa>1.
\end{cases}
\]
The \(\kappa=1\) low-noise case is especially notable because the regret decays exponentially [2604.18312].

The paper’s summary also highlights an intermediate deterministic-like regime for \(\gamma^2\kappa\le 1\), where the rate can become
\[
\nu \rho^{\sqrt n}
\]
depending on noise and reset assumptions. This suggests that the boundary between stochastic and deterministic behavior is governed not only by \(b\), but also by the geometry of the near-optimal tree and the discount factor.

## 6. Relation to prior methods, practical use, and caveats

The main benchmark is OLOP. OLOP uses upper confidence bounds of the form
\[
\hat u(a) = \tilde O\!\left( u(a) + \sum_{h=0}^{h(a)-1}\gamma^h \hat b \sqrt{\frac{1}{T_{a_{[h]}}} + \frac{\gamma^{h(a)}}{1-\gamma} \right),
\]
where \(\hat b\) is a user-provided bound on noise. According to the supplied account, this works if \(\hat b=b\), but if \(\hat b\) is too large then exploration is overly conservative, while if \(\hat b\) is too small then the confidence intervals are invalid [2604.18312]. Platypoos avoids this dependence by not building a UCB from \(b\); instead it runs a family of sample-allocation scales in parallel and validates them afterward.

The paper emphasizes four improvements over OLOP: adaptation to unknown noise, adaptation to unknown reward scale, adaptation to extra smoothness through \(\rho\) rather than the baseline \(\gamma\), and low-noise or noiseless speedups. In particular, when noise is absent or tiny, Platypoos can recover deterministic planning rates and, for \(\kappa=1\), exponentially decaying regret, whereas OLOP remains tied to a fixed confidence scale [2604.18312].

Platypoos is presented as practically relevant when a simulator or generative model is available, the state space is too large for full dynamic programming, only a good next action is needed under a fixed budget, rewards may be noisy, and the reward range, variance or noise range, or effective smoothness are not known in advance. The experiments described in the paper use a simple deterministic-dynamics MDP plus additive bounded noise, and Platypoos reportedly systematically outperforms OLOP while being insensitive to incorrect user inputs that can severely degrade OLOP [2604.18312].

Several limitations are explicit. The theory assumes deterministic transitions; the stochasticity is only in rewards. The noise must be bounded because the analysis uses Hoeffding- or Azuma-style confidence bounds. The setting is fixed-budget simple regret rather than cumulative-regret reinforcement learning. The guarantees depend on multiple regime splits involving \(\gamma^2\kappa\), high versus low noise, and reset assumptions. Reset constraints weaken exponential rates, with
\[
\rho^n \to \rho^{\sqrt n}, \qquad \rho^{\sqrt n}\to \rho^{n^{1/3}}.
\]
The appendix also notes that floor and ceiling operations in the practical schedule may underuse budget unless tuned carefully [2604.18312].

One point in the supplied material requires careful reading. The abstract states that the paper establishes a matching lower bound, but the detailed summary also states that the supplied text does not contain a separately stated formal lower-bound theorem in the same explicit style as the upper bounds. In that summary, the matching comparison is described primarily relative to existing upper bounds from OLOP and OPD rather than to an explicit minimax lower-bound statement [2604.18312]. This suggests that any interpretation of “matching” should distinguish the abstract’s claim from the level of detail visible in the provided exposition.

Source: https://www.emergentmind.com/topics/platypoos