OLPA: MDP Planning & PUCS Bandits

• OLPA is an acronym that denotes two distinct approaches: open-loop optimistic planning in discounted MDPs and a probing-augmented, stochastic combinatorial bandit algorithm in PUCS.
• In the MDP planning context, OLPA (via OLOP/KL-OLOP) organizes limited simulator calls into episodes and computes optimistic upper-confidence bounds to minimize simple regret with fixed action sequences.
• In the PUCS framework, OLPA sequentially probes arms to gather side information and assigns plays under resource uncertainty, achieving ζ-approximation regret guarantees with efficient online learning.

OLPA is an acronym used for two distinct algorithmic constructs in the arXiv literature. In one usage, it denotes Open-Loop Optimistic Planning, a planning approach for discounted Markov Decision Processes (MDPs) with access only to a generative model and with policies restricted to fixed action sequences; in that line of work, the central algorithms are OLOP and its KL-based refinement KL-OLOP. In another usage, OLPA denotes a stochastic combinatorial bandit algorithm for probing-augmented user-centric selection (PUCS), where a learner first probes a subset of arms to acquire side information and then assigns plays under resource uncertainty (Leurent et al., 2019, Xu et al., 27 Jul 2025). The shared acronym masks a substantive divergence in modeling assumptions, feedback structure, and regret notions.

1. Terminological scope and disambiguation

In the planning literature, the designation OLPA is used to cover “Open-Loop Optimistic Planning (OLOP/KL-OLOP),” with emphasis on online planning under simulator-call budget constraints. The central problem is to recommend a single root action after spending at most $n$ calls to a simulator, while reasoning over open-loop action sequences of finite horizon $L$ (Leurent et al., 2019).

In the PUCS literature, OLPA is the name of a specific online-learning algorithm. The setting is sequential decision-making with information acquisition: in each round, a learner probes at most $I$ arms, observes side information for those probed arms, and then assigns $K$ plays to $M$ arms. The objective is a $\zeta$-approximation regret criterion with $\zeta=(e-1)/(2e-1)$ (Xu et al., 27 Jul 2025).

This suggests that OLPA is not a uniquely identifying acronym across research areas. A plausible implication is that, in technical writing, the expansion should be made explicit on first use, because the planning and PUCS usages refer to different optimization problems, different observables, and different notions of performance.

2. OLPA as open-loop optimistic planning in discounted MDPs

The planning formulation considers a discounted MDP with state space $S$, action space $A$ with $|A|=K$, and discount $L$0. The transition and reward kernels are unknown; only a generative model is available, so that each query at $L$1 returns a next state $L$2 and reward $L$3. The available budget is $L$4: at most $L$5 simulator calls to select a single root action (Leurent et al., 2019).

Open-loop policies are fixed sequences of actions of some finite horizon $L$6, chosen without observing intermediate states. The objective is to recommend a first action minimizing the simple regret

$L$7

where $L$8 is the optimal $L$9-discounted return from the root state $I$0, and $I$1 is the value of the recommended first action.

Within this formulation, OLOP organizes the simulator budget into $I$2 episodes of length $I$3, where

$I$4

It builds a complete $I$5-ary tree of depth $I$6. Each node $I$7, for $I$8, records

$I$9

The practical significance of the open-loop restriction is stated directly in the recommendations: in high-dimensional or continuous state spaces where open-loop sequences suffice, such as highway driving, OLPA can give strong anytime guarantees with small code complexity. This does not eliminate the distinction between open-loop and closed-loop control; rather, it identifies a regime in which fixed action sequences are operationally adequate.

3. OLOP, KL-OLOP, and regret guarantees

At episode $K$0, OLOP computes for every node $K$1 an optimistic upper-confidence bound on the mean reward at that node,

$K$2

It then defines the optimistic value of a full sequence $K$3 of length $K$4 by

$K$5

For each leaf $K$6, it defines the $K$7-value

$K$8

selects the leaf maximizing $K$9, rolls it out in episode $M$0, and after $M$1 episodes recommends the first action of the most-visited leaf (Leurent et al., 2019).

The stated regret guarantee depends on a branching-complexity quantity $M$2 that measures the branching complexity of near-optimal prefixes. For any $M$3, the simple regret of OLOP satisfies

$M$4

KL-OLOP modifies the upper-confidence construction. The motivating observation is empirical: OLOP’s Hoeffding-based $M$5 can exceed $M$6 for small $M$7, which breaks monotonicity of $M$8 along prefixes and causes uniform exploration. KL-OLOP replaces the quadratic Chernoff-Hoeffding divergence by the Bernoulli Kullback-Leibler divergence on $M$9,

$\zeta$0

and sets

$\zeta$1

with

$\zeta$2

In practice, $\zeta$3 is found by a few Newton iterations. Because $\zeta$4, the $\zeta$5-value coincides with $\zeta$6: $\zeta$7

Theoretical analysis is unchanged in order terms: exactly the same regret analysis as OLOP goes through, yielding the identical $\zeta$8 bounds above. The distinction is therefore primarily practical rather than asymptotic: KL-OLOP inherits OLOP’s sample-complexity guarantees while tightening the UCB construction.

4. Efficient implementation, empirical behavior, and practical use in planning

A direct implementation is computationally prohibitive. Naïvely recomputing $\zeta$9 for all $\zeta=(e-1)/(2e-1)$0 nodes is exponential. The proposed implementation maintains only the explored subtree $\zeta=(e-1)/(2e-1)$1 after $\zeta=(e-1)/(2e-1)$2 episodes: $\zeta=(e-1)/(2e-1)$3 with leaf set $\zeta=(e-1)/(2e-1)$4. At each episode, one updates $\zeta=(e-1)/(2e-1)$5 only for $\zeta=(e-1)/(2e-1)$6, whose size is $\zeta=(e-1)/(2e-1)$7, and selects $\zeta=(e-1)/(2e-1)$8 maximizing $\zeta=(e-1)/(2e-1)$9. When rolling out $S$0, any newly encountered prefix $S$1 is added along with its $S$2 children to $S$3 (Leurent et al., 2019).

The resulting complexity statements are explicit. For naïve OLOP/KL-OLOP, time and memory are $S$4. For the lazy variant, time is $S$5 and memory is $S$6. Since $S$7 for $S$8, the lazy implementation is polynomial.

The empirical findings are equally explicit. On benchmarks described as highway driving, mini-grid world, and stochastic rewards, OLOP is overly conservative in low-budget regimes and behaves roughly like uniform sampling until very large budgets. KL-OLOP achieves the same final regret but converges one order of magnitude faster in sample budget. An additional “aggressive” variant with

$S$9

can help in near-deterministic settings at the risk of occasional over-commitment in high noise.

The paper’s practical recommendations are correspondingly concrete: use KL-OLOP by default with $A$0; implement lazy tree expansion to obtain polynomial time and memory; tune the aggressive variant if the environment is almost deterministic and faster early gains are needed; and choose

$A$1

with $A$2 as the largest integer such that $A$3.

5. OLPA as a stochastic combinatorial bandit for PUCS

In the PUCS framework, there are $A$4 arms indexed by $A$5 and $A$6 plays indexed by $A$7, over rounds $A$8. For each arm $A$9, the number of available resource units $|A|=K$0 is drawn i.i.d. from an unknown distribution with mass function $|A|=K$1. If play $|A|=K$2 is assigned to arm $|A|=K$3 and obtains a resource unit, it draws $|A|=K$4, i.i.d. from an unknown distribution $|A|=K$5 with mean $|A|=K$6 (Xu et al., 27 Jul 2025).

In each round, after probing, the learner chooses an assignment $|A|=K$7, where the sets $|A|=K$8 form a partition of $|A|=K$9. If arm $L$00 has $L$01 resource units and is assigned $L$02 plays, then the reward from $L$03 is

$L$04

when $L$05 is probed, and

$L$06

when it is unprobed, in expectation.

Probing is explicit and costly. In each round, the learner chooses $L$07, with $L$08, to probe. For each $L$09, it observes $L$10. Probing overhead is modeled by a loss factor $L$11, nondecreasing, with $L$12. The total reward is

$L$13

The offline objective is to maximize

$L$14

and seek a $L$15-approximation $L$16 such that $L$17, where $L$18. In the online case, the $L$19-approximation regret over $L$20 rounds is

$L$21

with the target of sublinear regret $L$22.

OLPA addresses this online problem. At time $L$23, for each $L$24, it maintains the empirical mean $L$25, empirical CDF $L$26, empirical mass $L$27, and the UCB radius

$L$28

Its round structure has two phases. In Phase I, it sets $L$29 OfflineGreedyProbing$L$30, probes each $L$31, and updates estimates. In Phase II, it replaces $L$32 by $L$33 for unprobed $L$34, solves

$L$35

by maximum-weighted matching with optimistic estimates, executes $L$36, observes rewards for plays that get resources, and updates estimates again. The paper notes that $L$37 ensures high-probability concentration uniformly in $L$38.

6. Regret analysis and empirical evaluation in PUCS

The regret analysis begins from the per-round comparison

$L$39

where $L$40 is the offline greedy probing on true parameters, and the $L$41 guarantee gives $L$42 (Xu et al., 27 Jul 2025).

The decomposition isolates two sources of statistical error. First, error from estimating $L$43 contributes an $L$44 term via the DKW inequality. Second, error from estimating $L$45 contributes $L$46, which sums to $L$47. The key lemmas are stated as follows: resource-mass estimation satisfies

$L$48

deviation in total reward due to $L$49 is bounded by

$L$50

the mean-reward UCB obeys

$L$51

and the deviation from using $L$52 instead of the true $L$53 is at most

$L$54

Combining these ingredients and choosing $L$55, Theorem 2 gives

$L$56

The paper also proves a lower bound $L$57 in expectation, by reduction to a two-arm Bernoulli test with gap $L$58 and standard information-theoretic arguments using Le Cam and Pinsker. The upper bound is therefore tight up to logarithmic factors.

The experimental evaluation uses two real-world datasets: NYYellowTaxi 2016, consisting of all yellow-taxi pickups in Manhattan from 3/22–3/31/2016, and Chicago Taxi Trips 2016, a random subset from 1/9–9/29/2016. Latitude and longitude are discretized into a $L$59 grid, each cell becomes an arm, passenger pickups of size $L$60 are counted and normalized into $L$61, $L$62 vehicle locations are pre-sampled and fixed, and Manhattan distances to pickups are normalized to $L$63. Two reward models are used: Bernoulli rewards $L$64, and a general discrete model with support $L$65 and probabilities from the empirical distance distribution.

The baselines are Non-Probing (OnLinActPrf), RR (Random-Probing + Random assignment), and GR (Greedy-Probing + Random assignment). Cumulative regret is the reported metric. On Chicago data, in setting $L$66 with $L$67, $L$68, and Bernoulli rewards, regret at $L$69 is $L$70 for OLPA, $L$71 for GR, $L$72 for NonP, and $L$73 for RR. Figures 2(a–d) on NYYellowTaxi confirm that OLPA consistently attains the lowest cumulative regret, with gains more pronounced in larger $L$74 or general reward settings. The interpretation given is that probing yields a substantial constant-factor reduction in regret by focusing plays on better arms more quickly, and that the two-phase OLPA policy effectively balances information acquisition and exploitation.

7. Conceptual comparison of the two OLPA usages

The two usages of OLPA are united by optimism under uncertainty and by explicit budget constraints, but they optimize different objects. In open-loop planning, the decision variable is a sequence of actions of horizon $L$75, and the resource budget is a simulator-call budget $L$76. In PUCS, the decision variables are a probing set $L$77 and an assignment $L$78, and the resource constraints arise from probe budget $L$79, probing overhead $L$80, and random arm capacities $L$81 (Leurent et al., 2019, Xu et al., 27 Jul 2025).

Their regret notions are correspondingly different. The planning line uses simple regret,

$L$82

after spending a fixed planning budget to recommend one root action. The PUCS line uses $L$83-approximation regret accumulated over $L$84 rounds,

$L$85

The first is a one-shot recommendation problem under discounted returns; the second is an online-learning problem with repeated interaction, side information, and assignment structure.

A further distinction lies in the role of optimism. OLOP and KL-OLOP construct optimistic values over prefixes in an action tree, with KL-OLOP specifically enforcing $L$86 through Bernoulli KL bounds. PUCS-OLPA uses optimistic mean-reward estimates $L$87 for unprobed arms and combines them with empirical resource distributions and greedy probing. This suggests a family resemblance at the level of algorithmic philosophy, but not at the level of state, action, or feedback semantics.

For readers encountering OLPA in citations or implementation repositories, the central disambiguator is therefore the surrounding problem statement: discounted MDP planning with a generative model points to Open-Loop Optimistic Planning, whereas probing, side information, assignment, and $L$88-approximation regret point to the PUCS algorithm named OLPA.