Extend average-case guarantees to large action spaces

Develop an efficient version of Value-Guided Backtracking (VGB) for large action spaces under only the average-case value-function error assumption—namely, for each h, the expectations E_{y_{1:h}∼π(·|x)}[V̂(x,y_{1:h})/V*(x,y_{1:h})] ≤ 1+V and E_{y_{1:h}∼π(·|x)}[V*(x,y_{1:h})/V̂(x,y_{1:h})] ≤ 1+V—without imposing additional uniform bounds. Design and analyze a transition mechanism (for example, rejection sampling with controlled failure) that yields provable coverage or accuracy guarantees comparable to those established for the small-action regime.

Background

In the paper’s analysis of VGB under average-case value-function error (Assumption 4.2), the theoretical guarantees are provided only for the small-action setting, where enumerating actions is tractable. For large action spaces, the proof in the uniform-error case uses rejection sampling with bounded density ratios, a step that does not straightforwardly carry over under average-case error assumptions.

The authors note that with only average-case bounds, it is unclear how to implement transitions efficiently and suggest that more delicate arguments—such as allowing rejection sampling to occasionally fail—might remove the need for extra assumptions. Formalizing such an approach and proving guarantees analogous to the small-action case remains unresolved.