Scale-invariant additive error: sufficiency and regimes

Ascertain whether the scale-invariant additive error condition |V̂(x,y_{1:h})−V*(x,y_{1:h})| ≤ V·(x) suffices for efficient test-time alignment when the mean outcome-level reward (x) is unknown; determine whether an average-case variant of this scale-invariant additive condition is sufficient; and characterize regimes of the parameter V in which the condition enables improvements over ActionLevelRS.

Background

The paper proposes a scale-invariant additive error assumption that bounds the discrepancy between the approximate value function and the true value function proportionally to the prompt-specific expected reward (x). When (x) is known, the authors show that this condition can imply the average-case multiplicative bounds used in their analysis by modifying the approximate value function.

However, the sufficiency of this condition when (x) is unknown, the viability of an average-case formulation of the condition, and the precise regimes of V in which it would improve upon the guarantees of ActionLevelRS are left unresolved, motivating a formal investigation of these assumptions and their algorithmic consequences.