Achieving sqrt-regret for low V-type Bellman Eluder dimension

Establish whether there exists a reinforcement learning algorithm that, under the assumption that the V-type Bellman Eluder (BE) dimension of the value-function class is finite, achieves O(√K) cumulative regret over K episodes; specifically, determine if techniques for low V-type Bellman rank can be adapted to the low V-type BE dimension setting to obtain √K-regret guarantees.

Background

The paper introduces Bellman Eluder (BE) dimension as a general complexity measure for reinforcement learning with function approximation, with Q-type and V-type variants. The authors provide both regret and sample-complexity guarantees for low Q-type BE dimension, but for the V-type case they only obtain PAC-style sample-complexity results due to the need for uniform random action selection in their V-type algorithms.

In contrast, prior work (Dong et al., 2020) obtained √T-regret for problems with low V-type Bellman rank. This raises the question of whether similar regret guarantees can be established for the broader class characterized by low V-type BE dimension.