Determine whether nonlinear KL-based distance shaping improves learning beyond direct distance guidance

Determine whether the normalized nonlinear distance penalty induced by the exact two-outcome KL proxy improves reinforcement-learning performance beyond direct successor-state distance guidance when observations, coefficient-selection opportunities, and training budgets are matched, and determine whether any performance difference depends on a constant cost incurred until episode termination.

Background

The paper’s practical implementation uses a hand-designed two-outcome proxy derived from the agent’s Euclidean distance to a target, together with a fixed preferred distribution. The resulting KL penalty is therefore a nonlinear distance-based shaping signal rather than a learned, calibrated prediction of goal attainment. The completed MiniGrid comparison contrasts only an unshaped condition with one nonzero shaping coefficient and does not isolate the KL transformation from simpler distance-based alternatives or from a constant per-step cost.

The authors propose a six-group prospective study to address this unresolved mechanism question. The planned comparison includes unshaped reward, constant cost, direct distance shaping, potential-based shaping, the full KL proxy, and a centered KL proxy. It is designed to match information, coefficient-selection opportunities, and training budgets, with the full-KL and direct-distance conditions serving as the primary comparison. The study is explicitly prospective, and no results resolving the question are reported.