Establish whether environment-based reward labeling is preferable at both hierarchy levels

Determine whether using the environment success criterion rather than the stricter exact-state success criterion for low-level reward labeling yields preferable Q-Shaped Options representations and state abstractions without sacrificing downstream performance.

Background

Q-Shaped Options uses different reward-labeling criteria at the two hierarchy levels: the environment success criterion at the high level and a stricter exact-match criterion at the low level. The stricter low-level criterion is intended to support fine-grained control, but the ablations indicate that environment-based labeling reduces the representation’s dependence on fast-moving state dimensions.

Although the reported experiments show no downstream performance loss from environment-based low-level labeling, the authors present its superiority as a hypothesis rather than a resolved conclusion.

References

Because we see no loss in downstream performance (Figure 1) and although we use the stricter success criterion for labelling reward in the low-level decision process across all experiments, we hypothesise that the environment success criterion might be preferable at both levels.

— Q-Shaped Options for Hierarchical Reinforcement Learning  (2610.12135 - Wibault et al., 8 Oct 2026) in Appendix H.5, paragraph beginning “Further discussion on the coherence of state abstractions.”