Online learning and exploitation of qualitative transition structure

Investigate how qualitative structural information about complex stochastic dynamics, including the relevant components of Bather’s unique hierarchical decomposition, can be learned through exploration and exploited for control without first learning the entire transition graph.

Background

The proposed reinforcement-learning algorithms assume that the MDP’s transition graph is known, even though transition probabilities and rewards are learned. The conclusion identifies removal of this assumption as an important direction and suggests dynamically refining an estimated Bather decomposition as new transitions are observed. The unresolved issue is how to discover and maintain the relevant qualitative structure online and use it for control while learning through exploration.

References

Rather than first learning the entire transition graph, it would be more interesting to discover and maintain the relevant structure online. Since Bather's UHD depends only on which transitions are possible, an estimated decomposition could be dynamically refined as new transitions are observed. This also raises broader questions about how qualitative structural information about complex stochastic dynamics can be learned through exploration and exploited for control, separately from learning quantitative transition probabilities. We plan to investigate these questions in future work.

— Average-Reward Reinforcement Learning for Multichain MDPs: A Hierarchical Decomposition Approach  (2610.10326 - Yu et al., 7 Oct 2026) in Section 6, Concluding remarks