States as goal-directed concepts: an epistemic approach to state-representation learning (2312.02367v2)
Abstract: Our goals fundamentally shape how we experience the world. For example, when we are hungry, we tend to view objects in our environment according to whether or not they are edible (or tasty). Alternatively, when we are cold, we may view the very same objects according to their ability to produce heat. Computational theories of learning in cognitive systems, such as reinforcement learning, use the notion of "state-representation" to describe how agents decide which features of their environment are behaviorally-relevant and which can be ignored. However, these approaches typically assume "ground-truth" state representations that are known by the agent, and reward functions that need to be learned. Here we suggest an alternative approach in which state-representations are not assumed veridical, or even pre-defined, but rather emerge from the agent's goals through interaction with its environment. We illustrate this novel perspective by inferring the goals driving rat behavior in an odor-guided choice task and discuss its implications for developing, from first principles, an information-theoretic account of goal-directed state representation learning and behavior.
- Same action, different meaning: neural substrates of action semantic meaning. Cerebral Cortex, 32(19):4293–4303, 2022.
- Value-complexity tradeoff explains mouse navigational learning. PLOS Computational Biology, 16(12):e1008497, 2020.
- E. Balcetis and D. Dunning. See what you want to see: motivational influences on visual perception. Journal of personality and social psychology, 91(4):612, 2006.
- Settling the reward hypothesis. arXiv preprint arXiv:2212.10420, 2022.
- T. M. Cover. Elements of information theory. John Wiley & Sons, 1999.
- M. Csikszentmihalyi and I. S. Csikszentmihalyi. Optimal experience: Psychological studies of flow in consciousness. Cambridge university press, 1992.
- J. D. Dunne. Foundations of Dharmakirti’s philosophy. Simon and Schuster, 2004.
- I. Erev and G. Barron. On adaptation, maximization, and reinforcement learning among cognitive strategies. Psychological review, 112(4):912, 2005.
- Parietal lobe: from action organization to intention understanding. Science, 308(5722):662–667, 2005.
- D. Hume. A treatise of human nature. Clarendon Press, 1896.
- L. P. Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pages 1094–8. Citeseer, 1993.
- Uncovering the ‘state’: Tracing the hidden state representations that structure learning and decision-making. Behavioural Processes, 167:103891, 2019.
- Neurocomputational mechanisms underlying motivated seeing. Nature human behaviour, 3(9):962–973, 2019.
- Towards a unified theory of state abstraction for MDPs. In AI&M, 2006.
- M. Minsky. Computation: Finite and infinite machines prentice hall. Inc., Engelwood Cliffs, NJ, 1967.
- G. Molinaro and A. G. E. Collins. A goal-centric outlook on learning. Trends in Cognitive Sciences, 2023.
- Y. Niv. Learning task-state representations. Nature Neuroscience, 22(10):1544–1553, 2019.
- Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron, 51(4):509–520, 2006.
- E. Rosch and C. B. Mervis. Family resemblances: Studies in the internal structure of categories. Cognitive Psychology, 7(4):573–605, 1975.
- Apoha: Buddhist nominalism and human cognition. Columbia University Press, 2011.
- Minimal cross-trial generalization in learning the representation of an odor-guided choice task. PLoS Computational Biology, 18(3):e1009897, 2022.
- A. Tversky. Intransitivity of preferences. Psychological review, 76(1):31, 1969.
- L. Wittgenstein. Philosophical investigations. Wiley-Blackwell, New York, NY, USA, 1953.