2000 character limit reached
Harmonizing Program Induction with Rate-Distortion Theory
Published 8 May 2024 in cs.HC, cs.CL, cs.LG, cs.SC, stat.ML, cs.IT, and math.IT | (2405.05294v1)
Abstract: Many aspects of human learning have been proposed as a process of constructing mental programs: from acquiring symbolic number representations to intuitive theories about the world. In parallel, there is a long-tradition of using information processing to model human cognition through Rate Distortion Theory (RDT). Yet, it is still poorly understood how to apply RDT when mental representations take the form of programs. In this work, we adapt RDT by proposing a three way trade-off among rate (description length), distortion (error), and computational costs (search budget). We use simulations on a melody task to study the implications of this trade-off, and show that constructing a shared program library across tasks provides global benefits. However, this comes at the cost of sensitivity to curricula, which is also characteristic of human learners. Finally, we use methods from partial information decomposition to generate training curricula that induce more effective libraries and better generalization.
- \NAT@swatrue
- (2020). Efficient data compression in perception and perceptual memory. Psychological review, 127(5), 891. \NAT@swatrue
- (2019, February). Adaptive allocation of human visual working memory capacity during statistical and categorical learning. Journal of vision, 19(2), 11. doi: 10.1167/19.2.11 \NAT@swatrue
- Berger, T. (1971). Rate-distortion theory: A mathematical basis for data compression. Prentice-Hall. \NAT@swatrue
- (2018). Decision by sampling implements efficient coding of psychoeconomic functions. Psychological Review, 125(6), 985. \NAT@swatrue
- (2003). Simplicity: a unifying principle in cognitive science? Trends in cognitive sciences, 7(1), 19–22. \NAT@swatrue
- Church, A. (1936). An unsolvable problem of elementary number theory. American Journal of Mathematics, 58(2), e1008367. \NAT@swatrue
- (2023). Exploring the hierarchical structure of human plans via program generation. arXiv preprint arXiv:2311.18644. \NAT@swatrue
- (2022). Curriculum learning for human compositional generalization. Proceedings of the National Academy of Sciences, 119(41), e2205582119. \NAT@swatrue
- (2021). Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the 42nd acm sigplan international conference on programming language design and implementation (pp. 835–850). \NAT@swatrue
- Feldman, J. (2000). Minimization of boolean complexity in human concept learning. Nature, 407(6804), 630–633. \NAT@swatrue
- Fodor, I. (1975). Language reforms of the past and in the developing countries. \NAT@swatrue
- (2020). Sequence generation using deep recurrent networks and embeddings: A study case in music. ArXiv e-prints, abs/2012.0. Retrieved from https://arxiv.org/abs/2012.01231 \NAT@swatrue
- Gershman, S. J. (2021). The rational analysis of memory. In Oxford handbook of human memory. Oxford, UK: Oxford University Press. \NAT@swatrue
- Gold, E. M. (1967). Language identification in the limit. Information and control, 10(5), 447–474. \NAT@swatrue
- (2008). A rational analysis of rule-based concept learning. Cognitive science, 32(1), 108–154. \NAT@swatrue
- Grünwald, P. D. (2007). The minimum description length principle. MIT press. \NAT@swatrue
- (2024). Commonality and variation in mental representations of music revealed by a cross-cultural comparison of rhythm priors in 15 countries. Nature Human Behaviour, 1–32. \NAT@swatrue
- (2006). Adaptor grammars: A framework for specifying compositional nonparametric bayesian models. Advances in neural information processing systems, 19. \NAT@swatrue
- (2021). Policy compression: An information bottleneck in action selection. In Psychology of learning and motivation (Vol. 74, pp. 195–232). Elsevier. \NAT@swatrue
- (2015). Human-level concept learning through probabilistic program induction. Science, 350(6266), 1332–1338. \NAT@swatrue
- (2010). Learning programs: A hierarchical bayesian approach. In Proceedings of the 27th international conference on machine learning (icml-10) (pp. 639–646). \NAT@swatrue
- (2020). Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and brain sciences, 43, e1. \NAT@swatrue
- (2010). Humans rapidly learn grammatical structure in a new musical scale. Music perception, 27(5), 377–388. \NAT@swatrue
- (2024). Information decomposition and the informational architecture of the brain. Trends in Cognitive Sciences. \NAT@swatrue
- (2021). Towards an extended taxonomy of information dynamics via integrated information decomposition. arXiv preprint arXiv:2109.13186. \NAT@swatrue
- (2019). Universality and diversity in human song. Science, 366(6468), eaax0868. \NAT@swatrue
- (2016). Episodic memory as a prerequisite for online updates of model structure. In Proceedings of the 38th Annual Conference of the Cognitive Science Society (pp. 1–7). \NAT@swatrue
- (2020). Optimal forgetting: Semantic compression of episodic memories. PLoS Computational Biology, 16(10), e1008367. \NAT@swatrue
- (2012). Bootstrapping in a language of thought: A formal model of numerical concept learning. Cognition, 123(2), 199–217. \NAT@swatrue
- (2022). Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks. arXiv preprint arXiv:2210.02996. \NAT@swatrue
- (2023). Compositionality under time pressure. In M. Goldwater, F. Anggoro, B. Hayes, and D. Ong (Eds.), Proceedings of the 45th Annual Conference of the Cognitive Science Society. Sydney, Australia: Cognitive Science Society. doi: 10.31234/osf.io/z2648
- (2018). Learning list concepts through program induction. BioRxiv, 321505.
- (2020). The child as hacker. Trends in cognitive sciences, 24(11), 900–915.
- (2022, December). A language of thought for the mental representation of geometric shapes. Cognitive Psychology, 139, 101527. Retrieved from https://www.sciencedirect.com/science/article/pii/S0010028522000639 doi: 10.1016/j.cogpsych.2022.101527
- Schönfinkel, M. (1924, September). Über die Bausteine der mathematischen Logik. Mathematische Annalen, 92(3), 305–316. Retrieved 2024-01-31, from https://doi.org/10.1007/BF01448013 doi: 10.1007/BF01448013
- Shannon, C. E. (1959). Coding theorems for a discrete source with a fidelity criterion. IRE Nat. Conv. Rec, 4(142-163), 1.
- Sims, C. R. (2016). Rate–distortion theory and human perception. Cognition, 152, 181–198.
- Solomonoff, R. J. (1964). A formal theory of inductive inference. part i. Information and control, 7(1), 1–22.
- (2003, March). Complexity distortion theory. IEEE Transactions on Information Theory, 49(3), 604–608. doi: 10.1109/TIT.2002.808135
- Teh, Y. W. (2006). A hierarchical bayesian language model based on pitman-yor processes. In Proceedings of the 21st international conference on computational linguistics and 44th annual meeting of the association for computational linguistics (pp. 985–992).
- (2005). Algorithmic Rate-Distortion Theory.
- (2010). Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515.
- (2023). A model of conceptual bootstrapping in human cognition. Nature Human Behaviour, 1–12.
- (2024). Predictive, scalable and interpretable knowledge tracing on structured domains. In The twelfth international conference on learning representations. Retrieved from https://openreview.net/forum?id=NgaLU2fP5D
Paper Prompts
Sign up for free to create and run prompts on this paper.