· Trending Papers
Emergent Mind

Compositional Generalization from First Principles

(2307.05596)
Published Jul 10, 2023 in cs.LG and stat.ML

Abstract

Leveraging the compositional nature of our world to expedite learning and facilitate generalization is a hallmark of human perception. In machine learning, on the other hand, achieving compositional generalization has proven to be an elusive goal, even for models with explicit compositional priors. To get a better handle on compositional generalization, we here approach it from the bottom up: Inspired by identifiable representation learning, we investigate compositionality as a property of the data-generating process rather than the data itself. This reformulation enables us to derive mild conditions on only the support of the training distribution and the model architecture, which are sufficient for compositional generalization. We further demonstrate how our theoretical framework applies to real-world scenarios and validate our findings empirically. Our results set the stage for a principled theoretical study of compositional generalization.

We're not able to analyze this paper right now due to high demand.

Please check back later (sorry!).

Sign up for a free account or log in to generate a summary of this paper:

Sign Up Now
or
Log In

We ran into a problem analyzing this paper.

Newsletter

Get summaries of trending comp sci papers delivered straight to your inbox:

Unsubscribe anytime.

References
  1. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1):3–71, March 1988. ISSN 0010-0277. doi: 10.1016/0010-0277(88)90031-5.
  2. Noam Chomsky. Aspects of the Theory of Syntax, volume 11. MIT press
  3. Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
  4. The role of Disentanglement in Generalisation. In International Conference on Learning Representations, February 2022a.
  5. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International Conference on Machine Learning, pages 2873–2882. PMLR
  6. Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks
  7. Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
  8. On the fairness of disentangled representations. Advances in neural information processing systems, 32
  9. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013. doi: 10.1109/TPAMI.2013.50.
  10. Are disentangled representations helpful for abstract visual reasoning? In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  11. Lost in Latent Space: Examining failures of disentangled models at combinatorial generalisation. In Advances in Neural Information Processing Systems, October 2022b.
  12. Embrace the Gap: VAEs Perform Independent Mechanism Analysis
  13. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979–12990. PMLR
  14. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451–16467
  15. Compositional generalization in a deep seq2seq model by separating syntax and semantics
  16. Learning to Recombine and Resample Data for Compositional Generalization
  17. Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networks
  18. Object-Centric Learning with Slot Attention. In Advances in Neural Information Processing Systems, volume 33, pages 11525–11538. Curran Associates, Inc.
  19. Matrix capsules with EM routing. In International Conference on Learning Representations
  20. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424–2433. PMLR
  21. MONet: Unsupervised Scene Decomposition and Representation, January 2019
  22. Conditional Object-Centric Learning from Video
  23. Training neural networks to encode symbols enables combinatorial generalization. Philosophical Transactions of the Royal Society B: Biological Sciences, 375(1791):20190309, February 2020. doi: 10.1098/rstb.2019.0309.
  24. Learning and generalization of compositional representations of visual scenes, March 2023
  25. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  26. Combinatorial optimization and reasoning with graph neural networks
  27. Bi-linear value networks for multi-goal reinforcement learning. In International Conference on Learning Representations
  28. Complex-Valued Autoencoders for Object Discovery, November 2022
  29. Domain adaptation–can quantity compensate for quality? Annals of Mathematics and Artificial Intelligence, 70:185–202
  30. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5)
  31. Towards Out-Of-Distribution Generalization: A Survey
  32. A theory of learning from different domains. Machine learning, 79:151–175
  33. Learning to Extrapolate: A Transductive Approach. In The Eleventh International Conference on Learning Representations, February 2023.
  34. First Steps Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domains. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, October 2022.

Show All 34