Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoning (2403.00352v1)

Published 1 Mar 2024 in cs.CV and cs.LG

Abstract: In representation learning, a disentangled representation is highly desirable as it encodes generative factors of data in a separable and compact pattern. Researchers have advocated leveraging disentangled representations to complete downstream tasks with encouraging empirical evidence. This paper further investigates the necessity of disentangled representation in downstream applications. Specifically, we show that dimension-wise disentangled representations are unnecessary on a fundamental downstream task, abstract visual reasoning. We provide extensive empirical evidence against the necessity of disentanglement, covering multiple datasets, representation learning methods, and downstream network architectures. Furthermore, our findings suggest that the informativeness of representations is a better indicator of downstream performance than disentanglement. Finally, the positive correlation between informativeness and disentanglement explains the claimed usefulness of disentangled representations in previous works. The source code is available at https://github.com/Richard-coder-Nai/disentanglement-lib-necessity.git.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Measuring abstract reasoning in neural networks. In International conference on machine learning, 511–520. PMLR.
  2. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 1798–1828.
  3. Scaling learning algorithms towards AI. Large-scale kernel machines, 34(5): 1–41.
  4. 3D Shapes Dataset. https://github.com/deepmind/3dshapes-dataset/.
  5. Understanding disentangling in beta-VAE. arXiv preprint arXiv:1804.03599.
  6. An Empirical Study on Disentanglement of Negative-free Contrastive Learning. arXiv preprint arXiv:2206.04756.
  7. Emerging Properties in Self-Supervised Vision Transformers. In Proceedings of the International Conference on Computer Vision (ICCV).
  8. Isolating Sources of Disentanglement in Variational Autoencoders. In Advances in Neural Information Processing Systems.
  9. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference on Neural Information Processing Systems, 2180–2188.
  10. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15750–15758.
  11. Comon, P. 1994. Independent component analysis, a new concept? Signal processing, 36(3): 287–314.
  12. On the Transfer of Disentangled Representations in Realistic Settings. In International Conference on Learning Representations.
  13. Theory and Evaluation Metrics for Learning Disentangled Representations. In International Conference on Learning Representations.
  14. DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability. arXiv preprint arXiv:2210.00364.
  15. A framework for the quantitative evaluation of disentangled representations. In ICML.
  16. Comparing machines and humans on a visual categorization test. Proceedings of the National Academy of Sciences, 108(43): 17621–17625.
  17. Deep learning. MIT press.
  18. Generative adversarial nets. Advances in neural information processing systems, 27.
  19. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733.
  20. Attention on abstract visual reasoning. arXiv preprint arXiv:1911.05990.
  21. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16000–16009.
  22. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9729–9738.
  23. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230.
  24. Unsupervised deep learning identifies semantic disentanglement in single inferotemporal face patch neurons. Nature communications, 12(1): 6456.
  25. beta-vae: Learning basic visual concepts with a constrained variational framework. arXiv preprint.
  26. IB-GAN: Disentangled representation learning with information bottleneck GAN. arXiv preprint.
  27. Disentangling by factorising. In International Conference on Machine Learning, 2649–2658. PMLR.
  28. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
  29. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00848.
  30. Building machines that learn and think like people. Behavioral and brain sciences, 40.
  31. Structure by architecture: disentangled representations without regularization. arXiv preprint arXiv:2006.07796.
  32. Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans. In International Conference on Machine Learning, 6127–6139. PMLR.
  33. On the fairness of disentangled representations. Advances in Neural Information Processing Systems, 32.
  34. Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML.
  35. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, 6348–6359. PMLR.
  36. Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven’s Progressive Matrices. arXiv preprint arXiv:2201.12382.
  37. dSprites: Disentanglement testing Sprites dataset. https://github.com/deepmind/dsprites-dataset/.
  38. DINOv2: Learning Robust Visual Features without Supervision. arXiv preprint arXiv:2304.07193.
  39. Elements of causal inference: foundations and learning algorithms. The MIT Press.
  40. Raven, J. C. 1941. Standardization of progressive matrices, 1938. British Journal of Medical Psychology.
  41. Learning deep disentangled embeddings with the f-statistic loss. arXiv preprint arXiv:1802.05312.
  42. Disentanglement of Correlated Factors via Hausdorff Factorized Support. arXiv preprint arXiv:2210.07347.
  43. Schmidhuber, J. 1992. Learning factorial codes by predictability minimization. Neural computation, 4(6): 863–879.
  44. Toward causal representation learning. Proceedings of the IEEE, 109(5): 612–634.
  45. Improving generalization for abstract reasoning tasks using disentangled feature representations. arXiv preprint arXiv:1811.04784.
  46. The Role of Pretrained Representations for the OOD Generalization of RL Agents. In International Conference on Learning Representations.
  47. Recent advances in autoencoder-based representation learning. arXiv preprint arXiv:1812.05069.
  48. Are disentangled representations helpful for abstract visual reasoning? arXiv:1905.12506.
  49. Attention is all you need. Advances in neural information processing systems, 30.
  50. Desiderata for representation learning: A causal perspective. arXiv preprint arXiv:2109.03795.
  51. Watanabe, S. 1960. Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4(1): 66–82.
  52. Disentangling with Biological Constraints: A Theory of Functional Cell Types. arXiv preprint arXiv:2210.01768.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Ruiqian Nai (6 papers)
  2. Zixin Wen (8 papers)
  3. Ji Li (186 papers)
  4. Yuanzhi Li (119 papers)
  5. Yang Gao (761 papers)
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now