Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
120 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
3 tokens/sec
DeepSeek R1 via Azure Pro
55 tokens/sec
2000 character limit reached

Residual Connections Harm Generative Representation Learning (2404.10947v4)

Published 16 Apr 2024 in cs.CV

Abstract: We show that introducing a weighting factor to reduce the influence of identity shortcuts in residual networks significantly enhances semantic feature learning in generative representation learning frameworks, such as masked autoencoders (MAEs) and diffusion models. Our modification notably improves feature quality, raising ImageNet-1K K-Nearest Neighbor accuracy from 27.4% to 63.9% and linear probing accuracy from 67.8% to 72.7% for MAEs with a ViT-B/16 backbone, while also enhancing generation quality in diffusion models. This significant gap suggests that, while residual connection structure serves an essential role in facilitating gradient propagation, it may have a harmful side effect of reducing capacity for abstract learning by virtue of injecting an echo of shallower representations into deeper layers. We ameliorate this downside via a fixed formula for monotonically decreasing the contribution of identity connections as layer depth increases. Our design promotes the gradual development of feature abstractions, without impacting network trainability. Analyzing the representations learned by our modified residual networks, we find correlation between low effective feature rank and downstream task performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Gpt-4 technical report. arXiv:2303.08774, 2023.
  2. Implicit regularization in deep matrix factorization. NeurIPS, 2019.
  3. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, 2023.
  4. Data2vec: A general framework for self-supervised learning in speech, vision and language. In ICML, 2022.
  5. Beit: Bert pre-training of image transformers. arXiv:2106.08254, 2021.
  6. Mechanism of feature learning in convolutional neural networks. arXiv:2309.00570, 2023.
  7. Representation learning: A review and new perspectives. TPAMI, 2013.
  8. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
  9. A simple framework for contrastive learning of visual representations. In ICML, 2020a.
  10. Big self-supervised models are strong semi-supervised learners. NeurIPS, 2020b.
  11. Improved baselines with momentum contrastive learning. arXiv:2003.04297, 2020c.
  12. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
  13. Context autoencoder for self-supervised representation learning. IJCV, 2024.
  14. Y. Cheng. Mean shift, mode seeking, and clustering. TPAMI, 1995.
  15. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
  16. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In ICML, 2021.
  17. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929, 2020.
  18. Optimal signal propagation in ResNets through residual scaling. arXiv:2305.07715, 2023.
  19. The emergence of clusters in self-attention dynamics. NeurIPS, 2024.
  20. X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In ICAIS, 2010.
  21. Generative adversarial nets. NeurIPS, 2014.
  22. Highway and residual networks learn unrolled iterative estimation. arXiv:1612.07771, 2016.
  23. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 2020.
  24. Deep residual learning for image recognition. In CVPR, 2016.
  25. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
  26. Masked autoencoders are scalable vision learners. In CVPR, 2022.
  27. Denoising diffusion probabilistic models. NeurIPS, 2020.
  28. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 1997.
  29. Densely connected convolutional networks. In CVPR, 2017.
  30. Contrastive masked autoencoders are stronger vision learners. TPAMI, 2023.
  31. The low-rank simplicity bias in deep networks. arXiv:2103.10427, 2021.
  32. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
  33. Implicit rank-minimizing autoencoder. NeurIPS, 2020.
  34. Progressive growing of gans for improved quality, stability, and variation. arXiv:1710.10196, 2017.
  35. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
  36. A style-based generator architecture for generative adversarial networks. TPAMI, 2021.
  37. Elucidating the design space of diffusion-based generative models. NeurIPS, 2022.
  38. D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
  39. Segment anything. arXiv:2304.02643, 2023.
  40. Large language models are zero-shot reasoners. NeurIPS, 2022.
  41. FractalNet: Ultra-deep neural networks without residuals. ICLR, 2017.
  42. A systematic study and comprehensive evaluation of chatgpt on benchmark datasets. arXiv:2305.18486, 2023.
  43. Mage: Masked generative encoder to unify representation learning and image synthesis. In CVPR, 2023.
  44. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv:1711.05101, 2017.
  45. Towards understanding the role of over-parametrization in generalization of neural networks. arXiv:1805.12076, 2018.
  46. Feature learning in neural networks and kernel machines that recursively learn features. arXiv:2212.13881, 2022.
  47. Linear recursive feature machines provably recover low-rank matrices. arXiv:2401.04553, 2024.
  48. Zero-shot text-to-image generation. In ICML, 2021.
  49. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  50. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
  51. O. Roy and M. Vetterli. The effective rank: A measure of effective dimensionality. In ESPC, 2007.
  52. P. Savarese and D. Figueiredo. Residual gates: A simple mechanism for improved network optimization. In ICLR, 2017.
  53. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
  54. Improving image captioning with better use of captions. arXiv:2006.11807, 2020.
  55. The effectiveness of mae pre-pretraining for billion-scale pretraining. arXiv:2303.13496, 2023.
  56. Denoising diffusion implicit models. ICLR, 2021.
  57. Score-based generative modeling through stochastic differential equations. arXiv:2011.13456, 2020.
  58. Consistency models. arXiv:2303.01469, 2023.
  59. Highway networks. arXiv:1505.00387, 2015a.
  60. Training very deep networks. NeurIPS, 2015b.
  61. Going deeper with convolutions. In CVPR, 2015.
  62. Gemini: a family of highly capable multimodal models. arXiv:2312.11805, 2023.
  63. Attention is all you need. NeurIPS, 2017.
  64. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
  65. ibot: Image bert pre-training with online tokenizer. arXiv:2111.07832, 2021.
  66. Sparsely aggregated convolutional networks. In ECCV, 2018a.
  67. Sparsely aggregated convolutional networks. In ECCV, 2018b.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now