Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Self-supervised Representation Learning From Random Data Projectors (2310.07756v2)

Published 11 Oct 2023 in cs.LG

Abstract: Self-supervised representation learning~(SSRL) has advanced considerably by exploiting the transformation invariance assumption under artificially designed data augmentations. While augmentation-based SSRL algorithms push the boundaries of performance in computer vision and natural language processing, they are often not directly applicable to other data modalities, and can conflict with application-specific data augmentation constraints. This paper presents an SSRL approach that can be applied to any data modality and network architecture because it does not rely on augmentations or masking. Specifically, we show that high-quality data representations can be learned by reconstructing random data projections. We evaluate the proposed approach on a wide range of representation learning tasks that span diverse modalities and real-world applications. We show that it outperforms multiple state-of-the-art SSRL baselines. Due to its wide applicability and strong empirical results, we argue that learning from randomness is a fruitful research direction worthy of attention and further study.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E, 64(6):061907, 2001.
  2. A public domain dataset for human activity recognition using smartphones. In The European Symposium on Artificial Neural Networks, 2013.
  3. Scarf: Self-supervised contrastive learning using random feature corruption. In International Conference on Learning Representations, 2022.
  4. Parameterized neural networks for high-energy physics. The European Physical Journal C, 76(5):235, Apr 2016. ISSN 1434-6052. doi: 10.1140/epjc/s10052-016-4099-4.
  5. Randall Balestriero. Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Head. arXiv:2302.10260, 2023.
  6. A Cookbook of Self-Supervised Learning. arXiv:2304.12210, 2023.
  7. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013.
  8. Flows for simultaneous manifold learning and density estimation. In Advances in Neural Information Processing Systems, volume 33, 2020.
  9. Machine Learning for First-Order Theorem Proving - Learning to Select a Good Heuristic. J. Autom. Reason., 53:141–172, 2014.
  10. John KC Chan. The wonderful colors of the hematoxylin–eosin stain in diagnostic surgical pathology. International Journal of Surgical Pathology, 22(1):12–32, 2014.
  11. Fast greedy map inference for determinantal point process to improve recommendation diversity. In Advances in Neural Information Processing Systems, volume 31, 2018.
  12. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp.  1597–1607, 13–18 Jul 2020.
  13. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  15750–15758, 2021.
  14. Subject-aware contrastive learning for biosignals. arXiv:2007.04871, 2020.
  15. TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders. arXiv preprint arXiv:2303.00320, 2023.
  16. Learning a similarity metric discriminatively, with application to face verification. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1, pp.  539–546, 2005. doi: 10.1109/CVPR.2005.202.
  17. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, pp.  4171–4186, 2019.
  18. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp.  1422–1430, 2015.
  19. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021.
  20. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp.  2352–2359, 2021.
  21. The Effectiveness of Image Augmentation in Deep Learning Networks for Detecting COVID-19: A Geometric Transformation Perspective. Frontiers in Medicine, 8:629134, 2021. doi: 10.3389/fmed.2021.629134.
  22. Why Do Self-Supervised Models Transfer? On the Impact of Invariance on Downstream Tasks. In Proceedings of the 33rd British Machine Vision Conference 2022, 2022.
  23. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, 2020.
  24. STab: Self-supervised Learning for Tabular Data. In NeurIPS 2022 First Table Representation Workshop, 2022. URL https://openreview.net/forum?id=EfR55bFcrcI.
  25. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):96, 2019.
  26. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  770–778, 2016.
  27. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  9729–9738, 2020.
  28. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  16000–16009, 2022.
  29. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006.
  30. An empirical survey of data augmentation for time series classification with neural networks. PLOS ONE, 16(7):1–32, 2021.
  31. Decentralized federated learning through proxy model sharing. Nature Communications, 14(1):2899, May 2023. ISSN 2041-1723. doi: 10.1038/s41467-023-38569-4.
  32. Benchmarking self-supervised learning on diverse pathology datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3344–3354, June 2023.
  33. Ron Kohavi. Scaling up the accuracy of Naive-Bayes classifiers: A decision-tree hybrid. In Second International Conference on Knowledge Discovery and Data Mining, volume 96, pp.  202–207, 1996.
  34. A mutual information maximization perspective of language representation learning. In International Conference on Learning Representations, 2020.
  35. Contrastive representation learning: A framework and review. IEEE Access, 8:193907–193934, 2020.
  36. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 35(1):857–876, 2021.
  37. Time series contrastive learning with information-aware augmentations. Proceedings of the AAAI Conference on Artificial Intelligence, 37(4):4534–4542, 2023. doi: 10.1609/aaai.v37i4.25575.
  38. MET: Masked Encoding for Tabular Data. In NeurIPS 2022 First Table Representation Workshop, 2022.
  39. Contrastive representation learning for electroencephalogram classification. In Machine Learning for Health, pp.  238–253. PMLR, 2020.
  40. Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In Proceedings of the 8th ACM on Multimedia Systems Conference, pp.  164–169, 2017.
  41. Judith M S Prewitt. Object enhancement and extraction. Picture processing and Psychopictorics, 10(1):15–19, 1970.
  42. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. In Advances in Neural Information Processing Systems, volume 33, 2020.
  43. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.
  44. Adversarial multiple-target domain adaptation for fault classification. IEEE Transactions on Instrumentation and Measurement, 70:1–11, 2020.
  45. Learning Internal Representations by Error Propagation, pp.  318–362. MIT Press, Cambridge, MA, USA, 1986. ISBN 026268053X.
  46. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  815–823, 2015.
  47. RandStainNA: Learning Stain-Agnostic Features from Histology Slides by Bridging Stain Augmentation and Normalization. In Medical Image Computing and Computer Assisted Intervention, pp.  212–221. Springer, 2022.
  48. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1–48, 2019.
  49. MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models. In Proceedings of the Fourth Conference on Medical Imaging with Deep Learning, volume 143, pp.  728–744, 2021.
  50. What makes for good views for contrastive learning? In Advances in neural information processing systems, volume 33, 2020.
  51. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pp.  10268–10278, 2021.
  52. SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning. In Advances in Neural Information Processing Systems, volume 34, pp.  18853–18865, 2021.
  53. Data Augmentation of Wearable Sensor Data for Parkinson’s Disease Monitoring Using Convolutional Neural Networks. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, pp.  216–220, 2017. doi: 10.1145/3136755.3136817.
  54. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
  55. Towards domain-agnostic contrastive learning. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp.  10530–10541, 2021.
  56. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine learning, pp.  1096–1103, 2008. doi: 10.1145/1390156.1390294.
  57. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning, 2013.
  58. Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International Joint Conference on Neural Networks, pp.  1578–1585. IEEE, 2017.
  59. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019. doi: 10.18653/v1/D19-1670.
  60. Conditional bert contextual augmentation. In International Conference onComputational Science, pp.  84–95, 2019.
  61. What Should Not Be Contrastive in Contrastive Learning. In International Conference on Learning Representations, 2021.
  62. SimMIM: A Simple Framework for Masked Image Modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  9653–9663, 2022.
  63. Neighborhood contrastive learning applied to online patient monitoring. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp.  11964–11974, 2021.
  64. Vime: Extending the success of self-and semi-supervised learning to tabular domain. In Advances in Neural Information Processing Systems, volume 33, pp.  11033–11043, 2020.
  65. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp.  12310–12320. PMLR, 2021.
  66. Rethinking the augmentation module in contrastive learning: Learning hierarchical augmentation invariance with expanded views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  16650–16659, 2022.
  67. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  1058–1067, 2017.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now