Papers
Topics
Authors
Recent
2000 character limit reached

TAP: The Attention Patch for Cross-Modal Knowledge Transfer from Unlabeled Modality (2302.02224v3)

Published 4 Feb 2023 in cs.LG and stat.ML

Abstract: This paper addresses a cross-modal learning framework, where the objective is to enhance the performance of supervised learning in the primary modality using an unlabeled, unpaired secondary modality. Taking a probabilistic approach for missing information estimation, we show that the extra information contained in the secondary modality can be estimated via Nadaraya-Watson (NW) kernel regression, which can further be expressed as a kernelized cross-attention module (under linear transformation). This expression lays the foundation for introducing The Attention Patch (TAP), a simple neural network add-on that can be trained to allow data-level knowledge transfer from the unlabeled modality. We provide extensive numerical simulations using real-world datasets to show that TAP can provide statistically significant improvement in generalization across different domains and different neural network architectures, making use of seemingly unusable unlabeled cross-modal data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Deep canonical correlation analysis. In International conference on machine learning, pages 1247–1255. PMLR.
  2. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443.
  3. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems, 32.
  4. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory, pages 92–100.
  5. Cha, J. (1994). Partial least squares. Advanced methods of marketing research, 407:52–78.
  6. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325.
  7. Skyformer: Remodel self-attention with gaussian kernel and nyström method. Advances in Neural Information Processing Systems, 34:2122–2135.
  8. Rethinking attention with performers. arXiv preprint arXiv:2009.14794.
  9. Formation of magnesium dendrites during electrodeposition. ACS Energy Letters, 4(2):375–376.
  10. Deng, L. (2012). The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142.
  11. The l⁢_⁢1𝑙_1l\_1italic_l _ 1 convergence of kernel density estimates. The Annals of Statistics, 7(5):1136–1139.
  12. Dietterich, T. G. et al. (2002). Ensemble learning. The handbook of brain theory and neural networks, 2(1):110–125.
  13. Semisupervised self-learning for hyperspectral image classification. IEEE transactions on geoscience and remote sensing, 51(7):4032–4044.
  14. When does cotraining work in real data? IEEE Transactions on Knowledge and Data Engineering, 23(5):788–799.
  15. Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention. IEEE Transactions on Multimedia, 15(7):1553–1568.
  16. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1933–1941.
  17. Devise: A deep visual-semantic embedding model. Advances in neural information processing systems, 26.
  18. Partial least-squares regression: a tutorial. Analytica chimica acta, 185:1–17.
  19. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16(12):2639–2664.
  20. Multimodal deep autoencoder for human pose recovery. IEEE transactions on image processing, 24(12):5659–5670.
  21. Semi-supervised multi-view deep discriminant representation learning. IEEE transactions on pattern analysis and machine intelligence, 43(7):2496–2509.
  22. Intra-view and inter-view supervised correlation analysis for multi-view feature learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28.
  23. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137.
  24. A random forest-based framework for crop mapping using temporal, spectral, textural and polarimetric observations. International Journal of Remote Sensing, 40(18):7221–7251.
  25. Msmd: maximum separability and minimum dependency feature selection for cropland classification from optical and radar data. International Journal of Remote Sensing, 39(8):2159–2176.
  26. Learning image embeddings using convolutional neural networks for improved multi-modal semantics. In Proceedings of the 2014 Conference on empirical methods in natural language processing (EMNLP), pages 36–45.
  27. Email classification with co-training. In Proceedings of the 2001 conference of the Centre for Advanced Studies on Collaborative research, page 8.
  28. Babytalk: Understanding and generating simple image descriptions. IEEE transactions on pattern analysis and machine intelligence, 35(12):2891–2903.
  29. Lee, D.-H. et al. (2013). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896.
  30. Cross-modal learning with adversarial samples. Advances in neural information processing systems, 32.
  31. A survey of multi-view representation learning. IEEE transactions on knowledge and data engineering, 31(10):1863–1883.
  32. Inter-modality face recognition. In European conference on computer vision, pages 13–26. Springer.
  33. Stable, fast and accurate: Kernelized attention with relative positional encoding. Advances in Neural Information Processing Systems, 34:22795–22807.
  34. Regularizing long short term memory with 3d human-skeleton sequences for action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3054–3062.
  35. Midge: Generating image descriptions from computer vision detections. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 747–756.
  36. Activity recognition using wearable physiological measurements: Selection of features from a comprehensive literature study. Sensors, 19(24).
  37. Nadaraya, E. A. (1964). On estimating regression. Theory of Probability & Its Applications, 9(1):141–142.
  38. Multimodal deep learning. In ICML.
  39. Multi-view clustering and semi-supervised classification with adaptive neighbours. In Thirty-first AAAI conference on artificial intelligence.
  40. Convex multiview semi-supervised classification. IEEE Transactions on Image Processing, 26(12):5718–5729.
  41. Multiview semi-supervised learning model for image classification. IEEE Transactions on Knowledge and Data Engineering, 32(12):2389–2400.
  42. Random feature attention. In International Conference on Learning Representations.
  43. Ccl: Cross-modal correlation learning with multigrained fusion by hierarchical network. IEEE Transactions on Multimedia, 20(2):405–420.
  44. Semi-supervised self-training of object detection models. In Applications of Computer Vision and the IEEE Workshop on Motion and Video Computing, IEEE Workshop on, volume 1, pages 29–36. IEEE Computer Society.
  45. Rosenblatt, M. (1956). Remarks on some nonparametric estimates of a density function. The annals of mathematical statistics, pages 832–837.
  46. Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1249.
  47. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
  48. Generalized multiview analysis: A discriminative latent space. In 2012 IEEE conference on computer vision and pattern recognition, pages 2160–2167. IEEE.
  49. Task Report: Memotion Analysis 1.0 @SemEval 2020: The Visuo-Lingual Metaphor! In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain. Association for Computational Linguistics.
  50. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596–608.
  51. Improved multimodal deep learning with variation of information. Advances in neural information processing systems, 27.
  52. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PMLR.
  53. Semi-supervised self-training for decision tree classifiers. International Journal of Machine Learning and Cybernetics, 8(1):355–370.
  54. Separating style and content with bilinear models. Neural computation, 12(6):1247–1283.
  55. Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowledge and Information systems, 42(2):245–284.
  56. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, page 6558. NIH Public Access.
  57. A survey on semi-supervised learning. Machine Learning, 109(2):373–440.
  58. Attention is all you need. Advances in neural information processing systems, 30.
  59. Translating videos to natural language using deep recurrent neural networks. In HLT-NAACL.
  60. Wan, X. (2009). Co-training for cross-lingual sentiment classification. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 235–243.
  61. Kernel Smoothing. CRC press.
  62. A comprehensive survey on cross-modal retrieval. arXiv preprint arXiv:1607.06215.
  63. Orcca: Optimal randomized canonical correlation analysis. IEEE Transactions on Neural Networks and Learning Systems.
  64. Watson, G. S. (1964). Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A, pages 359–372.
  65. Hysad: A semi-supervised hybrid shilling attack detector for trustworthy product recommendation. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 985–993.
  66. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687–10698.
  67. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14138–14148.
  68. Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29.
  69. Yarowsky, D. (1995). Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189–196.
  70. Zhu, X. J. (2005). Semi-supervised learning literature survey. Technical Report TR 1530.
  71. Rethinking pre-training and self-training. Advances in neural information processing systems, 33:3833–3845.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Slide Deck Streamline Icon: https://streamlinehq.com

Whiteboard

Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Sign up for free to view the 1 tweet with 1 like about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free