Papers
Topics
Authors
Recent
Search
2000 character limit reached

ArcSin: Adaptive ranged cosine Similarity injected noise for Language-Driven Visual Tasks

Published 27 Feb 2024 in cs.CV | (2402.17298v2)

Abstract: "A data scientist is tasked with developing a low-cost surgical VQA system for a 2-month workshop. Due to data sensitivity, she collects 50 hours of surgical video from a hospital, requiring two months for privacy approvals. Privacy restrictions prevent uploading data to platforms like ChatGPT, so she assembles one annotator and a medical expert to manually create QA pairs. This process takes three weeks and costs over $10,000. The trained model provides accurate responses within the limited data scope but lacks broader generalizability, completing the project in 3 months." To simplify the challenges presented in the scenario above. In this paper, we replace the image input with text for Vision-language training. Inspired by prior noise injection methods to reduce modality gaps, we introduce Adaptive ranged cosine Similarity injected noise (ArcSin). First, we introduce an innovative adaptive noise scale that effectively generates the textual elements with more variability while preserving the original text feature's integrity. Second, a similarity pool strategy is employed, expanding the domain generalization potential by broadening the overall noise scale. This dual strategy effectively broadens the scope of the original domain while safeguarding content integrity. Our empirical results demonstrate that these models closely rival those trained on images in terms of performance. Specifically, our method exhibits substantial improvements over the previous state-of-the-art, achieving gains of 1.9 and 1.1 CIDEr points in S-Cap and M-Cap, respectively. Additionally, we observe increases of 0.5 percentage points (pp), 1.4 pp, and 1.4 pp in accuracy for VQA, VQA-E, and VE, respectively, pushing the boundaries of what is achievable within the constraints of image-trained model benchmarks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. CrossPoint: Self-Supervised Cross-Modal contrastive learning for 3D point cloud understanding. arXiv preprint arXiv:2203.00680, 2022.
  2. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  3. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2425–2433, 2015.
  4. Vizwiz: Nearly real-time answers to visual questions. UIST: ACM Symposium on User Interface Software and Technology, 2010.
  5. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
  6. Knowledge boosting: Rethinking medical contrastive Vision-Language Pre-Training. arXiv:2307.07246, 2023.
  7. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104–120. Springer, 2020.
  8. CLAP: Learning audio concepts from natural language supervision. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
  9. Person re-identification method based on color attack and joint defence. CVPR workshop, pages 4313–4322, 2021.
  10. Generative adversarial nets. Advances in Neural Information Processing Systems, 27:2672–2680, 2014.
  11. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  12. I can’t believe there’s no images! learning visual tasks using only language data. arXiv preprint arXiv:2211.09778, 2022.
  13. AudioCLIP: Extending CLIP to image, text and audio. arXiv preprint arXiv:2106.13043, 2021.
  14. Robust multimodal representation learning with evolutionary adversarial attention networks. IEEE Transactions on Evolutionary Computation, 2021.
  15. Real-world super-resolution via kernel estimation and noise injection. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 466–467. IEEE, 2020.
  16. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv:2102.05918, 2021.
  17. Understanding and constructing latent modality structures in multi-modal representation learning. arXiv:2303.05952, 2023.
  18. Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging. Radiological Physics and Technology, 12(3):235–248, 2019.
  19. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015.
  20. A good sample is hard to find: Noise injection sampling and self-training for neural language generation models. In Proceedings of the 12th International Conference on Natural Language Generation, INLG 2019, Tokyo, Japan, October 29 - November 1, 2019, pages 584–593. Association for Computational Linguistics, 2019.
  21. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  22. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
  23. Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 552–567, 2018.
  24. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. arXiv:2203.02053, 2022.
  25. Cross-modal search method of technology video based on adversarial learning and feature fusion. arXiv:2210.05243, 2022.
  26. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019.
  27. Modeling semantic containment and exclusion in natural language inference. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 521–528, 2008.
  28. Text-only training for image captioning using noise-injected clip. arXiv preprint arXiv:2211.00575, 2022.
  29. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  30. A unified framework for multimodal domain adaptation. ACM Transactions on Multimedia Computing, Communications, and Applications, 2018.
  31. Integrating Multi-Label contrastive learning with dual adversarial graph neural networks for Cross-Modal retrieval. IEEE Trans. Pattern Anal. Mach. Intell., 45(4):4794–4811, 2023.
  32. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, 2021.
  33. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  34. Zero-shot text-to-image generation. arXiv:2102.12092, 2021.
  35. Clip models are few-shot learners: Empirical studies on vqa and visual entailment. arXiv preprint arXiv:2203.07190, 2022.
  36. COBRA: Contrastive Bi-Modal representation algorithm. arXiv:2005.03687, 2020.
  37. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  38. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  39. Show and tell: A neural image caption generator. IEEE Conference on Computer Vision and Pattern Recognition, 2015.
  40. Detecting adversarial examples for deep neural networks via layer directed discriminative noise injection. 2019 IEEE Asian Hardware-Oriented Security and Trust Symposium (AsianHOST), 2019.
  41. Introduction to big multimodal multimedia data with deep analytics. ACM Transactions on Multimedia Computing, Communications, and Applications, 2021.
  42. Connecting multi-modal contrastive representations. arXiv:2305.14381, 2023.
  43. Wav2CLIP: Learning robust audio representations from CLIP. arXiv:2110.11499, 2021.
  44. Visual entailment: A novel task for Fine-Grained image understanding. arXiv preprint arXiv:1901.06706, 2019.
  45. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687–10698, 2020.
  46. VideoCLIP: Contrastive pre-training for zero-shot Video-Text understanding. arXiv preprint arXiv:2109.14084, 2021.
  47. Multimodal knowledge alignment with reinforcement learning. arXiv preprint arXiv:2205.12630, 2022.
  48. Multimodal learning for fetal distress diagnosis using a multimodal medical information fusion framework. Frontiers in Physiology, 2022.
  49. Flow-based generative models for learning manifold to manifold mappings. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10711–10719. AAAI, 2020.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up