Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash 93 tok/s
Gemini 2.5 Pro 52 tok/s Pro
GPT-5 Medium 17 tok/s
GPT-5 High 14 tok/s Pro
GPT-4o 97 tok/s
GPT OSS 120B 455 tok/s Pro
Kimi K2 194 tok/s Pro
2000 character limit reached

Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning (2410.14633v2)

Published 18 Oct 2024 in cs.CV

Abstract: Vision Foundation Models (VFMs) have demonstrated outstanding performance on numerous downstream tasks. However, due to their inherent representation biases originating from different training paradigms, VFMs exhibit advantages and disadvantages across distinct vision tasks. Although amalgamating the strengths of multiple VFMs for downstream tasks is an intuitive strategy, effectively exploiting these biases remains a significant challenge. In this paper, we propose a novel and versatile "Swiss Army Knife" (SAK) solution, which adaptively distills knowledge from a committee of VFMs to enhance multi-task learning. Unlike existing methods that use a single backbone for knowledge transfer, our approach preserves the unique representation bias of each teacher by collaborating the lightweight Teacher-Specific Adapter Path modules with the Teacher-Agnostic Stem. Through dynamic selection and combination of representations with Mixture-of-Representations Routers, our SAK is capable of synergizing the complementary strengths of multiple VFMs. Extensive experiments show that our SAK remarkably outperforms prior state of the arts in multi-task learning by 10% on the NYUD-v2 benchmark, while also providing a flexible and robust framework that can readily accommodate more advanced model designs. Project page: https://innovator-zero.github.io/SAK/ .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (157)
  1. MTLoRA: Low-rank adaptation approach for efficient multi-task learning. In CVPR, 2024.
  2. Variational information distillation for knowledge transfer. In CVPR, 2019.
  3. Efficient controllable multi-task architectures. In ICCV, 2023.
  4. Do deep nets really need to be deep? In NeurIPS, 2014.
  5. Generative modeling for multi-task visual learning. In ICML, 2022.
  6. Knowledge distillation: A good teacher is patient and consistent. In CVPR, 2022.
  7. Stochastic filter groups for multi-task CNNs: Learning specialist and generalist convolution kernels. In ICCV, 2019.
  8. Exploring relational context for multi-task dense prediction. In ICCV, 2021.
  9. Automated search for resource-efficient branched multi-task networks. In BMVC, 2020.
  10. Model compression. In SIGKDD, 2006.
  11. Learning lightweight object detectors via multi-teacher progressive distillation. In ICML, 2023.
  12. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
  13. Rich Caruana. Multitask learning. Machine learning, 28(1):41–75, 1997.
  14. Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurIPS, 2022.
  15. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML, 2018.
  16. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In NeurIPS, 2020.
  17. Mod-Squad: Designing mixtures of experts as modular multi-task learners. In CVPR, 2023.
  18. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023.
  19. Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796, 2020.
  20. Vision transformers need registers. In ICLR, 2024.
  21. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
  22. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  23. Robert M. French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135, 1999.
  24. Efficient knowledge distillation from an ensemble of teachers. In Interspeech, 2017.
  25. NDDR-CNN: Layerwise feature fusing in multi-task CNNs by neural discriminative dimensionality reduction. In CVPR, 2019.
  26. Multi-task self-training for learning general representations. In ICCV, 2021.
  27. Dynamic task prioritization for multitask learning. In ECCV, 2018.
  28. Learning to branch for multi-task learning. In ICML, 2020.
  29. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016.
  30. A comprehensive overhaul of feature distillation. In ICCV, 2019.
  31. Distilling the knowledge in a neural network. In NeurIPS Deep Learning Workshop, 2014.
  32. Parameter-efficient transfer learning for NLP. In ICML, 2019.
  33. Going beyond multi-task dense prediction with synergy embedding models. In CVPR, 2024a.
  34. GNAS: A greedy neural architecture search method for multi-attribute learning. In ACM MM, 2018.
  35. YOLO-Med: Multi-task interaction network for biomedical images. In ICASSP, 2024b.
  36. Multi-task learning with attention for end-to-end autonomous driving. In CVPR, 2021.
  37. Online knowledge distillation for multi-task learning. In WACV, 2023.
  38. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.
  39. RotoGrad: Gradient homogenization in multitask learning. In ICLR, 2022.
  40. Reparameterizing convolutions for incremental multi-task learning without task interference. In ECCV, 2020.
  41. BRAVE: Broadening the visual encoding of vision-language models. In ECCV, 2024.
  42. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, 2018.
  43. Segment anything. In ICCV, 2023.
  44. Knowledge distillation for multi-task learning. In ECCV Workshops, 2020.
  45. Multi-task learning with 3D-aware regularization. In ICLR, 2024.
  46. M3ViT: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design. In NeurIPS, 2022.
  47. MTMamba: Enhancing multi-task dense scene understanding by mamba-based decoders. In ECCV, 2024.
  48. SPHINX: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023.
  49. Conflict-averse gradient descent for multi-task learning. In NeurIPS, 2021a.
  50. Towards impartial multi-task learning. In ICLR, 2021b.
  51. End-to-end multi-task learning with attention. In CVPR, 2019.
  52. Polyhistor: Parameter-efficient multi-task adaptation for dense vision tasks. In NeurIPS, 2022.
  53. Swin Transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021c.
  54. Wisdom of committee: Distilling from foundation model to specialized application model. arXiv preprint arXiv:2402.14035, 2024.
  55. Learning multiple tasks with multilinear relationship networks. In NeurIPS, 2017.
  56. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In CVPR, 2017.
  57. FedHCA2: Towards hetero-client federated multi-task learning. In CVPR, 2024a.
  58. Task indicating transformer for task-conditional dense predictions. In ICASSP, 2024b.
  59. Prompt guided transformer for multi-task dense prediction. IEEE Transactions on Multimedia, 2024c.
  60. Knowledge amalgamation from heterogeneous networks by common feature learning. In IJCAI, 2019.
  61. Collaboration by competition: Self-coordinated knowledge amalgamation for multi-talent student learning. In ECCV, 2020.
  62. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In ECCV, 2018.
  63. Lexicon3d: Probing visual foundation models for complex 3d scene understanding. In NeurIPS, 2024.
  64. Attentive single-tasking of multiple tasks. In CVPR, 2019.
  65. Learning to detect natural image boundaries using local brightness, color, and texture cues. TPAMI, 26(5):530–549, 2004.
  66. Time- Memory- and Parameter-Efficient Visual Adaptation. In CVPR, 2024.
  67. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. In ICLR, 2018.
  68. Cross-stitch networks for multi-task learning. In CVPR, 2016.
  69. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014.
  70. AdaMTL: Adaptive input-dependent inference for efficient multi-task learning. In CVPR, 2023.
  71. DINOv2: Learning robust visual features without supervision. TMLR, 2024.
  72. Frozen transformers in language models are effective visual encoder layers. In ICLR, 2024.
  73. In NeurIPS, 2019.
  74. Supervised evaluation of image segmentation and object proposal techniques. TPAMI, 38(7):1465–1478, 2015.
  75. Masked AutoDecoder is effective multi-task vision generalist. In CVPR, 2024.
  76. Learning transferable visual models from natural language supervision. In ICML, 2021.
  77. AM-RADIO: Agglomerative vision foundation model reduce all domains into one. In CVPR, 2024.
  78. SAM 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024.
  79. Controllable dynamic multi-task architectures. In CVPR, 2022.
  80. FitNets: Hints for thin deep nets. In ICLR, 2015.
  81. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In ICLR, 2018.
  82. Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
  83. Latent multi-task architecture learning. In AAAI, 2019.
  84. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015.
  85. UNIC: Universal classification models via multi-teacher distillation. In ECCV, 2024.
  86. Deep model compression: Distilling knowledge from noisy teachers. arXiv preprint arXiv:1610.09650, 2016.
  87. Multi-task learning as multi-objective optimization. In NeurIPS, 2018.
  88. Theia: Distilling diverse vision foundation models for robot learning. In CoRL, 2024.
  89. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR, 2017.
  90. Amalgamating knowledge towards comprehensive classification. In AAAI, 2019a.
  91. Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation. In ICCV, 2019b.
  92. Region-Based representations revisited. In CVPR, 2024.
  93. Efficient computation sharing for multi-task visual scene understanding. In ICCV, 2023.
  94. Indoor segmentation and support inference from RGBD images. In ECCV, 2012.
  95. Scale-aware task message transferring for multi-task learning. In ICME, 2023.
  96. Adaptive task-wise message passing for multi-task learning: A spatial interaction perspective. IEEE Transactions on Circuits and Systems for Video Technology, 2024a.
  97. Task-Interaction-Free multi-task learning with efficient hierarchical feature representation. In ACM MM, 2024b.
  98. SAM-Lightening: A lightweight segment anything model with dilated flash attention to achieve 30 times acceleration. arXiv preprint arXiv:2403.09195, 2024.
  99. Task switching network for multi-task learning. In ICCV, 2021.
  100. DIME-FM: Distilling multimodal and efficient foundation models. In ICCV, 2023.
  101. Semi-supervised knowledge amalgamation for sequence classification. In AAAI, 2021.
  102. Knowledge amalgamation for multi-label classification via label dependency transfer. In AAAI, 2023.
  103. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In NeurIPS, 2024a.
  104. Eyes wide shut? Exploring the visual shortcomings of multimodal LLMs. In CVPR, 2024b.
  105. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
  106. Branched multi-task networks: Deciding what layers to share. In BMVC, 2019.
  107. MTI-Net: Multi-scale task interaction networks for multi-task learning. In ECCV, 2020.
  108. Multi-task learning for dense prediction tasks: A survey. TPAMI, 44(7):3614–3633, 2021.
  109. Knowledge transfer from vision foundation models for efficient training of small task-specific models. In ICML, 2024.
  110. Unifying heterogeneous classifiers with distillation. In CVPR, 2019.
  111. Task adaptive parameter sharing for multi-task learning. In CVPR, 2022.
  112. RepViT-SAM: Towards real-time segmenting anything. arXiv preprint arXiv:2312.05760, 2023.
  113. SAM-CLIP: Merging vision foundation models towards semantic and spatial understanding. In CVPR, 2024a.
  114. TSP-Transformer: Task-specific prompts boosted transformer for holistic scene understanding. In WACV, 2024b.
  115. Task-aware low-rank adaptation of segment anything model. arXiv preprint arXiv:2403.10971, 2024c.
  116. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In ICLR, 2021.
  117. Class incremental learning with multi-teacher distillation. In CVPR, 2024.
  118. Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019.
  119. VMT-Adapter: Parameter-efficient transfer learning for multi-task dense scene understanding. In AAAI, 2024a.
  120. TFUT: Task fusion upward transformer model for multi-task learning on dense prediction. Computer Vision and Image Understanding, 244:104014, 2024b.
  121. PAD-Net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing. In CVPR, 2018.
  122. MTFormer: Multi-task learning via transformer and cross-task reasoning. In ECCV, 2022.
  123. Multi-task learning with multi-query transformer for dense prediction. IEEE Transactions on Circuits and Systems for Video Technology, 2023a.
  124. Deformable mixer transformer with gating for multi-task learning of dense prediction. arXiv preprint arXiv:2308.05721, 2023b.
  125. DeMT: Deformable mixer transformer for multi-task learning of dense prediction. In AAAI, 2023c.
  126. Multi-task learning with knowledge distillation for dense prediction. In ICCV, 2023d.
  127. CLIP-KD: An empirical study of clip model distillation. In CVPR, 2024a.
  128. Depth Anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024b.
  129. Depth Anything V2. In NeurIPS, 2024c.
  130. Contrastive multi-task dense prediction. In AAAI, 2023.
  131. Deep multi-task representation learning: A tensor factorisation approach. In ICLR, 2017.
  132. Multi-task dense prediction via mixture of low-rank experts. In CVPR, 2024d.
  133. Multi-objective meta learning. In NeurIPS, 2021.
  134. Inverted pyramid multi-task transformer for dense scene understanding. In ECCV, 2022.
  135. TaskExpert: Dynamically assembling multi-task representations with memorial mixture-of-experts. In ICCV, 2023a.
  136. TaskPrompter: Spatial-channel multi-task prompting for dense scene understanding. In ICLR, 2023b.
  137. InvPT++: Inverted pyramid multi-task transformer for visual scene understanding. TPAMI, 2024.
  138. Student becoming the master: Knowledge amalgamation for joint scene parsing, depth estimation, and more. In CVPR, 2019a.
  139. Amalgamating filtered knowledge: learning task-customized student from multi-task teachers. In IJCAI, 2019b.
  140. Learning from multiple teacher networks. In SIGKDD, 2017.
  141. Unleashing the power of multi-task learning: A comprehensive survey spanning traditional, deep, and pretrained foundation model eras. arXiv preprint arXiv:2404.18961, 2024.
  142. Gradient surgery for multi-task learning. In NeurIPS, 2020.
  143. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
  144. Sigmoid loss for language image pre-training. In ICCV, 2023.
  145. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289, 2023a.
  146. Rethinking of feature interaction for multi-task learning on dense prediction. arXiv preprint arXiv:2312.13514, 2023b.
  147. Adding conditional control to text-to-image diffusion models. In ICCV, 2023c.
  148. Transfer vision patterns for multi-task pixel learning. In ACM MM, 2021.
  149. Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34(12):5586–5609, 2021.
  150. Joint task-recursive learning for semantic segmentation and depth estimation. In ECCV, 2018.
  151. Pattern-affinitive propagation across depth, surface normal and semantic segmentation. In CVPR, 2019.
  152. TAMM: TriAdapter multi-modal learning for 3d shape understanding. In CVPR, 2024a.
  153. EfficientViT-SAM: Accelerated segment anything model without performance loss. arXiv preprint arXiv:2402.05008, 2024b.
  154. EdgeSAM: Prompt-in-the-loop distillation for on-device deployment of SAM. arXiv preprint arXiv:2312.06660, 2023.
  155. Image BERT pre-training with online tokenizer. In ICLR, 2022.
  156. Pattern-structure diffusion for multi-task learning. In CVPR, 2020.
  157. MoVA: Adapting mixture of vision experts to multimodal context. In NeurIPS, 2024.
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Ai Generate Text Spark Streamline Icon: https://streamlinehq.com

Paper Prompts

Sign up for free to create and run prompts on this paper using GPT-5.

List Streamline Icon: https://streamlinehq.com Explore 10 Community Prompts
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets