Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Less is more: Summarizing Patch Tokens for efficient Multi-Label Class-Incremental Learning (2405.15633v1)

Published 24 May 2024 in cs.CV and cs.AI

Abstract: Prompt tuning has emerged as an effective rehearsal-free technique for class-incremental learning (CIL) that learns a tiny set of task-specific parameters (or prompts) to instruct a pre-trained transformer to learn on a sequence of tasks. Albeit effective, prompt tuning methods do not lend well in the multi-label class incremental learning (MLCIL) scenario (where an image contains multiple foreground classes) due to the ambiguity in selecting the correct prompt(s) corresponding to different foreground objects belonging to multiple tasks. To circumvent this issue we propose to eliminate the prompt selection mechanism by maintaining task-specific pathways, which allow us to learn representations that do not interact with the ones from the other tasks. Since independent pathways in truly incremental scenarios will result in an explosion of computation due to the quadratically complex multi-head self-attention (MSA) operation in prompt tuning, we propose to reduce the original patch token embeddings into summarized tokens. Prompt tuning is then applied to these fewer summarized tokens to compute the final representation. Our proposed method Multi-Label class incremental learning via summarising pAtch tokeN Embeddings (MULTI-LANE) enables learning disentangled task-specific representations in MLCIL while ensuring fast inference. We conduct experiments in common benchmarks and demonstrate that our MULTI-LANE achieves a new state-of-the-art in MLCIL. Additionally, we show that MULTI-LANE is also competitive in the CIL setting. Source code available at https://github.com/tdemin16/multi-lane

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Token merging: Your ViT but faster. In ICLR, 2023.
  2. Dark experience for general continual learning: a strong, simple baseline. NeurIPS, 2020.
  3. A continual learning survey: Defying forgetting in classification tasks. TPAMI, 2021.
  4. On the effectiveness of layernorm tuning for continual learning in vision transformers. In ICCVW, 2023.
  5. Knowledge restore and transfer for multi-label class-incremental learning. In ICCV, 2023.
  6. Podnet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, 2020.
  7. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html, 2007.
  8. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021.
  9. Visual prompt tuning. In ECCV, 2022.
  10. Imbalanced continual learning with partitioning reservoir sampling. In ECCV, 2020.
  11. Adam: A method for stochastic optimization. In ICLR, 2015.
  12. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.
  13. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  14. Spvit: Enabling faster vision transformers via soft token pruning. In ECCV, 2021.
  15. Learning multiple layers of features from tiny images. 2009.
  16. General multi-label image classification with transformers. In CVPR, 2021.
  17. Learning without forgetting. TPAMI, 2017.
  18. Optimizing class distribution in memory for multi-label online continual learning. arXiv preprint arXiv:2209.11469, 2022.
  19. Not all patches are what you need: Expediting vision transformers via token reorganizations. In ICLR, 2022.
  20. Microsoft coco: Common objects in context. In ECCV, 2014.
  21. Query2label: A simple transformer way to multi-label classification. arXiv preprint arXiv:2107.10834, 2021.
  22. Packnet: Adding multiple tasks to a single network by iterative pruning. In CVPR, 2018.
  23. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In ECCV, 2018.
  24. Token pooling in vision transformers for image classification. In WACV, 2023.
  25. Class-incremental learning: survey and performance evaluation on image classification. TPAMI, 2022.
  26. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. 1989.
  27. Ia-red2: Interpretability-aware redundancy reduction for vision transformers. NeurIPS, 2021.
  28. Gdumb: A simple approach that questions our progress in continual learning. In ECCV, 2020.
  29. icarl: Incremental classifier and representation learning. In CVPR, 2017.
  30. Asymmetric loss for multi-label classification. In ICCV, 2021.
  31. Learning to learn without forgetting by maximizing transfer and minimizing interference. In ICLR, 2019.
  32. Tokenlearner: What can 8 learned tokens do for images and videos? arXiv preprint arXiv:2106.11297, 2021.
  33. Progress & compress: A scalable framework for continual learning. In ICML, 2018.
  34. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In CVPR, 2023.
  35. Topology-preserving class-incremental learning. In ECCV, 2020.
  36. Attention is all you need. NeurIPS, 2017.
  37. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487, 2023.
  38. Multi-label image recognition by recurrently discovering attentional regions. In ICCV, 2017.
  39. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, 2022a.
  40. Learning to prompt for continual learning. In CVPR, 2022b.
  41. Large scale incremental learning. In CVPR, 2019.
  42. Adavit: Adaptive tokens for efficient vision transformer. In CVPR, 2022.
  43. Prototype augmentation and self-supervision for incremental learning. In CVPR, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Thomas De Min (6 papers)
  2. Massimiliano Mancini (66 papers)
  3. Stéphane Lathuilière (79 papers)
  4. Subhankar Roy (51 papers)
  5. Elisa Ricci (137 papers)