Papers
Topics
Authors
Recent
Search
2000 character limit reached

Few-shot Learner Parameterization by Diffusion Time-steps

Published 5 Mar 2024 in cs.CV | (2403.02649v2)

Abstract: Even when using large multi-modal foundation models, few-shot learning is still challenging -- if there is no proper inductive bias, it is nearly impossible to keep the nuanced class attributes while removing the visually prominent attributes that spuriously correlate with class labels. To this end, we find an inductive bias that the time-steps of a Diffusion Model (DM) can isolate the nuanced class attributes, i.e., as the forward diffusion adds noise to an image at each time-step, nuanced attributes are usually lost at an earlier time-step than the spurious attributes that are visually prominent. Building on this, we propose Time-step Few-shot (TiF) learner. We train class-specific low-rank adapters for a text-conditioned DM to make up for the lost attributes, such that images can be accurately reconstructed from their noisy ones given a prompt. Hence, at a small time-step, the adapter and prompt are essentially a parameterization of only the nuanced class attributes. For a test image, we can use the parameterization to only extract the nuanced class attributes for classification. TiF learner significantly outperforms OpenCLIP and its adapters on a variety of fine-grained and customized few-shot learning tasks. Codes are in https://github.com/yue-zhongqi/tif.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
  2. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829, 2023.
  3. Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288, 2019.
  4. Alexander D’Amour. On multi-cause causal inference with unobserved confounding: Counterexamples, impossibility, and alternatives. In AISTATS, 2019.
  5. A baseline for few-shot image classification. In International Conference on Learning Representations, 2020.
  6. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
  7. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544, 2021.
  8. Generating classification weights with gnn denoising autoencoders for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  9. Calip: Zero-shot enhancement of clip with parameter-free attention. In AAAI, 2022.
  10. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230, 2018.
  11. Denoising diffusion probabilistic models. NeurIPS, 2020.
  12. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
  13. Openclip, 2021. If you use this software, please cite it as below.
  14. Maple: Multi-modal prompt learning. In CVPR, 2023.
  15. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
  16. Your diffusion model is secretly a zero-shot classifier. In ICCV, 2023.
  17. Cycle self-training for domain adaptation. In NeurIPS, 2021.
  18. Large-scale vehicle re-identification in urban surveillance videos. In 2016 IEEE International Conference on Multimedia and Expo (ICME), 2016a.
  19. A deep learning-based approach to progressive vehicle re-identification for urban surveillance. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, 2016b.
  20. Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML, 2019.
  21. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
  22. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023.
  23. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. NeurIPS, 14, 2001.
  24. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  25. Learning transferable visual models from natural language supervision. In ICML, 2021.
  26. Performance measures and a data set for multi-target, multi-camera tracking. In ECCV Workshops, 2016.
  27. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  28. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, 2015.
  29. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
  30. Imagenet large scale visual recognition challenge. IJCV, 2015.
  31. Toward causal representation learning. Proceedings of the IEEE, 2021.
  32. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  33. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
  34. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
  35. Meta-transfer learning for few-shot learning. In CVPR, 2019.
  36. Unsupervised domain adaptation via structurally regularized deep clustering. In CVPR, 2020.
  37. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 2018.
  38. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, 2016.
  39. Few-shot classification with feature map reconstruction networks. In CVPR, 2021.
  40. Interventional few-shot learning. In NeurIPS, 2020.
  41. Make the u in uda matter: Invariant consistency learning for unsupervised domain adaptation. In NeurIPS, 2023.
  42. Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers. 2020.
  43. Tip-adapter: Training-free clip-adapter for better vision-language modeling. In ECCV, 2022a.
  44. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In CVPR, 2023.
  45. Unsupervised representation learning from pre-trained diffusion probabilistic models. NeurIPS, 2022b.
  46. Conditional prompt learning for vision-language models. In CVPR, 2022a.
  47. Learning to prompt for vision-language models. IJCV, 2022b.
  48. Prompt-aligned gradient for prompt tuning. International Conference on Computer Vision, 2023.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up