Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Data-free Knowledge Distillation for Fine-grained Visual Categorization (2404.12037v1)

Published 18 Apr 2024 in cs.CV

Abstract: Data-free knowledge distillation (DFKD) is a promising approach for addressing issues related to model compression, security privacy, and transmission restrictions. Although the existing methods exploiting DFKD have achieved inspiring achievements in coarse-grained classification, in practical applications involving fine-grained classification tasks that require more detailed distinctions between similar categories, sub-optimal results are obtained. To address this issue, we propose an approach called DFKD-FGVC that extends DFKD to fine-grained visual categorization~(FGVC) tasks. Our approach utilizes an adversarial distillation framework with attention generator, mixed high-order attention distillation, and semantic feature contrast learning. Specifically, we introduce a spatial-wise attention mechanism to the generator to synthesize fine-grained images with more details of discriminative parts. We also utilize the mixed high-order attention mechanism to capture complex interactions among parts and the subtle differences among discriminative features of the fine-grained categories, paying attention to both local features and semantic context relationships. Moreover, we leverage the teacher and student models of the distillation framework to contrast high-level semantic feature maps in the hyperspace, comparing variances of different categories. We evaluate our approach on three widely-used FGVC benchmarks (Aircraft, Cars196, and CUB200) and demonstrate its superior performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Robust optimization, volume 28. Princeton university press, 2009.
  2. Poof: Part-based one-vs.-one features for fine-grained categorization, face verification, and attribute estimation. In CVPR, pages 955–962, 2013.
  3. Higher-order integration of hierarchical convolutional activations for fine-grained visual categorization. In ICCV, pages 511–520, 2017.
  4. Mixed high-order attention network for person re-identification. In CVPR, pages 371–381, 2019.
  5. Data-free learning of student networks. In ICCV, pages 3514–3522, 2019.
  6. Wasserstein contrastive representation distillation. In CVPR, pages 16296–16305, 2021.
  7. A simple framework for contrastive learning of visual representations. In ICML, pages 1597–1607. PMLR, 2020.
  8. Data-free network quantization with adversarial knowledge distillation. In CVPR, pages 710–711, 2020.
  9. Selective sparse sampling for fine-grained image recognition. In ICCV, pages 6599–6608, 2019.
  10. Momentum adversarial distillation: Handling large distribution shifts in data-free knowledge distillation. In Advances in NeurIPS, 2022.
  11. Data-free adversarial distillation. arXiv preprint arXiv:1912.11006, 2019.
  12. Contrastive model inversion for data-free knowledge distillation. In IJCAI, 2021.
  13. Weakly supervised complementary parts models for fine-grained image classification from the bottom up. In CVPR, pages 3034–3043, 2019.
  14. Attention mechanisms in computer vision: A survey. Computational Visual Media, 8(3):331–368, 2022.
  15. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016.
  16. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  17. Squeeze-and-excitation networks. pages 7132–7141, 2018.
  18. Part-stacked cnn for fine-grained visual categorization. In CVPR, pages 1173–1182, 2016.
  19. Attention convolutional binary neural tree for fine-grained visual categorization. In CVPR, pages 10468–10477, 2020.
  20. An attention-driven two-stage clustering method for unsupervised person re-identification. In ECCV, pages 20–36. Springer, 2020.
  21. Supervised contrastive learning. Advances in NeurIPS, 33:18661–18673, 2020.
  22. Adam: A method for stochastic optimization. In ICLR, 2014.
  23. 3d object representations for fine-grained categorization. In ICCV workshops, pages 554–561, 2013.
  24. Imagenet classification with deep convolutional neural networks. Advances in NeurIPS, 25, 2012.
  25. Pruning filters for efficient convnets. In ICLR, 2016.
  26. Filtration and distillation: Enhancing region attention for fine-grained visual categorization. In AAAI, volume 34, pages 11555–11562, 2020.
  27. Zero-shot adversarial quantization. In CVPR, pages 1512–1521, 2021.
  28. Data-free knowledge transfer: A survey. arXiv preprint arXiv:2112.15278, 2021.
  29. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
  30. Zero-shot knowledge transfer via adversarial belief matching. Advances in NeurIPS, 32, 2019.
  31. Multi-objective matrix normalization for fine-grained visual recognition. IEEE Transactions on Image Processing, 29:4996–5009, 2020.
  32. Spectral normalization for generative adversarial networks. In ICLR, 2018.
  33. Zero-shot knowledge distillation in deep networks. In ICML, pages 4743–4751. PMLR, 2019.
  34. Bam: Bottleneck attention module. BMVC, 2018.
  35. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2015.
  36. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pages 4510–4520, 2018.
  37. Channel-wise knowledge distillation for dense prediction. In ICCV, pages 5311–5320, 2021.
  38. Deep convolutional generative adversarial network. In Artificial Intelligence with Python, pages 289–301. Springer, 2022.
  39. Contrastive representation distillation. arXiv preprint arXiv:1910.10699, 2019.
  40. Residual attention network for image classification. In CVPR, pages 3156–3164, 2017.
  41. Graph-propagation based correlation learning for weakly supervised fine-grained image classification. In AAAI, volume 34, pages 12289–12296, 2020.
  42. Fine-grained image analysis with deep learning: A survey. TPAMI, 2021.
  43. Caltech-ucsd birds 200. 2010.
  44. Cbam: Convolutional block attention module. In ECCV, pages 3–19, 2018.
  45. Attention deep model with multi-scale deep supervision for person re-identification. IEEE Trans. ETCI, 5(1):70–78, 2021.
  46. Attention-aware compositional network for person re-identification. In CVPR, pages 2119–2128, 2018.
  47. Dreaming to distill: Data-free knowledge transfer via deepinversion. In CVPR, pages 8715–8724, 2020.
  48. Part-based r-cnns for fine-grained category detection. In ECCV, pages 834–849. Springer, 2014.
  49. Improving neural network quantization without retraining using outlier channel splitting. In ICML, pages 7543–7552. PMLR, 2019.
  50. Graph-based high-order relation discovery for fine-grained recognition. In CVPR, pages 15079–15088, 2021.
  51. Learning multi-attention convolutional neural network for fine-grained image recognition. In ICCV, pages 5209–5217, 2017.
  52. Channel distillation: Channel-wise attention for knowledge distillation. arXiv preprint arXiv:2006.01683, 2020.
  53. Learning attentive pairwise interaction for fine-grained classification. In AAAI, volume 34, pages 13130–13137, 2020.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets