Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing (2312.07875v1)
Abstract: Free-hand sketches are appealing for humans as a universal tool to depict the visual world. Humans can recognize varied sketches of a category easily by identifying the concurrence and layout of the intrinsic semantic components of the category, since humans draw free-hand sketches based a common consensus that which types of semantic components constitute each sketch category. For example, an airplane should at least have a fuselage and wings. Based on this analysis, a semantic component-level memory module is constructed and embedded in the proposed structured sketch recognition network in this paper. The memory keys representing semantic components of each sketch category can be self-learned and enhance the recognition network's explainability. Our proposed networks can deal with different situations of sketch recognition, i.e., with or without semantic components labels of strokes. Experiments on the SPG and SketchIME datasets demonstrate the memory module's flexibility and the recognition network's explainability. The code and data are available at https://github.com/GuangmingZhu/SketchESC.
- Abstracting sketches through simple primitives. In ECCV, 396–412.
- Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks. In WACV, 839–847.
- This looks like that: deep learning for interpretable image recognition. volume 32.
- Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR.
- Interpretable part-whole hierarchies and conceptual-semantic relationships in neural networks. In CVPR, 13689–13698.
- A peek into the reasoning of neural networks: Interpreting with structural visual concepts. In CVPR, 2195–2204.
- A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477.
- A Neural Representation of Sketch Drawings. In ICLR.
- Memory-Based Graph Networks. In ICLR.
- Deepgcns: Can gcns go as deep as cnns? In ICCV, 9267–9276.
- Universal sketch perceptual grouping. In ECCV, 582–597.
- Sketch-R2CNN: an RNN-rasterization-CNN architecture for vector sketch recognition. IEEE TVCG, 27(9): 3745–3754.
- Prediction with Visual Evidence: Sketch Classification Explanation via Stroke-Level Attributions. IEEE TIP.
- Miller, T. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267: 1–38.
- Smooth Grad-CAM++: An Enhanced Inference Level Visualization Technique for Deep Convolutional Neural Network Models. arXiv preprint arXiv:1908.01224.
- Distribution-aware binarization of neural networks for sketch recognition. In WACV, 830–838.
- SketchXAI: A First Look at Explainability for Human Sketches. In CVPR, 23327–23337.
- Ramaswamy, H. G.; et al. 2020. Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization. In WACV, 983–991.
- Enabling my robot to play pictionary: Recurrent neural networks for sketch recognition. In ACM MM, 247–251.
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In ICCV, 618–626.
- One explanation is not enough: structured attention graphs for image classification. volume 34, 11352–11363.
- Visualizing data using t-SNE. JMLR, 9(11).
- Interpretable counterfactual explanations guided by prototypes. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 650–665.
- Dynamic graph cnn for learning on point clouds. ACM TOG, 38(5): 1–12.
- Sketchsegnet: A rnn model for labeling sketch strokes. In MLSP, 1–6.
- Deep learning for free-hand sketch: A survey. IEEE TPAMI, 45(1): 285–312.
- Sketchmate: Deep hashing for million-scale human sketch retrieval. In CVPR, 8090–8098.
- Multigraph transformer for free-hand sketch recognition. IEEE TNNLS, 33(10): 5150–5161.
- Sketchgnn: Semantic sketch segmentation with graph neural networks. ACM TOG, 40(3): 1–13.
- Sketch-a-net: A deep neural network that beats humans. IJCV, 122: 411–425.
- Structpool: Structured graph pooling via conditional random fields. In ICLR.
- Interpreting CNN knowledge via an explanatory graph. In AAAI, 4454–4463.
- A survey on freehand sketch recognition and retrieval. IMAVIS, 89: 67–87.
- Sketch Input Method Editor: A Comprehensive Dataset and Methodology for Systematic Input Recognition. In ACM MM.