Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

What Sketch Explainability Really Means for Downstream Tasks (2403.09480v1)

Published 14 Mar 2024 in cs.CV and cs.AI

Abstract: In this paper, we explore the unique modality of sketch for explainability, emphasising the profound impact of human strokes compared to conventional pixel-oriented studies. Beyond explanations of network behavior, we discern the genuine implications of explainability across diverse downstream sketch-related tasks. We propose a lightweight and portable explainability solution -- a seamless plugin that integrates effortlessly with any pre-trained model, eliminating the need for re-training. Demonstrating its adaptability, we present four applications: highly studied retrieval and generation, and completely novel assisted drawing and sketch adversarial attacks. The centrepiece to our solution is a stroke-level attribution map that takes different forms when linked with downstream tasks. By addressing the inherent non-differentiability of rasterisation, we enable explanations at both coarse stroke level (SLA) and partial stroke level (P-SLA), each with its advantages for specific downstream tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (124)
  1. Sanity checks for saliency maps. In NeurIPS, 2018.
  2. Debugging tests for model explanations. arXiv preprint arXiv:2011.05429, 2020.
  3. Advances in adversarial attacks and defenses in computer vision: A survey. arXiv preprint arXiv:2108.00401, 2021.
  4. Prompting for discovery: Flexible sense-making for ai art-making with dreamsheets. arXiv preprint arXiv:2310.09985, 2023.
  5. Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104, 2017a.
  6. A unified view of gradient-based attribution methods for deep neural networks. In NIPSW, 2017b.
  7. On the robustness of semantic segmentation models to adversarial attacks. In CVPR, 2018.
  8. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 2015.
  9. Sketchinr: A first look into sketches as implicit neural representations. In CVPR, 2024a.
  10. Doodle your 3d: From abstract freehand sketches to precise 3d shapes. In CVPR, 2024b.
  11. Pixelor: A competitive sketching ai agent. so you think you can sketch? ACM TOG, 2020a.
  12. Sketch less for more: On-the-fly fine-grained sketch based image retrieval. In CVPR, 2020b.
  13. More photos are all you need: Semi-supervised learning for fine-grained sketch based image retrieval. In CVPR, 2021.
  14. Doodle it yourself: Class incremental learning by drawing a few sketches. In CVPR, 2022a.
  15. Sketching without worrying: Noise-tolerant sketch-based image retrieval. In CVPR, 2022b.
  16. B-cos: Networks: Alignment is all we need for interpretability. In CVPR, 2022.
  17. Approximating cnns with bag-of-local-features models works surprisingly well on imagenet. arXiv preprint arXiv:1904.00760, 2019.
  18. Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks. In ICCV, 2015.
  19. This looks like that: deep learning for interpretable image recognition. In NeurIPS, 2019.
  20. Deepfacedrawing: Deep generation of face images from sketches. ACM TOG, 2020a.
  21. Sketchygan: Towards diverse and realistic sketch to image synthesis. In ICCV, 2018.
  22. One-shot adversarial attacks on visual tracking with dual attention. In CVPR, 2020b.
  23. Partially does it: Towards scene-level fg-sbir with partial input. In CVPR, 2022a.
  24. Fs-coco: Towards understanding of freehand sketches of common objects in context. In ECCV, 2022b.
  25. Garment ideation: Iterative view-aware sketch-based garment modeling. In 3DV, 2022c.
  26. Livesketch: Query perturbations for guided sketch-based visual search. In CVPR, 2019.
  27. Real time image saliency for black box classifiers. In NeurIPS, 2017.
  28. Béziersketch: A generative model for scalable vector sketches. In ECCV, 2020.
  29. Doodle to search: Practical zero-shot sketch-based image retrieval. In CVPR, 2019.
  30. How important is a neuron? arXiv preprint arXiv:1805.12233, 2018.
  31. Sharcs: Shared concept space for explainable multimodal learning. arXiv preprint arXiv:2307.00316, 2023.
  32. Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval. In CVPR, 2019.
  33. How do humans sketch objects? ACM TOG, 2012.
  34. Visualizing higher-layer features of a deep network. Technical Report, Univeristé de Montréal, 2009.
  35. What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods. arXiv preprint arXiv:2112.04417, 2021.
  36. Understanding deep networks via extremal perturbations and smooth masks. In ICCV, 2019.
  37. Interpretable explanations of black boxes by meaningful perturbation. In ICCV, 2017.
  38. Sketchycoco: Image generation from freehand scene sketches. In CVPR, 2020.
  39. Sketch2mesh: Reconstructing and editing 3d shapes from sketches. In CVPR, 2021.
  40. A neural representation of sketch drawings. In ICLR, 2018.
  41. Deep residual learning for image recognition. In CVPR, 2016.
  42. Segmentations-leak: Membership inference attacks and defenses in semantic image segmentation. In ECCV, 2020.
  43. Aaron Hertzmann. Why do line drawings work? a realism hypothesis. Perception, 2020.
  44. Supportingreferenceimageryfordigitaldrawing. In ICCVW, 2021.
  45. Sketch-a-segmenter: Sketch-based photo segmenter generation. IEEE TIP, 2020.
  46. Study of ratling scales for subjective quality assessment of high definition video. IEEE TBC, 2010.
  47. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.
  48. Fooling detection alone is not enough: Adversarial attack against multiple object tracking. In ICLR, 2020.
  49. Bayesian safe policy learning with chance constrained optimization: Application to military security assessment during the vietnam war. arXiv preprint arXiv:2307.08840, 2023.
  50. Scaling up gans for text-to-image synthesis. In CVPR, 2023.
  51. Xrai: Better attributions through regions. In ICCV, 2019.
  52. Guided integrated gradients: An adaptive path method for removing noise. In CVPR, 2021.
  53. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
  54. Hive: Evaluating the human interpretability of visual explanations. In ECCV, 2022.
  55. The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, 2019.
  56. Picture that sketch: Photorealistic image generation from abstract sketches. In CVPR, 2023.
  57. Principles of explanatory debugging to personalize interactive machine learning. In IUI, 2015a.
  58. Principles of explanatory debugging to personalize interactive machine learning. In IUI, 2015b.
  59. An evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1902.00006, 2019.
  60. Photo-sketching: Inferring contour drawings from images. In WACV, 2019.
  61. Towards transferable targeted attack. In CVPR, 2020.
  62. Zero-shot everything sketch-based image retrieval, and in explainable style. In CVPR, 2023.
  63. Deflocnet: Deep image editing via flexible low-level controls. In CVPR, 2021.
  64. Deep sketch hashing: Fast free-hand sketch-based image retrieval. In CVPR, 2017.
  65. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023.
  66. Neural prototype trees for interpretable fine-grained image recognition. In CVPR, 2021.
  67. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. ICMLW, 2016.
  68. Feature visualization. Distill, 2017.
  69. Generalising fine-grained sketch-based image retrieval. In CVPR, 2019.
  70. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421, 2018.
  71. Adversarial imaging pipelines. In CVPR, 2021.
  72. Sketchxai: A first look at explainability for human sketches. In CVPR, 2023.
  73. Learning transferable visual models from natural language supervision. In ICML, 2021.
  74. Towards better understanding attribution methods. In CVPR, 2022.
  75. “why should i trust you?” explaining the predictions of any classifier. In KDD, 2016.
  76. High-resolution image synthesis with latent diffusion. In CVPR, 2022.
  77. Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 2019.
  78. Stylemeup: Towards style-agnostic sketch-based image retrieval. In CVPR, 2021.
  79. Clip for all things zero-shot sketch-based image retrieval, fine-grained or not. In CVPR, 2023.
  80. Evaluating the visualization of what a deep neural network has learned. IEEE TNNLS, 2016.
  81. The sketchy database: learning to retrieve badly drawn bunnies. ACM TOG, 2016.
  82. Pronimal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  83. Grad-cam: Visual explanations from deep networks via gradient-based localisation. In ICCV, 2017a.
  84. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017b.
  85. Do input gradients highlight discriminative features? In NeurIPS, 2021.
  86. Image information and visual quality. IEEE TIP, 2006.
  87. Zero-shot sketch-image hashing. In ICCV, 2018.
  88. Learning important features through propagating activation differences. In ICML, 2017.
  89. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLRW, 2014.
  90. Do users benefit from interpretable vision? a user study, baseline, and dataset. arXiv preprint arXiv:2204.11642, 2022.
  91. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017.
  92. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014.
  93. Striving for simplicity: The all convolutional net. In ICLRW, 2015.
  94. Axiomatic attribution for deep networks. In ICML, 2017.
  95. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013a.
  96. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013b.
  97. Diversity can be transferred: Output diversification for white- and black-box attacks. In NeurIPS, 2020.
  98. Sketch-guided object localization in natural images. In ECCV, 2020.
  99. Physically realizable adversarial examples for lidar object detection. In CVPR, 2020.
  100. Evaluation of post-hoc interpretability methods in time-series classification. Nature Machine Intelligence, 2023.
  101. Sketch your own gan. In CVPR, 2021.
  102. Learning and using the arrow of time. In CVPR, 2018.
  103. On the safety of interpretable machine learning: A maximum deviation approach. In NeurIPS, 2022.
  104. Sketchhairsalon: deep sketch-based hair image synthesis. ACM TOG, 2021.
  105. Sketchmate: Deep hashing for million-scale human sketch retrieval. In CVPR, 2018.
  106. Domain disentangled generative adversarial network for zero-shot sketch-based 3d shape retrieval. AAAI, 2022.
  107. Interactive liquid splash modeling by user sketches. ACM TOG, 2020.
  108. Semantic hierarchy emerges in deep generative representations for scene synthesis. IJCV, 2021.
  109. On the (in) fidelity and sensitivity of explanations. In NeurIPS, 2019.
  110. A zero-shot framework for sketch based image retrieval. In ECCV, 2018.
  111. Animating portrait line drawings from a single face photo and a speech signal. In ACM SIGGRAPH, 2022.
  112. Free-form image inpainting with gated convolution. In CVPR, 2019.
  113. Sketch-a-net that beats humans. In BMVC, 2015.
  114. Sketch me that shoe. In CVPR, 2016.
  115. Visualizing and understanding convolutional networks. In ECCV, 2014.
  116. Sketchedit: Mask-free local image manipulation with partial sketches. In CVPR, 2022.
  117. Towards adversarially robust object detection. In ICCV, 2019.
  118. Sketchnet: Sketch classification with web images. In CVPR, 2016.
  119. Top-down neural attention by excitation backprop. IJCV, 2018.
  120. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
  121. Sketch2model: View-aware 3d modeling from single free-hand sketches. In CVPR, 2021.
  122. Learning deep features for discriminative localization. In CVPR, 2016.
  123. Lg-gan: Label guided adversarial network for flexible targeted attack of point cloud-based deep networks. In CVPR, 2020.
  124. Sketchyscene: Richly-annotated scene sketches. In ECCV, 2018.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube