Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? (2403.07203v2)

Published 11 Mar 2024 in cs.CV

Abstract: In this paper, we propose a novel abstraction-aware sketch-based image retrieval framework capable of handling sketch abstraction at varied levels. Prior works had mainly focused on tackling sub-factors such as drawing style and order, we instead attempt to model abstraction as a whole, and propose feature-level and retrieval granularity-level designs so that the system builds into its DNA the necessary means to interpret abstraction. On learning abstraction-aware features, we for the first-time harness the rich semantic embedding of pre-trained StyleGAN model, together with a novel abstraction-level mapper that deciphers the level of abstraction and dynamically selects appropriate dimensions in the feature matrix correspondingly, to construct a feature matrix embedding that can be freely traversed to accommodate different levels of abstraction. For granularity-level abstraction understanding, we dictate that the retrieval model should not treat all abstraction-levels equally and introduce a differentiable surrogate Acc.@q loss to inject that understanding into the system. Different to the gold-standard triplet loss, our Acc.@q loss uniquely allows a sketch to narrow/broaden its focus in terms of how stringent the evaluation should be - the more abstract a sketch, the less stringent (higher q). Extensive experiments depict our method to outperform existing state-of-the-arts in standard SBIR tasks along with challenging scenarios like early retrieval, forensic sketch-photo matching, and style-invariant retrieval.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (90)
  1. Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? In CVPR, 2019.
  2. Labels4Free: Unsupervised Segmentation Using StyleGAN. In CVPR, 2021.
  3. Only a Matter of Style: Age Transformation Using a Style-Based Regression Model. ACM TOG, 2021a.
  4. ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement. In CVPR, 2021b.
  5. Abstracting Sketches through Simple Primitives. In ECCV, 2022.
  6. Style and Abstraction in Portrait Sketching. ACM TOG, 2013.
  7. Memetically Optimized MCWLD for Matching Sketches With Digital Face Images. IEEE TIFS, 2012.
  8. Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval. In CVPR, 2020.
  9. More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval. In CVPR, 2021a.
  10. Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting. In CVPR, 2021b.
  11. Sketching Without Worrying: Noise-Tolerant Sketch-Based Image Retrieval. In CVPR, 2022a.
  12. Adaptive Fine-Grained Sketch-Based Image Retrieval. In ECCV, 2022b.
  13. Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings. In CVPR, 2023.
  14. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In ICLR, 2019.
  15. Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval. In ECCV, 2020.
  16. SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis. In CVPR, 2018.
  17. Relational Deep Feature Learning for Heterogeneous Face Recognition. IEEE TIFS, 2020.
  18. Partially Does It: Towards Scene-Level FG-SBIR With Partial Input. In CVPR, 2022.
  19. SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text. In CVPR, 2023a.
  20. What Can Human Sketches Do for Object Detection? In CVPR, 2023b.
  21. LiveSketch: Query perturbations for guided sketch-based visual search. In CVPR, 2019.
  22. BézierSketch: A generative model for scalable vector sketches. In ECCV, 2020.
  23. SketchODE: Learning Neural Sketch Representation in Continuous Time. In ICLR, 2021a.
  24. Cloud2Curve: Generation and Vectorization of Parametric Sketches. In CVPR, 2021b.
  25. Residual Compensation Networks for Heterogeneous Face Recognition. In AAAI, 2019.
  26. Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval. In CVPR, 2019.
  27. SoDeep: a Sorting Deep net to learn ranking loss surrogates. In CVPR, 2019.
  28. StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN. In BMVC, 2021.
  29. Sketch-based Image Retrieval using Generative Adversarial Networks. In ACM MM, 2017.
  30. Dimensionality Reduction by Learning an Invariant Mapping. In CVPR, 2006.
  31. Scops: Self-supervised co-part segmentation. In CVPR, 2019.
  32. Study of Rating Scales for Subjective Quality Assessment of High-Definition Video. IEEE TBC, 2010.
  33. Image-to-Image Translation with Conditional Adversarial Networks. In CVPR, 2017.
  34. Categorical reparameterization with gumbel-softmax. In ICLR, 2016.
  35. Scaling up GANs for Text-to-Image Synthesis. In CVPR, 2023.
  36. A Style-Based Generator Architecture for Generative Adversarial Networks. In CVPR, 2019.
  37. Analyzing and Improving the Image Quality of StyleGAN. In CVPR, 2020.
  38. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
  39. Picture that Sketch: Photorealistic Image Generation from Abstract Sketches. In CVPR, 2023.
  40. You’ll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval. In CVPR, 2024a.
  41. Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers. In CVPR, 2024b.
  42. It’s All About Your Sketch: Democratising Sketch Control in Diffusion Models. In CVPR, 2024c.
  43. SketchGAN: Joint Sketch Completion and Recognition with Generative Adversarial Network. In CVPR, 2019.
  44. DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos. ACM TOG, 2022.
  45. Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval. In CVPR, 2017.
  46. Unsupervised Sketch-to-Photo Synthesis. In ECCV, 2020.
  47. Learning Deep Sketch Abstraction. In CVPR, 2018.
  48. Goal-Driven Sequential Data Abstraction. In ICCV, 2019.
  49. Deep Metric Learning via Lifted Structured Feature Embedding. In CVPR, 2016.
  50. Solving Mixed-modal Jigsaw Puzzle for Fine-Grained Sketch-Based Image Retrieval. In CVPR, 2020.
  51. Deep Face Recognition. In BMVC, 2015.
  52. StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery. In ICCV, 2021.
  53. Recall@k Surrogate Loss With Large Batches and Similarity Mixup. In CVPR, 2022.
  54. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In ICLR, 2016.
  55. Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation. In CVPR, 2021.
  56. Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval. In BMVC, 2020.
  57. StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval. In CVPR, 2021.
  58. Sketch3T: Test-Time Training for Zero-Shot SBIR. In CVPR, 2022.
  59. CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not. In CVPR, 2023a.
  60. Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR. In CVPR, 2023b.
  61. StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets. In SIGGRAPH, 2022.
  62. FaceNet: A Unified Embedding for Face Recognition and Clustering. In CVPR, 2015.
  63. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR, 2015.
  64. Fine-Grained Image Retrieval: the Text/Sketch Input Dilemma. In BMVC, 2017a.
  65. Deep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval. In ICCV, 2017b.
  66. Learning to Sketch with Shortcut Cycle Consistency. In CVPR, 2018.
  67. Rethinking the Inception Architecture for Computer Vision . In CVPR, 2016.
  68. The iNaturalist Species Classification and Detection Dataset. In CVPR, 2018.
  69. CLIPasso: Semantically-Aware Object Sketching. ACM TOG, 2022.
  70. Sketch Your Own GAN. In ICCV, 2021.
  71. Fine-grained image analysis with deep learning: A survey. IEEE TPAMI, 2021.
  72. Distance Metric Learning for Large Margin Nearest Neighbor Classification. JMLR, 2009.
  73. A Discriminative Feature Learning Approach for Deep Face Recognition. In ECCV, 2016.
  74. Sampling Matters in Deep Embedding Learning. In ICCV, 2017.
  75. Coupled Deep Learning for Heterogeneous Face Recognition. In AAAI, 2018.
  76. Local Shannon entropy measure with statistical tests for image randomness. Information Sciences, 2013.
  77. Empirical Evaluation of Rectified Activations in Convolutional Network. arXiv preprint arXiv:1505.00853, 2015.
  78. Domain Disentangled Generative Adversarial Network for Zero-Shot Sketch-Based 3D Shape Retrieval. In AAAI, 2022.
  79. Generative Hierarchical Features from Synthesizing Images. In CVPR, 2021.
  80. Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis. IJCV, 2021a.
  81. SketchAA: Abstract Representation for Abstract Sketches. In ICCV, 2021b.
  82. Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches. In ECCV, 2020.
  83. Fine-Grained Visual Comparisons with Local Learning. In CVPR, 2014.
  84. Generative Image Inpainting with Contextual Attention. In CVPR, 2018.
  85. Sketch Me That Shoe. In CVPR, 2016.
  86. Sketch-a-Net: A Deep Neural Network that Beats Humans. IJCV, 2017.
  87. SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches. In CVPR, 2022.
  88. Coupled information-theoretic encoding for face photo-sketch recognition. In CVPR, 2011.
  89. Generative Visual Manipulation on the Natural Image Manifold. In ECCV, 2016.
  90. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. In ICCV, 2017.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube