DiffAugment: Diffusion based Long-Tailed Visual Relationship Recognition (2401.01387v2)
Abstract: The task of Visual Relationship Recognition (VRR) aims to identify relationships between two interacting objects in an image and is particularly challenging due to the widely-spread and highly imbalanced distribution of <subject, relation, object> triplets. To overcome the resultant performance bias in existing VRR approaches, we introduce DiffAugment -- a method which first augments the tail classes in the linguistic space by making use of WordNet and then utilizes the generative prowess of Diffusion Models to expand the visual space for minority classes. We propose a novel hardness-aware component in diffusion which is based upon the hardness of each <S,R,O> triplet and demonstrate the effectiveness of hardness-aware diffusion in generating visual embeddings for the tail classes. We also propose a novel subject and object based seeding strategy for diffusion sampling which improves the discriminative capability of the generated visual embeddings. Extensive experimentation on the GQA-LT dataset shows favorable gains in the subject/object and relation average per-class accuracy using Diffusion augmented samples.
- Combining Local Context and WordNet Similarity for Word Sense Identification. In WordNet: An Electronic Lexical Database. The MIT Press, 1998.
- Exploring long tail visual relationship recognition with large vocabulary, 2021.
- Blended latent diffusion. ACM Transactions on Graphics, 42(4):1–11, 2023.
- Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022.
- Reltransformer: A transformer-based long-tail visual relationship recognition, 2022.
- An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
- Denoising diffusion probabilistic models, 2020.
- Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. arXiv preprint arXiv:2204.09934, 2022.
- Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019.
- Decoupling representation and classifier for long-tailed recognition, 2020.
- Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761, 2020.
- Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016.
- Priorgrad: Improving conditional denoising diffusion models with data-driven adaptive prior. arXiv preprint arXiv:2106.06406, 2021.
- Q-diffusion: Quantizing diffusion models, 2023.
- Focal loss for dense object detection, 2018.
- Large-scale long-tailed recognition in an open world, 2019.
- SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
- George A. Miller. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41, 1995.
- Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection, 2023.
- Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pages 8599–8608. PMLR, 2021a.
- Diffusion-based voice conversion with fast maximum likelihood sampling scheme. arXiv preprint arXiv:2109.13821, 2021b.
- Class-balancing diffusion models, 2023.
- Learning transferable visual models from natural language supervision, 2021.
- Faster r-cnn: Towards real-time object detection with region proposal networks, 2016.
- High-resolution image synthesis with latent diffusion models, 2022.
- Image super-resolution via iterative refinement, 2021.
- Generating images of rare concepts using pre-trained diffusion models, 2023.
- Relay backpropagation for effective learning of deep convolutional neural networks, 2016.
- Very deep convolutional networks for large-scale image recognition, 2015.
- Curriculum learning: A survey, 2022.
- Equalization loss for long-tailed object recognition, 2020.
- Improved vector quantized diffusion models, 2023.
- Attention is all you need, 2023.
- Manifold mixup: Better representations by interpolating hidden states, 2019.
- Vision transformer-based spatially conditioned graphs for long tail visual relationship recognition cvpr 2023 ltvrr challenge. arXiv preprint arXiv, 2023.
- Spatially conditioned graphs for detecting human-object interactions, 2021.
- Large-scale visual relationship understanding, 2019.
- Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations, 2023.
- Fast sampling of diffusion models via operator learning, 2023.