Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

EGTR: Extracting Graph from Transformer for Scene Graph Generation (2404.02072v5)

Published 2 Apr 2024 in cs.CV and cs.LG

Abstract: Scene Graph Generation (SGG) is a challenging task of detecting objects and predicting relationships between objects. After DETR was developed, one-stage SGG models based on a one-stage object detector have been actively studied. However, complex modeling is used to predict the relationship between objects, and the inherent relationship between object queries learned in the multi-head self-attention of the object detector has been neglected. We propose a lightweight one-stage SGG model that extracts the relation graph from the various relationships learned in the multi-head self-attention layers of the DETR decoder. By fully utilizing the self-attention by-products, the relation graph can be extracted effectively with a shallow relation extraction head. Considering the dependency of the relation extraction task on the object detection task, we propose a novel relation smoothing technique that adjusts the relation label adaptively according to the quality of the detected objects. By the relation smoothing, the model is trained according to the continuous curriculum that focuses on object detection task at the beginning of training and performs multi-task learning as the object detection performance gradually improves. Furthermore, we propose a connectivity prediction task that predicts whether a relation exists between object pairs as an auxiliary task of the relation extraction. We demonstrate the effectiveness and efficiency of our method for the Visual Genome and Open Image V6 datasets. Our code is publicly available at https://github.com/naver-ai/egtr.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. End-to-end object detection with transformers. In ECCV, pages 213–229, 2020.
  2. Context refinement for object detection. In ECCV, pages 71–86, 2018.
  3. Reltr: Relation transformer for scene graph generation. IEEE TPAMI, 2023.
  4. Bgt-net: Bidirectional gru transformer network for scene graph generation. In CVPR, pages 2150–2159, 2021.
  5. Image captioning with scene-graph based semantic concepts. In ICMLC, pages 225–229, 2018.
  6. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016.
  7. Relation networks for object detection. In CVPR, pages 3588–3597, 2018.
  8. Image retrieval using scene graphs. In CVPR, pages 3668–3678, 2015.
  9. Iterative scene graph generation. In NeurIPS, 2022.
  10. Dense relational captioning: Triple-stream networks for relationship-based captioning. In CVPR, pages 6271–6280, 2019.
  11. Relation transformer network. ECCV, pages 422–439, 2022.
  12. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32–73, 2017.
  13. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 128(7):1956–1981, 2020.
  14. Relation-aware graph attention network for visual question answering. In ICCV, pages 10313–10322, 2019.
  15. Compositional feature augmentation for unbiased scene graph generation. In ICCV, pages 21685–21695, 2023.
  16. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In CVPR, pages 11109–11119, 2021.
  17. Sgtr: End-to-end scene graph generation with transformer. In CVPR, pages 19486–19496, 2022.
  18. Feature pyramid networks for object detection. In CVPR, pages 2117–2125, 2017a.
  19. Focal loss for dense object detection. In CVPR, pages 2980–2988, 2017b.
  20. Gps-net: Graph property sensing network for scene graph generation. In CVPR, pages 3746–3753, 2020.
  21. Fully convolutional scene graph generation. In CVPR, pages 11546–11556, 2021.
  22. Ssd: Single shot multibox detector. In ECCV, pages 21–37, 2016.
  23. Structure inference net: Object detection using scene-level context and instance-level relationships. In CVPR, pages 6985–6994, 2018.
  24. Decoupled weight decay regularization. In ICLR, 2019.
  25. Visual relationship detection with language priors. In ECCV, pages 852–869, 2016.
  26. Context-aware scene graph generation with seq2seq transformers. In ICCV, pages 15931–15941, 2021.
  27. Environment-invariant curriculum relation learning for fine-grained scene graph generation. In ICCV, pages 13296–13307, 2023.
  28. Pixels to graphs by associative embedding. NeurIPS, 30, 2017.
  29. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS, 28, 2015.
  30. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR, pages 658–666, 2019.
  31. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language, pages 70–80. Association for Computational Linguistics, 2015.
  32. Relationformer: A unified framework for image-to-graph generation. In ECCV, pages 422–439, 2022.
  33. Vision relation transformer for unbiased scene graph generation. In ICCV, pages 21882–21893, 2023.
  34. Energy-based learning for scene graph generation. In CVPR, pages 13936–13945, 2021.
  35. Sparse r-cnn: End-to-end object detection with learnable proposals. In CVPR, pages 14454–14463, 2021.
  36. Learning to compose dynamic tree structures for visual contexts. In CVPR, pages 6619–6628, 2019.
  37. Unbiased scene graph generation from biased training. In CVPR, pages 3716–3725, 2020.
  38. Structured sparse r-cnn for direct scene graph generation. In CVPR, pages 19437–19446, 2022.
  39. Attention is all you need. NeurIPS, 30, 2017.
  40. Linknet: Relational embedding for scene graph. NeurIPS, 31, 2018.
  41. Aggregated residual transformations for deep neural networks. In CVPR, pages 1492–1500, 2017.
  42. Scene graph generation by iterative message passing. In CVPR, pages 5410–5419, 2017.
  43. Graph r-cnn for scene graph generation. In ECCV, pages 670–685, 2018.
  44. Auto-encoding scene graphs for image captioning. In CVPR, pages 10685–10694, 2019.
  45. Neural motifs: Scene graph parsing with global context. In CVPR, pages 5831–5840, 2018.
  46. An empirical study on leveraging scene graphs for visual question answering. In BMVC, 2019a.
  47. Visual translation embedding network for visual relation detection. In CVPR, page 3107–3115, 2017.
  48. Graphical contrastive losses for scene graph parsing. In CVPR, pages 11535–11543, 2019b.
  49. Learning to generate scene graph from head to tail. In ICME, pages 1–6. IEEE, 2022.
  50. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Jinbae Im (8 papers)
  2. Nokyung Park (5 papers)
  3. Hyungmin Lee (1 paper)
  4. Seunghyun Park (26 papers)
  5. Jeongyeon Nam (6 papers)
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now