Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Cycle-Consistency Learning for Captioning and Grounding (2312.15162v1)

Published 23 Dec 2023 in cs.CV

Abstract: We present that visual grounding and image captioning, which perform as two mutually inverse processes, can be bridged together for collaborative training by careful designs. By consolidating this idea, we introduce CyCo, a cyclic-consistent learning framework to ameliorate the independent training pipelines of visual grounding and image captioning. The proposed framework (1) allows the semi-weakly supervised training of visual grounding; (2) improves the performance of fully supervised visual grounding; (3) yields a general captioning model that can describe arbitrary image regions. Extensive experiments show that our fully supervised grounding model achieves state-of-the-art performance, and the semi-weakly supervised one also exhibits competitive performance compared to the fully supervised counterparts. Our image captioning model has the capability to freely describe image regions and meanwhile shows impressive performance on prevalent captioning benchmarks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Spice: Semantic propositional image caption evaluation. In ECCV.
  2. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR.
  3. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In ACL Workshop.
  4. Say as you wish: Fine-grained control of image caption generation with abstract scene graphs. In CVPR.
  5. Uniter: Universal image-text representation learning. In ECCV.
  6. Show, control and tell: A framework for generating controllable and grounded captions. In CVPR.
  7. Transvg: End-to-end visual grounding with transformers. In ICCV.
  8. Transvg++: End-to-end visual grounding with language conditioned vision transformer. IEEE TPAMI.
  9. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  10. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
  11. Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone. arXiv preprint arXiv:2206.07643.
  12. Large-scale adversarial training for vision-and-language representation learning. In NeurIPS.
  13. Mscap: Multi-style image captioning with unpaired stylized text. In CVPR.
  14. Dual learning for machine translation. In NeurIPS.
  15. Scaling up vision-language pre-training for image captioning. arXiv preprint arXiv:2111.12233.
  16. Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training. In AAAI.
  17. Attention on attention for image captioning. In ICCV.
  18. Densecap: Fully convolutional localization networks for dense captioning. In CVPR.
  19. MDETR-modulated detection for end-to-end multi-modal understanding. In ICCV.
  20. Deep visual-semantic alignments for generating image descriptions. In CVPR.
  21. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP.
  22. A region-based image caption generator with refined descriptions. Neurocomputing, 272: 416–424.
  23. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123(1): 32–73.
  24. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. arXiv preprint arXiv:2201.12086.
  25. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning. In ACL.
  26. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV.
  27. Microsoft coco: Common objects in context. In ECCV.
  28. Learning to assemble neural module tree networks for visual grounding. In ICCV.
  29. Relation-aware instance refinement for weakly supervised visual grounding. In CVPR.
  30. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
  31. Multi-task collaborative network for joint referring expression comprehension and segmentation. In CVPR.
  32. Generation and comprehension of unambiguous object descriptions. In CVPR.
  33. Im2text: Describing images using 1 million captioned photographs. In NeurIPS.
  34. X-linear attention networks for image captioning. In CVPR.
  35. Bleu: a method for automatic evaluation of machine translation. In ACL.
  36. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV.
  37. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020.
  38. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR.
  39. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114.
  40. Cycle-consistency for robust visual question answering. In CVPR.
  41. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL.
  42. Language adaptive weight generation for multi-task visual grounding. In CVPR.
  43. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR.
  44. Attention is all you need. In NeurIPS.
  45. Cider: Consensus-based image description evaluation. In CVPR.
  46. Show and tell: A neural image caption generator. In CVPR.
  47. Improving weakly supervised visual grounding by contrastive knowledge distillation. In CVPR.
  48. Unsupervised deep tracking. In CVPR.
  49. Efficient image captioning for edge devices. In AAAI.
  50. Controllable image captioning via prompting. In AAAI.
  51. Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML.
  52. Learning correspondence from the cycle-consistency of time. In CVPR.
  53. End-to-End Transformer Based Model for Image Captioning. In AAAI.
  54. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904.
  55. Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoning. In CVPR.
  56. Dynamic graph attention for referring expression comprehension. In ICCV.
  57. A fast and accurate one-stage approach to visual grounding. In ICCV.
  58. Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Grounding. In CVPR.
  59. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934.
  60. Modeling context in referring expressions. In ECCV.
  61. A joint speaker-listener-reinforcer model for referring expressions. In CVPR.
  62. Vinvl: Revisiting visual representations in vision-language models. In CVPR.
  63. Unified vision-language pre-training for image captioning and vqa. In AAAI.
  64. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Ning Wang (300 papers)
  2. Jiajun Deng (75 papers)
  3. Mingbo Jia (3 papers)
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now