Papers
Topics
Authors
Recent
Search
2000 character limit reached

Make VLM Recognize Visual Hallucination on Cartoon Character Image with Pose Information

Published 22 Mar 2024 in cs.CV, cs.AI, cs.LG, and cs.MM | (2403.15048v3)

Abstract: Leveraging large-scale Text-to-Image (TTI) models have become a common technique for generating exemplar or training dataset in the fields of image synthesis, video editing, 3D reconstruction. However, semantic structural visual hallucinations involving perceptually severe defects remain a concern, especially in the domain of non-photorealistic rendering (NPR) such as cartoons and pixelization-style character. To detect these hallucinations in NPR, We propose a novel semantic structural hallucination detection system using Vision-LLM (VLM). Our approach is to leverage the emerging capability of LLM, in-context learning which denotes that VLM has seen some examples by user for specific downstream task, here hallucination detection. Based on in-context learning, we introduce pose-aware in-context visual learning (PA-ICVL) which improve the overall performance of VLM by further inputting visual data beyond prompts, RGB images and pose information. By incorporating pose guidance, we enable VLMs to make more accurate decisions. Experimental results demonstrate significant improvements in identifying visual hallucinations compared to baseline methods relying solely on RGB images. Within selected two VLMs, GPT-4v, Gemini pro vision, our proposed PA-ICVL improves the hallucination detection with 50% to 78%, 57% to 80%, respectively. This research advances a capability of TTI models toward real-world applications by mitigating visual hallucinations via in-context visual learning, expanding their potential in non-photorealistic domains. In addition, it showcase how users can boost the downstream-specialized capability of open VLM by harnessing additional conditions. We collect synthetic cartoon-hallucination dataset with TTI models, this dataset and final tuned VLM will be publicly available.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  3. Webtoonme: A data-centric approach for full-body portrait stylization. In SIGGRAPH Asia 2022 Technical Communications, pages 1–4. ACM, 2022.
  4. Ffhq-uv: Normalized facial uv-texture dataset for 3d face reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 362–371, 2023.
  5. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023.
  6. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.
  7. Transfer learning based parameterized 3d mesh deformation with 2d stylized cartoon character. KSII Transactions on Internet & Information Systems, 17(11), 2023.
  8. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
  9. Pixart-δ𝛿\deltaitalic_δ: Fast and controllable image generation with latent consistency models, 2024.
  10. Pixart-α𝛼\alphaitalic_α: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023.
  11. Trb: a novel triplet representation for understanding 2d human body. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9479–9488, 2019.
  12. Pose anything: A graph-based approach for category-agnostic pose estimation. arXiv preprint arXiv:2311.17891, 2023.
  13. Scaling up vision-language pre-training for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17980–17989, 2022.
  14. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232, 2023.
  15. Minecraft-ify: Minecraft style image generation with text-guided image editing for in-game application. arXiv preprint arXiv:2402.05448, 2024.
  16. Toonaging: Face re-aging upon artistic portrait style transfer. arXiv preprint arXiv:2402.02733, 2024.
  17. Cross-domain style mixing for face cartoonization. arXiv preprint arXiv:2205.12450, 2022.
  18. Karlo-v1.0.alpha on coyo-100m and cc15m. https://github.com/kakaobrain/karlo, 2022.
  19. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023.
  20. Microsoft coco: Common objects in context, 2015.
  21. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
  22. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023.
  23. Lcm-lora: A universal stable-diffusion acceleration module. arXiv preprint arXiv:2311.05556, 2023.
  24. MMPose Contributors. OpenMMLab Pose Estimation Toolbox and Benchmark. https://github.com/open-mmlab/mmpose, Aug. 2020. License: Apache-2.0.
  25. Video face re-aging: Toward temporally consistent face re-aging. arXiv preprint arXiv:2311.11642, 2023.
  26. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
  27. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
  28. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  29. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  30. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693–5703, 2019.
  31. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  32. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6830–6839, 2023.
  33. Make your own sprites: Aliasing-aware and cell-controllable pixelization. ACM Transactions on Graphics (TOG), 41(6):1–16, 2022.
  34. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817, 2024.
  35. Pastiche master: Exemplar-based high-resolution portrait style transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7693–7702, 2022.
  36. Articulated pose estimation with flexible mixtures-of-parts. In CVPR 2011, pages 1385–1392. IEEE, 2011.
  37. Styleavatar3d: Leveraging image-text diffusion models for high-fidelity 3d avatar generation. arXiv preprint arXiv:2305.19012, 2023.
  38. Vision-language models for vision tasks: A survey. arXiv preprint arXiv:2304.00685, 2023.
  39. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.
  40. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36, 2024.
  41. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
  42. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now