Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2406.09403v3)

Abstract: Humans draw to facilitate reasoning: we draw auxiliary lines when solving geometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacity working memory. However, such actions are missing in current multimodal LLMs (LMs). Current chain-of-thought and tool-use paradigms only use text as intermediate reasoning steps. In this work, we introduce Sketchpad, a framework that gives multimodal LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts planning and reasoning according to the visual artifacts it has drawn. Different from prior work, which uses text-to-image models to enable LMs to draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is closer to human sketching and better facilitates reasoning. Sketchpad can also use specialist vision models during the sketching process (e.g., draw bounding boxes with object detection models, draw masks with segmentation models), to further enhance visual perception and reasoning. We experiment with a wide range of math tasks (including geometry, functions, graphs, and chess) and complex visual reasoning tasks. Sketchpad substantially improves performance on all tasks over strong base models with no sketching, yielding an average gain of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in https://visualsketchpad.github.io/.

Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal LLMs

Overview

The paper "Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal LLMs" introduces a novel framework named Sketchpad, which integrates visual sketching into the reasoning process of multimodal LLMs (LMs). The primary motivation behind this research is to enhance the cognitive capabilities of LMs by enabling them to incorporate visual sketches as intermediate steps, similar to how humans use auxiliary lines, diagrams, and notations in problem-solving. Sketchpad allows these models to draw on a virtual sketchpad using lines, boxes, and other marks, providing a more human-like approach to reasoning and planning in tasks that require visuo-spatial understanding.

Key Contributions

1. Framework Design: Sketchpad provides a detailed mechanism for multimodal LMs to generate and utilize visual artifacts during the reasoning process. This is a significant departure from the traditional chain-of-thought frameworks that rely solely on text.
2. Specialist Vision Models: The framework supports the integration of specialist vision models, such as object detection and segmentation models, to enhance the perception and reasoning abilities of the LMs.
3. Wide Range of Tasks: The authors demonstrate the effectiveness of Sketchpad across diverse tasks, including mathematical problem-solving (geometry, functions, graph algorithms, and strategy games) and complex visual reasoning tasks from benchmarks like $V^*$Bench, BLINK, and MMVP.

Numerical Results

The experimental results detailed in the paper highlight substantial performance improvements across a variety of benchmarks:

• On mathematical tasks, Sketchpad yields an average performance gain of 12.7% in math tasks and 8.6% in vision tasks over strong baseline models.
• Notably, GPT-4o integrated with Sketchpad sets new state-of-the-art results across all tested benchmarks, with specific improvements showcased as follows:
  • $V^*$Bench: 80.3%
  • BLINK spatial reasoning: 83.9%
  • Visual correspondence: 80.8%

Implications

Practical Implications:

The incorporation of visual sketching in multimodal LMs can significantly enhance the usability and application of such models in real-world scenarios. For instance, in education, LMs equipped with Sketchpad can aid in teaching complex mathematical concepts through visual demonstrations. In professional fields like engineering or architecture, these models can assist in visualizing solutions and planning more effectively.

Theoretical Implications:

From a theoretical perspective, the introduction of Sketchpad paves the way for more advanced multimodal intelligence systems. The ability to autonomously generate and adapt visual plans demonstrates a leap towards achieving more human-like problem-solving capabilities in artificial intelligence.

Future Directions

The development of Sketchpad opens several avenues for future research and development:

1. Integration with Training Paradigms: Future models could be trained with native integration of Sketchpad-like functionality, potentially leading to even more significant improvements in performance and versatility.
2. Expansion to Other Domains: Beyond mathematical and simple visual tasks, the principles of Sketchpad could be extended to more specialized domains such as medical imaging, robotics, and complex scientific computations.
3. Enhanced Computational Efficiency: While the current implementation demonstrates substantial benefits, optimization techniques could be employed to reduce the computational overhead, making Sketchpad more practical for a wider range of applications.

Conclusion

The paper presents a compelling case for enhancing multimodal LLMs with visual sketching capabilities through the Sketchpad framework. By leveraging visual artifacts as part of the reasoning process, Sketchpad not only improves performance across a range of tasks but also aligns closer with human cognitive strategies. This research marks an important step towards creating more intuitive and powerful AI systems capable of addressing complex reasoning challenges in a holistic manner.