SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Images (2412.17606v1)

Abstract: Building a large-scale figure QA dataset requires a considerable amount of work, from gathering and selecting figures to extracting attributes like text, numbers, and colors, and generating QAs. Although recent developments in LLMs have led to efforts to synthesize figures, most of these focus primarily on QA generation. Additionally, creating figures directly using LLMs often encounters issues such as code errors, similar-looking figures, and repetitive content in figures. To address this issue, we present SBSFigures (Stage-by-Stage Synthetic Figures), a dataset for pre-training figure QA. Our proposed pipeline enables the creation of chart figures with complete annotations of the visualized data and dense QA annotations without any manual annotation process. Our stage-by-stage pipeline makes it possible to create diverse topic and appearance figures efficiently while minimizing code errors. Our SBSFigures demonstrate a strong pre-training effect, making it possible to achieve efficient training with a limited amount of real-world chart data starting from our pre-trained weights.

An Expert Overview of "SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Images"

The paper "SBS Figures: Pre-training Figure QA from Stage-by-Stage Synthesized Images" introduces a novel approach to addressing the challenges in building large-scale figure-based Question Answering (QA) datasets. This research is driven by the necessity for models capable of accurately interpreting and reasoning with figures found in documents. The authors propose a dataset named SBS Figures, which is generated via a structured pipeline focused on synthesizing diverse and detailed chart figures paired with question-answer annotations. The central contribution lies in the proposed stage-by-stage synthesis pipeline that automates and enhances the creation of training data for figure-based QA models.

Methodology and Dataset Generation

The paper outlines a three-module pipeline aimed at generating figure images from abstract data in JSON format. This process involves:

1. Data Topic and Content Generation: Using LLMs, the authors first generate topics and relevant data contents in a structured JSON format that describe what the visualized data should represent.
2. Figure Rendering: The data generated is then used to create figure images using pre-defined, error-free Python scripts. The process allows for the randomization of visual components such as fonts, legend positioning, and marker styles, enhancing diversity without manual intervention.
3. QA Pair Generation: The final module generates dense and accurate QA pairs from the visualization data, leveraging LLMs for sophisticated and reliable question generation without relying on Optical Character Recognition (OCR).

The SBS Figures dataset, comprising 1 million figures and 4.2 million QA pairs, is designed to cover a broad spectrum of topics and visual styles. This comprehensive dataset is made publicly available to facilitate further research and improve pre-training in figure-based QA tasks.

Results and Implications

Empirical evaluations underscore the efficacy of pre-training with SBS Figures, demonstrating superior performance on real-world figure QA datasets such as ChartQA, PlotQA, and FigureQA. Models pre-trained on SBS Figures exhibit enhanced capability in understanding complex visual information and reasoning over diverse data topics compared to those trained from scratch or on other synthetic datasets.

The paper also explores several key factors influencing the pre-training effectiveness of synthetic datasets, such as the diversity of figure appearances, the quality of generated QA pairs, and the scale of synthesized data. These findings highlight the importance of detailed and varied training data in developing models with robust generalization abilities.

By successfully synthesizing a large and diverse set of chart figures and QA annotations, this research provides significant insights into leveraging synthetic data for enhancing model training in AI. The SBS Figures dataset and its underlying synthesis pipeline represent a valuable resource for future advancements in document understanding and figure-based reasoning tasks.

Future Directions

The methodology introduced in this paper opens avenues for not only improving figure-based QA systems but also optimizing synthetic data generation mechanisms in AI research. Potential future work includes extending the synthesis pipeline to accommodate even more figure types and exploring its integration with real-world data to further validate and refine model capabilities. Additionally, fine-tuning hyperparameters and exploring larger datasets could lead to even more pronounced gains in performance.

In summary, the development of the SBS Figures dataset and the accompanying methodology reflects a significant advancement in automating figure QA dataset creation. The structured, step-by-step approach offers a scalable, efficient, and versatile solution to generating complex training data, paving the way for more effective model training and robust AI systems capable of nuanced figure interpretation and reasoning.