ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution (2502.00989v1)

Abstract: LLMs can perform chart question-answering tasks but often generate unverified hallucinated responses. Existing answer attribution methods struggle to ground responses in source charts due to limited visual-semantic context, complex visual-text alignment requirements, and difficulties in bounding box prediction across complex layouts. We present ChartCitor, a multi-agent framework that provides fine-grained bounding box citations by identifying supporting evidence within chart images. The system orchestrates LLM agents to perform chart-to-table extraction, answer reformulation, table augmentation, evidence retrieval through pre-filtering and re-ranking, and table-to-chart mapping. ChartCitor outperforms existing baselines across different chart types. Qualitative user studies show that ChartCitor helps increase user trust in Generative AI by providing enhanced explainability for LLM-assisted chart QA and enables professionals to be more productive.

• The paper introduces ChartCitor, a multi-agent framework using LLMs to improve chart question-answering accuracy and verifiability through fine-grained visual grounding and evidence.
• ChartCitor utilizes specialized agents for tasks such as converting charts to tables, reformulating answers, generating entity captions, retrieving table cells, and localizing cells visually.
• Evaluation results demonstrate that ChartCitor significantly outperforms existing baselines in visual chart understanding (IoU metric), enhancing the trustworthiness of AI-generated responses for data-intensive professional applications.

ChartCitor: Enhancing Chart Question-Answering with Multi-Agent LLM Retrieval

The paper introduces ChartCitor, a novel multi-agent framework aimed at addressing the challenges in chart-based question-answering (ChartQA) using LLMs. ChartCitor is designed to overcome the common pitfalls of LLMs, such as the generation of unverifiable and inaccurate responses when reasoning over chart data. This research highlights the necessity of providing fine-grained visual grounding to enhance the accuracy and reliability of responses generated by LLMs.

Key Components and Methodology

The architecture of ChartCitor is a multi-agent system that tackles the complexity of chart interpretation through a series of specialized components. Here are the main agents utilized within the framework:

1. Chart2Table Extraction Agent: This agent utilizes GPT-4V to convert visual chart data into structured formats such as CSV or HTML. The agent employs techniques like visual self-reflection to ensure consistency and accuracy between the extracted data and the original chart, iteratively refining outputs when discrepancies are found.
2. Answer Reformulation Agent: This agent dissects complex answers into comprehensible reasoning steps, ensuring logical coherence and facilitating accurate evidence retrieval. It reformulates responses to match the structure of the extracted table data, enabling precise citation.
3. Entity Captioning Agent: By generating detailed row, column, and cell-level contextual descriptions, this agent enriches the semantic understanding of tabular data, aiding in the extraction of relevant evidence and accurate mapping to visual chart elements.
4. Table Cell Retrieval: A two-step process that includes:
   • LLM Pre-filtering Agent: This pre-selects rows and columns that are relevant to the answer, improving the efficiency of downstream processing by reducing noise.
   • LLM Re-ranking Agent: This ranks potential evidence cells using RankGPT to ensure the fidelity and logical integration of evidence into the citation mechanism.
5. Cell Localization Agent: Combines object detection models like DETR with LLMs to map selected table cells to their corresponding visual components in the chart, using bounding box annotations to highlight relevant chart regions.

Evaluation and Results

The effectiveness of ChartCitor is measured using Intersection over Union (IoU) as a metric, demonstrating substantial improvements over existing baselines. Specifically, the framework outperforms models like Kosmos-2 and LISA in visual chart understanding across various chart types by margins of 9-15%. This performance underscores its robustness in handling complex multimodal tasks, where traditional approaches fall short, particularly in non-rectangular visual contexts such as pie charts.

Implications and Future Directions

The introduction of ChartCitor presents significant advancements for applications that require enhanced trust and reliability in AI-generated responses to chart-based queries. By grounding answers in visually verified sources, ChartCitor promises to improve professional productivity, especially in fields heavily reliant on data visualization like finance and healthcare.

The research opens pathways for future exploration in several areas, including the development of methodologies to manage multi-chart interactions and mitigation strategies for LLM hallucinations. A continued expansion on explicit citation mechanisms will further bolster the trustworthiness of outputs from generative AI systems. Such endeavors will be crucial for leveraging AI in professional sectors where decision-making heavily depends on data integrity and accuracy.