ReChartPrompt: Structured Multi-Chart QA

• ReChartPrompt is a structured prompting methodology designed for precise question answering on figures containing multiple sub-charts.
• It serializes sub-chart metadata and uses explicit, step-by-step reasoning instructions to extract, align, and compare visual information.
• Its self-verification phase and uniform metadata ordering boost model accuracy by approximately 5.39 percentage points on benchmarks like PolyChartQA.

ReChartPrompt

ReChartPrompt is a structured prompting methodology designed for question answering (QA) on multi-chart figures, especially in contexts where analytic figures comprise multiple sub-charts requiring coordinated cross-chart reasoning. The method, introduced and evaluated in "Beyond Single Plots: A Benchmark for Question Answering on Multi-Charts" (Efat et al., 23 Apr 2026), systematically serializes chart metadata, scaffolds explicit cross-chart reasoning, and includes a self-verification step, resulting in substantial improvements in accuracy for vision-LLMs on the PolyChartQA benchmark.

1. Motivation and Objectives

ReChartPrompt is motivated by the observation that real-world figures frequently present as composite images containing two or more sub-charts, and that natural questions rarely specify which sub-chart is being referenced. Existing prompting strategies for single plots are inadequate when (a) relevant visual information must be localized within a particular sub-chart, (b) all visual elements are naively pooled regardless of structure, and (c) questions require alignment, comparison, or chaining across multiple sub-charts.

The method explicitly addresses these deficits by:

• Forcing localization of relevant sub-charts via serialized metadata blocks.
• Making the mapping from question clues to chart structure explicit.
• Staging the reasoning process: extraction, computation, then comparison.
• Enforcing a final self-verification step to reduce errors common in multi-chart reasoning.

This design leads to a documented increase in LLM-based accuracy (L-Accuracy) of approximately 5.39 percentage points on PolyChartQA (Efat et al., 23 Apr 2026).

2. Prompt Template Architecture

The ReChartPrompt template is always structured in the following progression:

1. Template Prefix: Establishes the role and context, e.g., "You are a chart reasoning assistant. A figure contains two sub-charts side by side. Follow the plan below to answer the question and then self-verify your steps."
2. Metadata Serialization: For each sub-chart $i=1\dots n$ (where $n$ is the number of sub-charts), a standardized metadata block is provided:

=== Sub-chart [i] === Title : <chart_i_title> X-axis : <x_label_i> Y-axis : <y_label_i> Legend : <list_of_legend_entries_i> Data Series: <series_names_and_encodings> Chart Type : <bar/line/scatter…>

All blocks are listed sequentially by index.

1. Explicit Cross-Chart Interpretation Instructions: A fixed, numbered skeleton that operationalizes the cross-chart reasoning plan:

Instructions: 1. Identify which sub-chart(s) are relevant given the question. 2. From each relevant sub-chart, extract needed values or trends. 3. If comparison is required, align the same metric or legend across charts. 4. Combine results to answer the question. 5. At the end, verify that sub-chart references and numeric readings match the metadata above.

2. Question and Answer Format:

Question: <natural_question_text> Answer:

The model fills in a detailed, chained reasoning trace under "Answer:" and concludes with a concise final response.

3. Underlying Design Principles

The essential design heuristics and principles underlying ReChartPrompt are:

• Metadata-first: All sub-chart metadata is grouped before presenting the question, enforcing a "where/what" grounding phase before reasoning.
• Clear Delimiters: Each sub-chart and metadata field is unambiguously delimited, ensuring that index-based referencing is reliable and learnable by models.
• Explicit Reasoning Plan: Rather than leaving cross-chart logic implicit, a concrete, stepwise "plan skeleton" is embedded within the instructions, guiding the model to perform extraction, alignment, and comparison operations in an interpretable order.
• Self-Verification: A final mandated step in the plan compels the model to check that its sub-chart references and numeric readings align with the serialized metadata, reducing axis and series misalignment errors.
• Uniform Field Ordering: Enforcing identical order and naming for each field within metadata blocks encourages schema learning and more robust parsing by the model.

4. Formal Specification

The composition of the ReChartPrompt can be formalized as follows. Let $\text{Serialize}(\mathsf{Chart}_i)$ denote the metadata serialization block for sub-chart $i$:

$$\text{ReChartPrompt} = \text{PREFIX}_{\text{role + task}} \;\bigg|\bigg| \; \sum_{i=1}^{n} \text{Serialize}(\mathsf{Chart}_i) \;\bigg|\bigg|\; \text{INSTRUCTIONS} \;\bigg|\bigg|\; \text{Question}$$

with:

$$\text{Serialize}(\mathsf{Chart}_i) = [ \texttt{=== Sub-chart [}i\texttt{] ===} \;||\; \texttt{Title: }t_i \;||\; \texttt{X-axis: }x_i \;||\; \texttt{Y-axis: }y_i \;||\; \texttt{Legend: }L_i \;||\; \texttt{Type: }c_i ]$$

where "$||$" denotes string concatenation with newlines. The model receives the entire composite as a single input string.

5. Representative Example

Consider a human-authored question from PolyChartQA: "Which colour is used to represent the optimized prompt for phi-2 model?" on a figure with two bar charts ("phi-1 model results" and "phi-2 model results").

ReChartPrompt encoding for this example:

$n$0

Expected model output demonstrates indexed localization, explicit legend reference, and a verification step:

$n$1

6. Empirical Gains and Impact

Applying ReChartPrompt to the PolyChartQA benchmark produces a substantial 5.39 percentage point increase in L-Accuracy relative to baseline multi-chart prompting. This gain stems from:

• Elimination of sub-chart localization errors via index-referenced metadata.
• Instructional scaffolding that enforces necessary extraction, alignment, and comparison steps within and between sub-charts.
• An explicit verification phase that greatly reduces errors due to axis confusion, incorrect bar selection, or improper legend matching.
• Inductive schema learning due to uniform serialization and field-order constraints.

Empirical evaluation shows that off-the-shelf single-plot prompts incur significant performance drops primarily due to failures in step 1 (relevant sub-chart identification) and step 3 (comparison/alignment). ReChartPrompt bridges this performance gap (Efat et al., 23 Apr 2026).

7. Implementation Guidelines

To implement ReChartPrompt in a practical workflow:

1. Chart Parsing: Use OCR or chart-specific parsers (e.g., DePlot, Matcha) to extract sub-chart metadata (title, axes, legend, series, type).
2. Prompt Formatting: Format all extracted metadata into sequential Serialize blocks, following the prescribed field and delimiter order.
3. Template Concatenation: Prepend the standardized PREFIX, append the fixed INSTRUCTIONS, and end with the user's question.
4. Model Input: Provide the full prompt (metadata, instructions, question) to the vision-LLM in a single input.
5. Model Output Parsing: Parse the model’s answer block to separate stepwise reasoning from the final answer statement.
6. Verification: Optionally post-process to enforce that referenced chart elements in the answer coincide with the serialized metadata.

This three-stage design—metadata serialization, explicit structured reasoning, and self-verification—enables robust handling of multi-chart images, providing higher reliability, cross-chart grounding, and improved answer accuracy compared to prior approaches (Efat et al., 23 Apr 2026).