Chart-RL: Policy Optimization Reinforcement Learning for Enhanced Visual Reasoning in Chart Question Answering with Vision Language Models

Abstract: The recent advancements in Vision LLMs (VLMs) have demonstrated progress toward true intelligence requiring robust reasoning capabilities. Beyond pattern recognition, linguistic reasoning must integrate with visual comprehension, particularly for Chart Question Answering (CQA) tasks involving complex data visualizations. Current VLMs face significant limitations in CQA, including imprecise numerical extraction, difficulty interpreting implicit visual relationships, and inadequate attention mechanisms for capturing spatial relationships in charts. In this work, we address these challenges by presenting Chart-RL, a novel reinforcement learning framework that enhances VLMs chart understanding through feedback-driven policy optimization of visual perception and logical inference. Our key innovation includes a comprehensive framework integrating Reinforcement Learning (RL) from Policy Optimization techniques along with adaptive reward functions, that demonstrates superior performance compared to baseline foundation models and competitive results against larger state-of-the-art architectures. We also integrated Parameter-Efficient Fine-Tuning through Low-Rank Adaptation (LoRA) in the RL framework that only requires single GPU configurations while preserving performance integrity. We conducted extensive benchmarking across open-source, proprietary, and state-of-the-art closed-source models utilizing the ChartQAPro dataset. The RL fine-tuned Qwen3-VL-4B-Instruct model achieved an answer accuracy of 0.634, surpassing the 0.580 accuracy of the Qwen3-VL-8B-Instruct foundation model despite utilizing half the parameter count, while simultaneously reducing inference latency from 31 seconds to 9 seconds.

• The paper introduces a novel RL framework that directly fine-tunes vision-language models to improve chart comprehension and answer accuracy.
• It leverages policy optimization techniques such as GRPO, DAPO, and GSPO alongside LoRA-based fine-tuning to drastically reduce computational costs.
• Experimental results demonstrate significant gains in reasoning quality, accuracy, and inference latency compared to both open-source and proprietary models.

Policy Optimization for Enhanced Visual Reasoning in Chart Question Answering

Introduction

The paper "Chart-RL: Policy Optimization Reinforcement Learning for Enhanced Visual Reasoning in Chart Question Answering with Vision LLMs" (2604.03157) proposes a reinforcement learning (RL) framework for improving the chart comprehension and reasoning capabilities of vision-LLMs (VLMs) in the Chart Question Answering (CQA) task. This work addresses persistent bottlenecks in CQA, including inaccuracies in numerical extraction, inability to interpret implicit chart relationships, and poor attention to spatial data dependencies. The authors present a systematic RL pipeline, integrating advanced policy optimization techniques, parameter-efficient fine-tuning (PEFT) via LoRA, and adaptive, LLM-based reward systems. Empirical evaluation demonstrates that the RL-fine-tuned models achieve substantial gains in accuracy and efficiency compared to both proprietary and open-source SOTA baselines.

Advances in Chart Visual Reasoning: Challenges and Methodological Innovations

Traditional VLMs have shown limited reliability in CQA due to intrinsic model weaknesses. These include inability to extract precise numeric values from high-density chart elements, failures in visual-perceptual disambiguation (e.g., overlapping graphical components), and central limitations in reasoning about visually grounded, hierarchical, or nonlinear relationships. The proposed Chart-RL framework introduces a unified RL-based chart reasoning methodology that circumvents major failures of sequential, static, discriminative training regimes.

Key innovations include:

• Direct RL Fine-Tuning without Initial SFT: The approach omits supervised fine-tuning stages, relying instead on policy optimization to dynamically refine model responses and reasoning quality.
• Group-based RL from Policy Optimization (GRPO), Direct Advantage Policy Optimization (DAPO), and Group Sequence Policy Optimization (GSPO): These techniques address high-variance reward signals and unstable gradient flows associated with multimodal, long-form output spaces in VLM reasoning.
• Adaptive LLM-based Reward Signal: Leveraging automated evaluation via GPT-4-based LLMs, the framework computes composite rewards considering format, accuracy, and logical reasoning coherence, mapping model outputs to ground-truth answers with semantic tolerance.

Parameter-Efficient Fine-Tuning and RL Implementation

The adoption of LoRA-based PEFT under RL policy optimization is central for scalable and cost-effective deployment. Fine-tuning is limited to low-rank parameter dimensions in the query and value projection layers, drastically reducing the number of weights subject to update (<0.5% of total parameters). This design allows training and deployment on commodity hardware (single 24GB GPU), enables rapid experimentation, mitigates catastrophic forgetting, and preserves domain-general representations in the model backbone.

Experimental Evaluation and Results

Experimental Setup

All experiments use the ChartQAPro dataset, a high-complexity CQA benchmark with diverse chart types and question modalities, and rigorous human-verified ground-truth labels. The dataset’s heterogeneity includes factoid, conversational, hypothetical, multi-chart, and unanswerable cases, mirroring real deployment variability.

Model Benchmarking

Performance is measured by answer accuracy (LLM-as-a-judge) and inference latency. RL-fine-tuned variants of Qwen3-VL-4B-Instruct (using GRPO, DAPO, GSPO) are directly compared with baseline VLMs (Qwen3-VL-8B-Instruct, Janus-pro, InternVL, LLava) and SOTA closed-source MLLMs (Claude Sonnet 3.7/4.5).

Strong Numerical Outcomes

• The DAPO-fine-tuned Qwen3-VL-4B-Instruct achieves the highest answer accuracy among fine-tuned models (0.634), outperforming the Qwen3-VL-8B-Instruct baseline (0.580) despite using half the parameter count.
• Computational efficiency is significantly enhanced, with inference latency reduced by ~71% (from 31.59 seconds for Qwen3-VL-8B-Instruct to 9.48 seconds for Qwen3-VL-4B-Instruct-DAPO).
• All RL-optimized 4B models achieve higher accuracy than the larger open-source competitors, and approach the performance of SOTA closed-source solutions—achieving a favorable latency-efficiency trade-off.

Enhanced Chain of Thought (CoT) Reasoning

Qualitative analyses indicate that RL-fine-tuned models exhibit markedly improved multi-step, explainable reasoning over baseline foundation models. For example tasks requiring aggregation across visual features or extrapolation via causal and statistical inference, baseline models (including SOTA proprietary models) tend to miss intermediate dependencies or misclassify visual elements, leading to propagate errors. RL-optimized models, in contrast, show greater robustness in intermediate reasoning, correct identification of chart semantics, and logical sequence construction in CoT outputs.

Implications for AI Autonomy and Future Vision-Language Modeling

This work has direct implications for practical and theoretical expansion in multimodal reasoning systems:

• Production Viability: The methodology enables resource-efficient deployment without sacrificing reasoning performance, promoting integration in real-world business intelligence, scientific data analysis, and automated reporting pipelines.
• Scalable Policy Optimization Schemes: The paradigm provides a foundation for future integration of multi-stage RL with LLM/human-in-the-loop reward refinement, potentially closing residual gaps in reasoning generalization.
• Advances in RL Reward Engineering: The use of an LLM judge as a verifier for both answer and reasoning process quality sets a new direction for multi-objective RL calibration in complex, weakly structured domains, though further investigation into reward signal reliability and reduction of reward hacking is warranted.
• Nonlinear Multimodal Reasoning: The demonstrated improvements in the model’s ability to synthesize numerical, contextual, and statistical cues from charts underscore the value of RL for endowing VLMs with capabilities approaching the demands of generalized visual reasoning.

Conclusion

The Chart-RL framework substantiates the efficacy of RL-based policy optimization for parameter- and inference-efficient enhancement of VLM chart reasoning. The multi-pronged methodology yields strong improvements in both final answer accuracy and intermediate reasoning quality, at substantially reduced computational cost. This positions RL as a critical subfield for advancing the multimodal, explainable intelligence needed for advanced chart understanding. Continuing work will likely focus on optimizing reward function design—potentially via ensemble or human-in-the-loop mechanisms—and extending these benefits to broader classes of structured visual reasoning tasks.