ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents

Abstract: Tool-augmented vision-LLMs (VLMs) can solve multimodal, multi-step tasks by calling external tools, yet they remain fragile in practice. Existing works have two common gaps. Supervised fine-tuning (SFT) is built mostly on successful trajectories and offers little signal for recovery after tool failures, while sparse trajectory-level RL rewards provide limited guidance on which step failed and how to repair it. We introduce ReGRPO (Reflection-augmented Group Relative Policy Optimization), a framework that learns reflection-guided correction in tool-using agents. ReGRPO starts with a structured reflective data engine: we execute near-miss actions to collect grounded failure observations, then build Reflection-of-Thought triplets (ErrorType, Evidence, FixPlan) paired with corrected actions for warm-start SFT. We then optimize reflection tokens and corrective actions jointly within local trajectories using group-relative advantages, and include a reflection-cost term to reduce unnecessary reflection. Experiments on GTA and GAIA show that, under the same backbone and tool suite, ReGRPO consistently outperforms strong open-source baselines and achieves the best results among the compared open-source controllers. Code and RoT data are available at https://github.com/showlab/ReGRPO.

• The paper introduces learnable reflective tokens that allow tool-using agents to recover from failures by explicitly diagnosing and correcting errors.
• It uses a Structured Reflective Data Engine to convert tool failures into actionable Error–Reflection–Correction triplets.
• Benchmark evaluations on GTA and GAIA show significant accuracy improvements over existing supervised fine-tuning and reinforcement learning methods.

Reflection-Augmented Policy Optimization for Robust Tool-Using Agents: An Expert Analysis of ReGRPO

Introduction and Motivation

Tool-augmented vision-LLMs (VLMs) substantially advance visuolinguistic task automation by enabling agents to invoke external APIs (e.g., OCR, tabular parsers, web search). However, these tool-using agents are fragile in deployment due to two core deficiencies: (1) Supervised fine-tuning (SFT) predominantly uses successful trajectories, yielding a lack of targeted supervision for recovery from tool failures; and (2) scalar reinforcement learning (RL) rewards, as typically employed in trajectory-level optimization, do not provide granular guidance for fault localization and repair. The absence of process-level, causal supervision restricts the reliability of these systems when exposed to distributional shift or unfamiliar failure modes.

ReGRPO (Reflection-augmented Group Relative Policy Optimization) directly addresses these limitations by introducing learnable, structured "reflection" variables into the agent policy. This framework leverages a Structured Reflective Data Engine that internally generates and consumes Reflection-of-Thought (RoT) triplets upon tool failures, providing grounded causal supervision and enabling efficient recovery through explicit Error–Reflection–Correction loops.

Figure 1: Pipeline of the Structured Reflective Data Engine for converting raw failures into RoT triplets and recovery supervision.

Methodology

Structured Reflective Data Engine

The proposed engine departs from conventional dataset generation by systematically perturbing successful MAT-AGENT tool-use trajectories to synthesize realistic, recoverable failure cases. For each perturbed action, a teacher model (e.g., GPT-4o) generates a RoT triplet: ErrorType (categorical diagnosis), Evidence (explicit grounding in failed observation), and FixPlan (actionable correction strategy). This structured annotation directly identifies what failed, why, and how to recover, forming a robust SFT corpus closely tied to the actual error modalities encountered by agents.

Reflection-Optimized Policy Learning

ReGRPO extends standard GRPO by jointly optimizing over both reflection and correction trajectory tokens. It instantiates a new action space that incorporates, for each failure:

• The initial (failed) action,
• The observed failure,
• The generated reflection triplet,
• The corrective action.

Within each RL batch, local groups of candidate trajectories are sampled. Process rewards are computed using a composite function that includes:

• Successful task completion,
• Reflection length penalty (imposing computational and verbosity regularization),
• (Optionally) a deterministic, verifier-derived sub-reward calculated strictly from offline metadata rather than any run-time model call.

  Figure 2: ReGRPO training pipeline: offline generation of reflective supervision, group-based RL with reflection, and efficient deployment with deterministic triggers.

The reflection token cost induces parsimonious reflection usage; agents learn to only generate diagnostic explanations and correction plans when empirically beneficial, controlling unnecessary overhead.

Deterministic, Zero-Verifier Inference

At deployment, agents utilize a fixed, easily reproducible trigger to gate the invocation of a local reflection-correction block after an action. Failures are recognized using a canonical schema for tool outputs (runtime errors, empty payloads, or low tokenwise policy confidence relative to the adaptive trajectory average). Only one reflection-correction is permitted per step, maintaining both efficiency and determinism.

Experimental Results

Comprehensive benchmarking was conducted on the GTA and GAIA datasets, which probe multi-step, tool-use chains in both image-centric and document-centric settings. Under strictly controlled backbones and tool suites, ReGRPO demonstrated consistent, significant improvements over leading open-source baselines:

Method	GTA ToolAcc	GTA AnsAcc	GAIA AnsAcc
MAT-Agent SFT	64.63	53.85	16.97
SPORT Agent	72.41	60.26	20.61
ReGRPO (default)	76.35	67.66	23.35
ReGRPO (+verifier)	77.26	68.49	24.01

Table: Main results for tool-use and answer accuracy under a single-path, verifier-free protocol.

ReGRPO's structured reflection mechanism offers an approximately +7.4 AnsAcc improvement over SPORT on GTA, and +2.7 on GAIA. These gains were robust to ablation: even with $\lambda_{\text{val}} = 0$, most of the effect was due to RoT SFT initialization and structured reflection RL. Optional inclusion of deterministic verifier reward provided minor further improvements.

Figure 3: Comparative diagram of PPO/DPO/GRPO/ReGRPO, highlighting the explicit reflection decision variable only in ReGRPO’s group RL.

Qualitative Mechanism Evidence

Analysis of recovered trajectories reveals that ReGRPO can diagnose both explicit execution failures and silent distributional drifts (e.g., OCR returning empty, or model focusing on irrelevant visual context). The agent's output reflection explicitly grounds its diagnosis in failed observations and synthesizes actionable plans, before generating corrected actions. Unlike free-form reflection, the strict triplet schema prevents hallucinatory or vague rationales, ensuring alignment with factual failure modes.

Reflective blocks are invoked exclusively when empirically beneficial, controlled via token cost penalty and confidence-based triggering. This leads to both improved recovery rates and tightly scoped agent explanations.

Implications and Future Directions

Practically, ReGRPO enables tool-using multimodal agents to robustly recover from a broader range of failure modes, even under distributional shift. The explicit modeling and optimization of diagnostic reflections internalizes the Error–Reflection–Correction loop, replacing offline heuristics or external iterative self-correction (as in Reflexion-style LLM agents) with parameter-level adaptation.

Theoretically, ReGRPO demonstrates that joint optimization of diagnosis and correction within a group RL framework produces more informative credit assignment than scalar rewards or preference-based optimization alone. This architecture could be further extended to multi-agent settings, continuous or open-ended exploration, or domains requiring both reflection and verification (e.g., robust synthesis in program induction with tool APIs).

Potential future work includes scaling the taxonomy of ErrorTypes, integrating structured reflection into toolchain design for robotic or embodied VLM agents, and exploring the interaction between reflection quality, reflection cost, and downstream efficiency in long-horizon planning.

Conclusion

ReGRPO introduces a rigorous, reflection-augmented architecture for tool-using vision-language agents. By combining a Structured Reflective Data Engine with group-based joint optimization of action and reflection, agents exhibit improved sample efficiency, robust recovery from failure, and state-of-the-art tool-use performance on established multimodal benchmarks. This work underscores the necessity and tractability of learning explicit diagnostic and recovery capabilities in complex tool-augmented decision-making systems.

Figure 4: Menu image used for grounded reasoning in constrained selection task (Case A1).

Figure 5: Product image for multi-step entity resolution and knowledge retrieval (Case A2).