Self-Evolving Reasoning Cycle (SERC)

• SERC is a reinforcement-driven framework that enables vision-language agents to continually improve reasoning using self-generated rewards and integrated tool use.
• It employs a paired Solver–Verifier system with nested inner and outer loops to perform tool-augmented reasoning, self-evaluation, and adaptive self-repair.
• Empirical results demonstrate significant performance gains on benchmarks like MathVerse and HallBench through iterative self-evolution cycles.

The Self-Evolving Reasoning Cycle (SERC) is a reinforcement-driven, closed-loop framework that enables vision-LLMs (VLMs) and large multimodal agents to autonomously improve their multimodal reasoning capabilities without any human-provided supervision or external reward models. SERC is instantiated in Agent0-VL as a paired Solver–Verifier system utilizing tool-integrated reasoning, fine-grained policy interaction, structured self-reward, and a dual-loop (inner/outer) learning mechanism, allowing the model to achieve continual self-alignment and skill acquisition on complex multimodal tasks (Liu et al., 25 Nov 2025).

1. Core Structure and Objective

The primary purpose of SERC is the continual, self-supervised improvement of a vision–language agent’s step-wise reasoning accuracy on challenging problems that require both perception (from images) and symbolic manipulations (via tool use). SERC operationalizes this by unifying two parameter-sharing roles:

• Solver: Performs multi-turn, tool-integrated reasoning, selecting from both natural language and structured tool actions (e.g., code execution, OCR, image cropping).
• Verifier: Acts as a tool-grounded self-critic, issuing dense process-level feedback, scoring the factuality of reasoning steps, and optionally invoking its own tool-based verification.

This interleaving allows evidence-based introspection and the repair of errors, jointly optimizing reasoning and verification behaviors in a zero-external-reward evolution cycle. No human-annotated rewards or external evaluators are required; all process and outcome rewards are generated internally and reinforced through repeated self-play.

2. Nested Reasoning Loops: Inner and Outer

SERC is composed of two interdependent loops:

• Inner Loop: For each task, the agent cycles through:
  1. Solver Step: Given the current state (image, question, context, tool traces), the Solver emits either a text action or a tool invocation.
  2. Tool Execution: If the action is a tool call, the corresponding tool output is appended to context.
  3. Verifier Step: The Verifier, seeing (state, action, tool output), issues a tuple (score, confidence, critique) and may re-call the tool for factual grounding.
  4. Self-Repair: If the confidence falls below a threshold (τ_c), a gating function triggers a minimal patch, prompting the Solver to resample a repaired step and pay a repair penalty.
  5. Reward Accumulation: The process-level reward is computed from tool correctness, factually-scored step accuracy, and regularization terms. This is accumulated stepwise over the trajectory.

• Outer Loop: After collecting a batch of such trajectories and their self-evaluated returns, the agent updates its joint Solver–Verifier policy by Group Relative Policy Optimization (GRPO). This reinforcement learning variant uses advantage normalization, PPO-style clipping, and KL regularization to stabilize self-improvement despite the dense reward feedback being model-generated.

3. Formal Algorithmic Framework

A full trajectory through SERC can be schematized as follows:

for k in range(num_iterations): T = [] for (I, q) in batch: # I: image, q: question s1, b1 = (I, q), [] tau = [] for t in range(T_max): # Solver: text or tool action a_t = Solver(s_t, m='S') o_t = execute_tool_if_needed(a_t) tau.append((s_t, a_t, o_t)) # Verifier: feedback and critique V_t = Verifier(s_t, a_t, o_t, m='V') # Compute reward with tool signal and KL-div r_proc_t = lambda_tool * r_tool(a_t) + V_t.score * V_t.conf - beta_div * D_KL(Verifier || Solver) # Self-Repair if low confidence if V_t.conf < tau_c: a_t_prime = Solver(s_t, delta_patch(V_t), m='S') o_t_prime = execute_tool_if_needed(a_t_prime) C_rep = eta # repair penalty else: C_rep = 0 r_t = r_proc_t - C_rep # Advance state s_t = update_state(s_t, a_t, o_t) # Outcome reward r_out = compute_final_accuracy() g_tau = alpha_out * r_out + sum([gamma**t * r for t, r in enumerate(r_t)]) T.append((tau, g_tau)) # Policy update with GRPO policy_update_GRPO(T)

Here, each step’s reward is process-dense and incorporates (i) the Verifier’s tool-grounded accuracy, (ii) step confidence weighting, (iii) explicit tool correctness, and (iv) a regularization aligning Verifier and Solver output distributions [(Liu et al., 25 Nov 2025), Eq. 4, 6, 7, 9].

4. Tool-Integrated Verification and Self-Repair

A central innovation in SERC is the integration of external tool APIs into both the reasoning and self-evaluation pathways. The Solver augments its context by interleaving tool outputs after execution (e.g., Python numerics, OCR results, image crops), which are then referenced in subsequent steps. The Verifier, operating on (state, action, tool output), may invoke the same API to regenerate factual evidence and ground its scoring. This two-level tool mediation addresses failures of purely text-based self-evaluation, directly combating hallucinations and aligning both reasoning and verification distributions.

When low epistemic certainty is detected (conf_t < τ_c), SERC triggers a minimal corrective action (Δ_t), resamples the Solver’s next step in light of Verifier feedback, and applies a small penalty (η) to discourage unnecessary repairs.

5. Hyperparameterization and Model Instantiation

SERC, as realized in Agent0-VL, is instantiated with the following critical settings:

• Unified base models: Qwen2.5-VL-7B or Qwen3-VL-8B (parameter-shared for Solver and Verifier).
• Confidence threshold τ_c=0.7 and sharp gating parameter κ for repair triggering.
• Tool reward scaling λ_tool, divergence penalty β_div, outer reward α_out, and discount γ.
• GRPO: clipping ε, KL penalty β_KL=0.001, group size N=8, entropy bonus β_ent=0.01.
• Supported tools: JSON API for Python execution, image cropping/zoom, OCR, general calculators.
• Mixed-precision bfloat16 training, distributed over 8 GPU nodes.

6. Empirical Characterization and Ablation Results

The isolated impact of SERC is evidenced through ablation on major VL benchmarks:

Configuration	MathVerse Avg	HallBench	ChartQA	MMMU
Full SERC	59.4%	72.9%	87.3%	61.1%
– Self-Repair	57.5%	71.6%	—	—
– Tool Use (text)	53.1%	67.5%	—	—
– SERC (SFT only)	51.8%	65.8%	—	—

Iterative self-evolution yields compound gains: the Agent0-VL-7B model improves from base → +5.2% → +4.0% → +2.8% after three SERC rounds, totaling +12.5% over the backbone on geometric and scientific reasoning (Liu et al., 25 Nov 2025).

7. Structural Workflow

The overall SERC cycle can be conceptualized as two parallel columns:

• Inner Loop (left): Solver’s step → tool execution/output → Verifier’s feedback → optional self-repair/resampling → reward computation/accumulation.
• Outer Loop (right): Batch of trajectories from the inner loop is passed to the GRPO policy optimizer, which updates both Solver and Verifier’s shared parameters, regularized for stable alignment. Through repeated outer-loop cycles, the system’s behavior shifts: the Solver becomes more adept at leveraging tools for correct outcomes, while the Verifier’s confidence and factuality discrimination become sharper, supporting stable lifelong self-improvement.

8. Significance and Outlook

The SERC paradigm addresses longstanding obstacles in autonomous agent reasoning: the inability of language-only self-evaluation to ground complex inference, and the instability of self-reward RL in the absence of external feedback. By interleaving tool-augmented reasoning, evidence-based critique, and adaptive self-repair within a unified policy, SERC in Agent0-VL achieves robust, annotation-free improvement. Empirical results indicate that tool-based verification and self-repair are strongly complementary, with the absence of either leading to substantial performance degradation.

This framework defines a rigorous, extensible pattern for building self-aligning reasoning agents operating in complex multimodal domains, with the potential for further advances through more sophisticated tool integration, richer forms of verification, and expansion to larger distributed agent collectives (Liu et al., 25 Nov 2025).